In Japan, 57% of female and 48% of male students seek counseling voluntarily [1]. Others seek counseling after being referred by teachers, families, friends, or health service centers. Health service centers use health surveys as clues for counseling calls. A health survey is mainly conducted as a self-reported screening test or an interview along with health checkups [2].

The style of mental health surveys differs among universities. Participants in the survey range from first-year students to undergraduate students to all students, including graduate students. The content is also diverse, with the University Personality Inventory (UPI) [3] and General Health Questionnaire (GHQ) [4,5] being frequently used [6]. Similarly, the criteria for calling using a mental health survey are inconsistent [7]. It is challenging to devise uniform criteria that will aid in efficient calling. To address this issue, we propose that machine learning (ML) would aid in developing a universal method for detecting and predicting mental health problems among students.

ML has several advantages in modeling highly individual phenomena with many variables, such as allowing the simultaneous testing of numerous input variables and their complex interactions [8], permitting nonlinearity in producing predictive algorithms [9], and having the ability to test all possible relationships to identify the superlative algorithm and model without a priori hypothesis by researchers [8]. These advantages of ML can help detect and predict mental health problems.

Many trials detect mental health problems using biological, behavioral, and subjective data [10,11]. Some studies have diagnosed suicidal ideation or behavior using various predictors, such as psychological problems and self-injuries [12,13]. To predict mental health problems, the following broader variables were tested: birth information; physical illness; environmental factors; behavioral data (eg, number of naps, study duration, and use of a cell phone); biological data (eg, sleep onset time, skin conductance, and temperature) collected with wearable sensors, and text data posted on social media [14-17].

There are also some examples of ML with on-campus data, known as learning analytics and educational data mining [18]. In some studies, repeaters and dropouts have been predicted using log data from a learning management system or e-learning, information about entrance examinations, registered courses, attendance, grades, and submission status of assignments [19-24].

On-campus mental health surveys are insufficiently used as ML predictors. We consider them suitable for statistical analysis for 3 reasons. First, the mental health survey is open to all students, and the response rate is high. They must complete it to receive health checkups. Second, mental health surveys are conducted annually at many universities. Thus, it is possible to observe a change over time. Third, web-based mental health surveys, which have become a standard under COVID-19, can collect well-organized self-reported data and answering time data.

We use the word answering time as a broad concept, including response time (RT), which is the amount of time in which a student responds to the question item after it is shown on the screen; RT stamp (the time of day when a student accesses the survey form); and answer date (the number of days elapsed in which a student completes the survey after it was announced). In particular, RT has recently attracted attention mainly in psychological assessment and educational testing; however, little is known about how to use RT in mental health measurement [25,26]. A few studies have shown that RT is related to depression estimation and the prediction of suicide ideation and attempt status [25,27]. RT would be worth investigating for predicting mental health based on the response behavior.

This study aimed to predict students’ mental health problems even without a specific mental health score. In this study, we conducted 2 analyses. In analysis 1, we attempted to predict mental health problems from other items when responding to the survey. Analysis 2 attempted to predict mental health problems in 2021 from the responses in the survey conducted in 2020 (1 year later). We also tested the effect of answering time on the prediction of mental health problems.

The main objective of this study was to build an exploratory predictive model. We also attempted to evaluate the impact of each input variable on the outcome based on ML and examine the interpretable relationship between input variables and mental health problems.

This study demonstrates the possibility of using data collected from health surveys conducted at most universities to precisely detect students experiencing mental health problems and reduce call costs. This will also pave the way to using existing health survey data even without mental health scores.

The data set was obtained from a closed health survey conducted at the beginning of 2020 and 2021 school years at University A, a national university in Japan. According to the email announcement of the annual health checkup, all students were asked to answer the survey on the learning management system. Although incentives were not offered and penalties were not levied, the students were required to answer this survey to obtain a health certification. However, the survey did not comprise adaptive questions. There were 28 nonrandomized questions (health survey in Table 1) that were presented one per page without completeness check. Students could review their answers before submission and resubmit them before the deadline. When there were duplicate submissions from identical IDs, the last submission was recorded.

In this study, data that met the following conditions were excluded: during the 2020 or 2021 survey, the participant was, not an undergraduate, did not answer by the deadline (27 days in 2020 and 39 days in 2021), did not respond to any items, did not respond to all items on the Kessler 6 mental health scale [29] (it was used as an outcome and detailed in Outcome and Input Variables section), was a nonregular student (eg, credited auditors), and did not give consent for data use. Incomplete submissions or atypical response duration were not excluded.

In 2020 and 2021, a total of 5690 students may have been affiliated with the university based on student enrollment conditions in 2020. In both the 2020 and 2021 surveys, 3680 students answered the survey. In total, 119 (3.23%) students’ data were excluded: 10 (8.4%) students did not consent, 105 (88.2%) students did not complete Kessler 6 scale in 2020 or 2021 survey, and 4 (3.3%) students belonged to old affiliations. The resulting data set used for the analysis contains 3561 students (Table 2).

The mean age was 20.54 (SD 1.75) years. The distribution of students by sex was 1941 (54.51%) male and 1620 (45.49%) female students. The data set included 1234 (34.65%) first-year students, 1005 (28.22%) second-year students, 1018 (28.59%) third-year students, and 304 (8.54%) fourth- or higher-year students. The distribution of students by faculty was 1361 (38.22%) in humanities and social sciences (6 departments); 943 (26.48%) in science and engineering (8 departments); 1175 (33%) in medical, pharmaceutical, and health (4 departments); and 82 (2.3%) in others (2 departments). Furthermore, 10 (0.28%) students were international students.

This study received ethics approval from the Medical Ethics Committee of Kanazawa University (test number:2011-113 [017]). All students were asked to complete the survey; however, they had the option to not consent for its use for research purposes. We obtained written informed consent from all participants. Data were anonymized and saved offline. The usability and technical functionality were tested by staff members in advance. The research was performed according to relevant guidelines and regulations.

Our model did not use the diagnosis but used the Japanese version of the Kessler 6 [28] as an indicator of mental health problems. Diagnoses of the students were not necessarily shared with the university, whereas Kessler 6 could be obtained from the health survey conducted in the university. The Japanese version of the Kessler 6 demonstrates high performance in detecting mood or anxiety disorders according to the Diagnostic and Statistical Manual of Mental Disorders Fourth Edition [28,30]. Although diagnoses are not made based only on Kessler 6 scores and need consultations with physicians, they have been used widely for screening and surveys [31]. Analyses 1 and 2 used the Kessler 6 scores for 2020 and 2021, respectively, as outcomes. Both analyses divided scores into a dichotomous variable (1: problem and 0: no problem) according to a ≥13 cutoff point [29,30].

In this study, we examined several candidate ML models to determine which model to use. All candidate models are probabilistic prediction models: logistic regression, elastic net, random forest, XGBoost, and LightGBM. Elastic net is a logistic regression model with L1 and L2 regularization. Random forest combines independent decision trees and outputs the most popular class as a prediction [32]. XGBoost and LightGBM are implementations of the gradient boosting decision tree (GBDT) [33,34]. The GBDT is known to achieve high performance in many ML tasks, such as predicting diseases [35] or identifying psychological health risk factors [36].

We used the same input variables for analyses 1 and 2. All the responses to the 2020 questionnaire listed in Table 1 were coded differently according to the model. For tree-based models (random forest, XGBoost, and LightGBM), they were coded numerically, with missing values filled with −99. For example, the answer for blood type item, which was one among A, B, O, AB, do not know, and no response, was coded as 1, 2, 3, 4, 5, and −99, respectively. For the regression models (logistic regression and elastic net), different coding schemes were applied based on the type of input variables. Continuous or ordinal variables such as hours of internet use or age were numerically coded in the same manner as tree-based models, except that missing or nonresponse data were filled with the grand mean. In contrast, effect coding was applied to nominal variables such as case history or blood type. Missing data were treated as independent categories. As a result, the response to blood type (Q7), for instance, was transformed into 5 input variables: ANS7_B, ANS7_O, ANS7_AB, ANS7_DN (do not know), and ANS7_NA (missing). When the respondent answered “B,” ANS7_B was coded as 1 and the remaining 4 were coded as 0.

Because the aim of this study was to investigate which type of input variables were useful in the prediction of mental health problems, we used all variables except RT without dimensionality reduction. Notably, the tree-based models are less susceptible to the inclusion of irrelevant variables. In addition, an elastic net can reduce the number of input variables because of regularization.

In addition to the raw response variables in Table 1, we generated several input variables with the expectation that some patterns would appear on each variable, as shown in Table 3. These conjectures are based on existing studies on students’ mental health and suicide [37], shared knowledge among experts, or authors’ clinical experiences at the university.

To deal with the content of answers to the free description question (What you want to tell us and questions), the answers were analyzed using KH Coder [38,39], a free text mining analysis software. First, words that appeared ≥10 times were clustered using the Ward method, and Jaccard distances between clusters were calculated. Six clusters were adopted for interpretation. Second, each free description was labeled as 0 or 1 for each cluster. If words in a cluster did not appear in a free description, the description was labeled as 0 for the cluster; if words contained in the cluster appeared even once in a free description, the description was labeled as 1 for the cluster. Cluster 1 contained, among other words, “counseling,” “current,” and “under treatment.” Cluster 2 contained, among other words, “syndrome,” “surgery,” and “diagnosis.” Cluster 3 contained, among other words, “menstruation” and “anemia.” Cluster 4 contained, among other words, “left,” “ear,” and “right.” Cluster 5 contained, among other words, “depression,” “mental,” “stress,” and “anxiety.” Cluster 6 contained, among other words, “hospital,” “allergy,” and “skin.”

This study examined the effectiveness of answering time in predicting mental health problems. Table 4 lists the answering time–related input variables.

The 2 coordinates Xs and Ys represent the position of the time stamp in a 24-hour clock and are introduced to indicate that 0:00 and 24:00 are identical. They were calculated by

where s is the number of seconds elapsed in a day; for instance, when the time stamp is 13:34:42, s will be 13 × 3600 + 34 × 60 + 42 = 48882. RT-related variables were included as the first and second principal component scores because the RT of every single item is difficult to interpret, and these RTs can moderately affect the prediction.

Except for the text-mining analysis, the abovementioned feature engineering processes were executed using the R programming language [40] on a Windows 10 machine, with the missMDA package [41] for principal components analysis with missing data.

In this procedure, we did not split the data into training and test data. This is because the values for the obtained metrics varied considerably depending on the data split owing to data imbalance. Thus, only 5.5% (196/3561) of the students in 2020 and 6.88% (245/3561) in 2021 had a higher risk of mental illness problems as assessed by Kessler 6, including 2.33% (83/3561) of the students who had problems both in 2020 and 2021. Typically, such imbalanced data are dealt with by applying sample weight to equal the total number of positive and negative observations. However, our preliminary comparison performed better when we did not impose sample weights on all candidate models. Therefore, we did not use any particular procedures for dealing with imbalanced data.

First, we compared the performance of candidate ML models to determine which model to use. The procedure was divided into 2 parts: parameter tuning and performance evaluation. Both parts were conducted based on the following procedure (K repeats of 5-fold cross-validation; Figure 1 shows the schema of the procedure):

In the parameter tuning part, the model’s hyperparameters were optimized based on the tree-structured Parzen estimator [42]. The log-loss, as a model performance measure, in a specific parameter set was measured by K=50 repeats of 5-fold cross-validation. Log-loss at fold f of the k-th repeat is calculated by

where y_i and p_i are the realized value and predicted probability of respondent i, respectively, and n is the number of observations in the fold f. As a result, the log-loss for a specific parameter set was obtained by

All the hyperparameters tuned in this study are summarized in Table 5 [43-46]. These parameters can cope with overfitting if they are appropriately tuned. In addition, the learning rate parameters in the GBDT models (eta in XGBoost and learning rate in LightGBM) were fixed at 0.01. Generally, the smaller the learning rate, the more accurate is the prediction, although the computation time increases.

As a result, the performance in each parameter set was measured as the average of K=50 log-losses. Finally, the parameter values that minimize the objective performance measure were adopted, and we proceeded to the performance evaluation. By using the optimized parameter set, we obtained 8 model performance measures (log-loss, Brier loss, area under the curve of receiver operating characteristic curve [AUC-ROC], area under the curve of precision-recall curve [AUC-PR], specificity, precision, recall [also known as sensitivity], and Matthews correlation coefficient [MCC]). Log-loss, Brier loss, AUC-ROC, and AUC-PR are measures based on predictive probability (p_i). In contrast, the specificity, precision, recall, and MCC are obtained from the confusion matrix. For example, MCC is calculated as follows:

where TP, TN, FP, and FN are the numbers of true-positive, true-negative, false-positive, and false-negative respondents, respectively. As previously mentioned, this study used imbalanced data. Therefore, we included the AUC-PR and MCC, which are commonly used in such cases [47].

On the basis of the performance measures obtained, we selected the best model. To test the extent to which answering time–related variables contribute to prediction performance in the selected model, we also conducted parameter tuning and performance evaluation phases without all answering time–related variables (shown in Table 4). We call the analysis without answering time–related variables “without condition, whereas the analysis with all input variables is called with condition.

Furthermore, we calculated feature importance measures (Gain and Shapley additive explanations [SHAP]) on both with and without conditions. SHAP is based on the Shapley values, the average marginal contribution in game theory. With the firm theoretical basis of game theory, SHAP can explain the extent to which each input variable contributes to the model’s prediction [48]. Although several approaches exist for interpreting the prediction, we used SHAP because it is easy to implement and understand. Using the SHAP dependence plot, we can understand the relationship between the actual input value and its effect on the predicted probability. The abovementioned procedures were executed using all the data in K=100 repeats of 5-fold cross-validation. After model evaluation, we obtained SHAP dependence plots on several variables to interpret the model prediction.

All the procedures described in this section were executed using the Julia programming language [49] with several libraries: MLJ.jl [50], TreeParzen.jl [51], ScikitLearn.jl [52], LightGBM.jl [53], and ShapML.jl [54]. Compared with several famous languages, such as R and Python, Julia is still a young language and is rapidly developing. However, it is already perceived as sufficiently stable to be used for research purposes [55,56].

Among all the models, elastic net and LightGBM performed the best in analyses 1 and 2 (Tables 6 and 7). We adopted the LightGBM in this study based on the confusion matrix (Figure 2); however, the elastic net can also have an adequate ability to predict mental health problems not only in real time but also 1 year in the future.

Some of the answering time findings are listed in Table 8. Whether the with condition outperformed the without condition depended on the performance measure considered. This is possibly because the LightGBM can obtain excellent performance even without answering time–related variables on these data. However, as shown in Tables 9 and 11, certain variables are apparently helpful in improving the prediction and affecting the prediction probability. In summary, it is difficult to determine whether we should use answering time–related variables to predict mental health problems.

Focusing on input variables, those from nine questions about campus life improved performance (Tables 9 and 10) and were commonly effective for prediction (Tables 11 and 12). Comparing Gain and SHAP in with condition (Tables 9 and 11), it is suggested that some input variables converted from the answering time substantially impacted the learning. With respect to the without condition (Tables 10 and 12), it is suggested that the input variable, years of university, could improve the prediction to some extent.

In Tables 9-12, the input variables from nine questions about campus life held a top-ranking position. In particular, the item “I have a strong anxiety about my campus life” had a stable and powerful influence on prediction. The questions were easy to use because there were only 9 yes or no questions. Figure 3 suggests that students were more likely to have mental health problems when they responded positively to ≥5 items of the 9 questions. Furthermore, the variable nine questions about campus life could be used by faculty members to advise student orally in addition to health staff members. Their response will be useful for whole-university support and for early detection.

Sleeping hours affected learning and prediction more in analysis 2 than in analysis 1 on the “without” condition (Tables 10 and 11). This implies that sleeping hours were less effective in learning and predicting mental health problems within 1 year. Sleeping hours may be a fluid measure that is affected by lifestyle and busyness. Figure 4 suggested that 6 to 8 hours of sleep were desirable in terms of mental health.

In Japan, 7 hours and 22 minutes are the average sleeping hours [57]. More than one-third of the students in this survey slept for 7 hours. Sleeping too little or too much is a representative symptom of mental health disorder (eg, depressive disorders) [58]. Even without any disorder, sleep diminution because of overwork and busyness could cause mental health problems, and sleeping too much may cause problems in academics. It can be useful for students sleeping <6 or >8 hours to be instructed on how to improve their life rhythm using flyers. Faculty members should keep in mind the need for students to sleep for >6 hours when handing assignments.

Years of university was an effective parameter in both analyses 1 and 2, and in both with and without conditions (Figure 5). The average SHAP effect of second-year students indicated that predictive probability decreased in the second year. It was reported that first- and last-year students tend to experience academic stress [59]. The results depicted in Figure 5 may imply that students adapt to the campus life from the beginning of the second year to their third year. Although some universities hold health surveys only for first-year students, this study indicated that third- or higher-year students should also be monitored.

Figure 6 is consistent with the hypothesis that some students who belonged to departments with few same-sex students may have had difficulties. By contrast, a low proportion of the same sex in a department equals a high proportion of the other sex in that department, indicating a low predictive probability of the other sex. Considering that even an approximately 0.50 proportion of the same sex in the department showed positive averages of the SHAP effect, it is unclear how much the male:female ratio is desirable for students’ mental health. A low proportion of female students in the science and engineering departments reflected the gap in field of science, technology, engineering, and mathematics. The aspect of sex associated with these data was only male or female; the lesbian, gay, bisexual, transgender, queer community should be considered in future studies. Regardless of sex, being in a minority in a department should be a parameter to be examined; such examinations may afford opportunities for interaction.

There are certain findings regarding the input variables generated based on answering time. Concerning the RT stamp, the results depicted in Figure 7 were partly consistent with the hypothesis that RT stamps may reflect students’ life rhythm and sleep quality, which may be related to their mental health state. Students whose answers were recorded at midnight may have stayed up late, which may have been related to mental health problems. In contrast, it is unclear why students whose answers were recorded at noon (approximately 12:00) or evening (approximately 18:00) had negative SHAP values. This might suggest that they had been alone when groups of other students enjoyed lunch or after school. RT stamp data have not been used before; such data can be informative for understanding students’ lifestyles. Although the answer date was not ranked in Tables 9-12, the results depicted in Figure 8 were consistent with the hypothesis that late answer data may have reflected some problems. We believe that the high SHAP of late responses may be caused by the lack of ability to check their emails and obtain essential information from them. Students who answered the survey at a later date should be asked about difficulty in schoolwork (eg, submission of assignments) in health checkup interview. Demonstrating that answering time affects the prediction and learning of mental health problems is meaningful because this parameter can easily be collected through a web survey.

The data were collected from only one university during COVID-19; therefore, the model cannot be generalized to other universities and ages, whereas the approach can be. Campus life during COVID-19 differed from that without the virus. Moreover, this study could not reveal the condition of the 1994 students who did not provide answers in 2020 and 2021, even though they accounted for a major proportion of the population and may have had some mental health problems. The nine questions about campus life, which were the most powerful items in this study, are unique to University A and are not widely used. This model should be tested in other time periods or with other universities’ data. Using this model to call for psychological counseling for students at risk in a specific field is entirely different. This model should be evaluated to determine whether it helps in precisely detecting students experiencing mental health problems and to estimate the impact on cost reduction.

Students’ mental health problems were predicted in real time and for 1 year in the future. The nine questions about campus life, especially the question “I have a strong anxiety about my campus life” was an overwhelmingly powerful item. It was indicated that demographic data (eg, years of university, proportion of the same sex in department, etc) and behavioral data (sleeping hours and answering time), as well as self-rating items were effective.

The developed model itself should be adjusted for each university because it depends on the items used in the survey of the cooperating university. Nevertheless, this model demonstrates the possibility of synergistically using the characteristics of health surveys and advantages of ML. Consequently, it can be used for predicting mental health status from existing health data without a mental health scale.

Furthermore, the impact of some items on the prediction was discussed. These findings can improve health survey items and define the criteria for inviting to student counseling. Accurate calls for student counseling will lead to early detection and intervention and operational efficiency.