In this research, we used highly deidentified retrospective administrative data from Ontario Shores to build ML models for predicting new outpatients’ wait times. Our focus was on 8 outpatient clinics, namely, Anxiety and Mood Disorders (AMD) Clinic, Traumatic Stress Clinic, Borderline Personality Self-Regulation Clinic, Women’s Clinic, Prompt Care Clinic, Prompt AMD, Prompt Transitional Aged Youth, and Prompt Adolescent Consultation. Our data included 4998 patients whose first appointment was between April 1, 2017 and September 30, 2019 (both days inclusive). We excluded 30 (0.6%) patients because of missing referral dates. Selection of patients based on the date of their first appointment, rather than their referral date, had caused a selection bias in favor of 2 groups of patients: (1) those who had longer wait times among patients whose referral date was just before April 1, 2017 and (2) those who had shorter wait times among patients whose referral date was just before September 30, 2019. To address this selection bias, we removed all patients whose referral date was before April 1, 2017, and for each clinic, we removed all patients whose referral date was after September 30, 2019 minus the 80th percentile of the wait times in that specific clinic. After removing the biased data, we were left with 4187 referral entries. Table 1 shows the breakdown of patient count by clinic.

Some patients were missing their designated priority level at triage. However, those patients had a priority designated to them at a later date (possibly owing to re-evaluations while waiting). If a priority level at triage was missing, we replaced that with the priority level designated at the closest date after triage.

High dimensionality, that is, having too many variables in a model, may cause many complications, including overfitting and producing a higher sampling variance (ie, sensitivity to small fluctuations in the training set) [32-34]. We reduced the dimensionality of our data by selecting a subset of variables (and discarding the rest) while retaining as much information as possible from all variables. We kept all of the medical variables, and among variables obtained from systems analysis, we calculated the pairwise Pearson correlation and removed the redundant information if the correlation was larger than 90%, as in [35].

We used the generalized extreme studentized deviate method to identify the outliers [36]. This method iteratively applies the generalized extreme studentized deviate test and progressively evaluates anomalies by removing potential outliers and recalculating the test statistic and the associated critical value. The procedure continues until all outliers are identified.

This study was approved by the Research Ethics Boards at Ontario Tech University (15596) and Ontario Shores (19-009-D).

The summary statistics of our data for different clinics are shown in Table 1. The average wait time widely varied between the clinics from 29 days for Prompt Care Clinic to 186 days for Prompt AMD Consultation. For more than half of the clinics, the average wait time was longer than 3 months. We also calculated the median wait times, which varied between 21 days for Prompt Care Clinic and 205.5 days for Prompt AMD consultation, and confirmed long wait times. Our data also showed large standard deviation values, which varied between 23.9 days for Prompt Care Clinic and 138.16 days for Prompt AMD consultation. Having a high variation among patients’ wait times made it difficult for both care providers and patients to be able to plan ahead.

The summary statistics of the appointments across different clinics are shown in Table 2. In total, the data set included 30,342 appointments out of which 4862 (16%) were no-shows. The number of scheduled appointments widely varied among clinics, from 307 appointments for Prompt Adolescent Consultation to 10,506 appointments for Borderline Personality Self-Regulation Clinic. The proportion of no-shows also widely varied across different clinics, from 2.9% (17/584) for Prompt AMD and Transitional Aged Youth Clinics to 22.2% (623/2804) for Women’s Clinic. Figure 2 shows the average wait time and 95% CI of all patients stratified by the priority level and clinic. This figure illustrates that low-priority patients had the longest wait time in all clinics, except for the Women’s Clinic, where the medium-priority patients had the longest average wait time.

We first applied our dimensionality reduction approach to the training set. Variables introduced in the section “Engineered Predictor Variables” appeared to contain similar information. We dropped all of them except for “Number of patients from each priority level in primary and follow-up queues.” We used these variables along with variables introduced in the section “Predictor Variables Included in Our Data Set” to build our ML models. Table 3 displays the best RMSE values obtained from each of the models for each of the clinics when we included the engineered predictors. For each clinic (ie, each row), the best performing method is italicized. As different clinics followed different operational schemes and their wait times had different profiles, there was not a single ML model that outperformed all of the rest across all clinics. However, the random forest method appeared to be the most promising method as it provided the minimum RMSE values for 4 (out of the 8 clinics) and the second minimum RMSE for the other 4 clinics. It is notable that the linear regression method, regardless of its simplicity, outperformed other ML methods at some clinics such as Women’s Clinic. This can be attributed to the existence of linear patterns in the data [39], having small sample sizes [40], and the fact that the grid search method provides an optimal combination of the selected subset of hyperparameter values, but it cannot guarantee the global optimality of the output [13].

Table 4 displays the list of hyperparameters that we used to tune each of the ML methods, the range of values for each hyperparameter, and the selected values for the AMD clinic. The selected values varied across clinics.

Table 5 displays the selected values of the random forest hyperparameters across different clinics. For some settings, the neural network method with 1 hidden layer could be the same as the linear regression method [12]; to avoid duplications, we did not consider such settings for the neural network method.

The random forest method provides measures of importance for predictor variables. These measures of importance help the user to identify variables that have the most and the least impacts on the outcome variable. Figure 3 displays the importance of predictor variables, measured by impurity (variance of the responses) at the AMD Clinic. The rankings of the importance of predictor variables were similar to those shown in Figure 3 at other clinics. According to Figure 3, priorityUpdate, countCurrentlyInService, and queueSize were the most influential variables. In our models, priorityUpdate variables denoted the last priority assigned to each patient, countCurrentlyInService variables denoted how many patients of different priority levels were currently receiving service (ie, were in the follow-up queues) at the referral time, and queueSize variables denoted how many patients of different priority levels were waiting in the primary queue at the referral time. The seasonality variables did not play important roles in wait time prediction.

Although operations management and ML tools have been widely used in different sectors of care for physical diseases to improve the waiting list management, there has not been such studies in the mental health care sector. Previous research has demonstrated positive effects of operational and policy improvements that have taken place to improve the wait list management in physical health care, for example, cancer care [41]. In 2015, the Canadian Wait Time Alliance [42] reported that although there had been significant progress in the wait time management in 5 areas of focus in the 2004 Health Accord (hip and knee replacement, cataract, bypass surgery, radiation therapy, and diagnostic imaging), mental health care struggled with long waiting times and required immediate attention nationwide. That report also outlined that universal measures did not even exist to track access to psychiatric care across the country. Loebach and Ayoubzadeh [43] compared the waiting times for psychiatric patients and patients with physical problems in the province of Ontario, Canada, and concluded that while the former group often ended up waiting beyond the target waiting times specified by the province, the latter group often received their treatments within the target time window.

In addition to adverse impacts that long wait times have on patients’ health conditions, they also cause higher no-show rates that cause operational complications for health care managers [44,45]. The no-show rate may depend on factors such as wait time and quality of care and may vary between 5% and 80% in different health care sectors [44,45]. Figure 4 illustrates the positive correlation between longer wait times and higher no-show rates in our data, which indicates that shortening wait times may decrease the no-show rates as well.

In this research, we applied 6 different ML methods to predict wait times in mental health care and to identify factors that drive the long wait times. The input data were highly deidentified, which limited the data utility. The random forest method enhanced by the system’s knowledge turned out to be the most promising method. The good performance of this model could be attributed to some appealing computational features of the random forest method, including its low sensitivity to outliers and its ability in capturing complex interactions between predictor variables [46]. From a practical point of view, another appealing feature of the random forest method is its relatively low sensitivity to parameter tuning [46]. Identifying the random forest method as a superior ML method to predict wait times is consistent with findings of [47] that identify this method as the most accurate method to calculate the probability of waiting more than 1 day before receiving treatment for patients with opioid use disorder.

The impurity measure of variable importance, which was the basis for ranking predictive variables in Figure 3, indicates that the total amount of increase in the mean square error of the model output resulted from a random permutation of each variable in the test set. According to Figure 3, the long wait times can be attributed to the usage of the priority system in assigning the care resources to patients. In priority systems, low-priority patients are preceded by patients from higher priority levels and may end up waiting an extended period of time to receive a relatively simple treatment. It is likely that during their long wait times, low-priority patients are changed to higher-priority patients owing to condition deterioration. This phenomenon has been observed in other health care sectors, and using the “fast-track” system for low-priority patients has been suggested as a potential solution [48-50]. In a fast-track system, the waiting line is broken into 2 separate lines: one for low-priority patients and one for patients with higher priorities. One potential advantage of this approach is that because of the simpler nature of care required by low-priority patients, they can be attended by “less-trained” clinicians, freeing up the “more-trained” clinicians for patients with more complex needs. One potential disadvantage of the fast-track system is that the improvement in waiting time of low-priority patients may come at the cost of longer wait times for patients with higher priorities.

Small sample sizes coupled with very large variations within wait times of each clinic imposed the main limitation in this study. There was also a significant difference between the wait time profiles of clinics such that the generalized models for all clinics performed poorly in comparison to models for individual clinics. In addition, the following approximations also impacted the accuracy of model predictions.

In this study, we used retrospective highly deidentified administrative data from 8 clinics at Ontario Shores to build 6 different ML models to predict wait times. We enhanced our models by system knowledge to mitigate the limiting impact of deidentification on our data utility. The data included 4187 patients who received care through 30,342 appointments. The random forest method provided the minimum RMSE values for 4 of the 8 clinics and the second minimum RMSE for the other 4 clinics. The priority system was identified as a factor that contributed to long wait times, and a fast-track system was suggested as a potential solution. Despite the challenges with the wait time source data, this research provided Ontario Shores with a deeper understanding of the extent of and contributors to their wait times on a clinic-by-clinic basis. This research provided Ontario Shores with information and knowledge to pursue quality improvement initiatives to reduce wait times.