Participants consisted of 2280 young people aged 12 to 25 years (mean 17.8, SD 3.2), who registered with eheadspace Web chat between September 20, 2016 and February 20, 2018, wanted help for themselves (rather than a friend), and completed at least one service satisfaction survey. Young people were included if they received a direct eheadspace service, which could be either (1) a brief service provision engaging the young person, identifying the presenting problems, and using brief therapeutic strategies to address these problems; or (2) an esupport-targeted strategy delivered following initial contact and engagement of the young person guided by a care plan and following a full assessment.

All young people registering for eheadspace Web chat during the survey period were invited to voluntarily participate. The survey response rate was 8.7%, from a total of 26,234 eligible young people who accessed the service during this period. The eheadspace users can complete a feedback survey each time they access the service; however, if users had completed more than one survey, only the first complete survey was analyzed in this paper.

Service satisfaction surveys were collected from eheadspace clients via an online questionnaire advertised and accessed through the eheadspace service platform. The platform provides other service options (eg, a link to the Digital Work and Study Service and support for family and friends), but only young people who used the eheadspace online service were offered the service satisfaction questionnaire. To access eheadspace online, a user clicks on a link to “online chat” and then provides required registration or log-in information. When a young person logged in and selected the online chat option, a blue box appeared on the side, asking: “Do you have a spare minute or two to complete our survey about your chat today? Thanks in advance if you do.” Young people were invited to click a link to access the questionnaire once they had finished their online chat, which subsequently directed them to the participant information page. Participation was voluntary.

Survey data were combined with demographic information, clinical characteristics, and service information that was extracted from the eheadspace administrative Minimum Data Set. All young people electronically enter data when they first log in to access the service, and each time they access the service and it has been 14 or more days since they last did so. All clinicians enter data at the end of each service session.

Ethics approval was obtained from Melbourne Health.

To develop a measure of eheadspace service satisfaction, relevant items were adapted from the headspace center Youth Service Satisfaction Scale [34] and supplemented by items from a review of existing satisfaction surveys and consultation with eheadspace staff and clients. This resulted in 13 items comprising statements that were each responded to on a 5-point Likert scale from strongly disagree to strongly agree, with higher scores indicating higher levels of satisfaction. The items focused on three different satisfaction domains:

Data were analyzed using IBM SPSS version 24 and AMOS version 25. An exploratory principal components factor analysis of the 13 original eheadspace feedback survey items was first conducted. Confirmatory factor analysis (CFA) with maximum-likelihood estimation was then used to further test the factor structure. Cronbach reliability analysis was conducted to confirm internal consistency.

Descriptive statistics were computed for all variables. Independent t tests and one-way analyses of variance were then used to determine group differences in satisfaction scores based on client characteristics (gender, age group, sexuality, Aboriginal and/or Torres Strait Islander status, areas of residence, reason for attending, clinical stage), and service contact characteristics (type of service, wait time before session, session start time, duration, prior access to a headspace center, number of previous direct service sessions, word count per client and clinician, referral information). Significance was set at P<.01 due to the high power from the large sample size.

Of the 2280 participants, 77.76% (n=1773) were female, and 55.04% (n=1255) were aged between 12 and 17 years, whereas 44.96% (n=1025) were between 18 and 25 years. There were 64.96% (1481/2280) who identified as heterosexual, 70.44% (1606/2280) who lived in a metropolitan area, and 3.77% (86/2280) who identified as an Aboriginal and/or Torres Strait Islander person.

For presenting issue, 58.82% (1341/2280) contacted eheadspace for mental health problems and 26.32% (600/2280) contacted for another issue (14.87%, 339/2280 missing). Of those with a clinical stage rating (ie, excluding “not applicable” and “not enough information available”), more than half (54.38%, 522/960) were rated stage 1a indicating mild to moderate DSM-IV disorder of depression, GAD, or impairment in functioning.

The majority of clients (85.13%, 1941/2280) had an esupport session, whereas 14.87% (339/2280) had a brief service provision session on the day of completing the survey. Half (50.39%, 1149/2280) the sessions lasted 31 to 60 minutes, but sessions ranged from less than 15 minutes to 90 or more minutes. More than half (57.96%, 1125/1941) of esupport sessions lasted 31 to 60 minutes, whereas only 7.08% (24/339) of brief service provisions lasted this long. Only 5.04% (115/2280) did not have to wait to access eheadspace when they visited, 41.49% (946/2280) had to wait less than 5 minutes, and 53.46% (1219/2280) had to wait more than 5 minutes (maximum wait time was 106 minutes). Most (83.29%, 1899/2280) sessions occurred before 11 pm, but 16.71% occurred between 11 pm to 1 am. On average, young people inputted a mean 397.01 (SD 290.05) words, compared to a mean 686.18 (SD 343.25) words from clinicians. The majority of young people (71.84%, 1638/2280) input fewer than 500 words per session, compared with only 30.04% of clinicians (685/2280). Most young people (67.72%, 1544/2280) self-referred to eheadspace, with 30.35% (692/2280) referred by others. Only 21.80% (497/2280) of participants had visited a headspace center prior to accessing eheadspace. Before the session on the day of survey completion, 64.04% (1460/2280) had not previously received esupport, 16.49% (376/2280) had previously received one, 7.54% (172/2280) had previously received two, and 11.84% (270/2280) had previously received three or more esupport sessions.

The 13 original items of the eheadspace feedback survey were subjected to principal components analysis (PCA) to identify the factorial structure. Inspection of the correlation matrix revealed all coefficients above 0.30. The Kaiser-Meyer-Olkin value observed was 0.95, which exceeded the recommended value of 0.60 [36], and the Bartlett test of sphericity [37] was statistically significant, supporting the factorability of the correlation matrix.

The PCA revealed the presence of one component with an eigenvalue of 8.3, explaining 63.8% of the variance. Standardized factor loadings for 12 of 13 items were strong and positive (range 0.64-0.87), except for “eheadspace is easy to use,” which was 0.50. The eigenvalues and factor loadings supported a unidimensional solution, and an inspection of the scree plot revealed a clear break after the first factor [38].

A CFA was run to further test the factor structure (see Table 1 and Figure 1). The following fit indexes were employed to assess model fit: chi-square closest to zero, root mean square error of approximation less than 0.06 (with the lower bound of its 90% confidence interval less than 0.05 to indicate close fit), the comparative fit index greater than 0.90, and adjusted goodness of fit index greater than 0.90 [39]. Based on the recommendation of Holmes-Smith et al [40], the model with the smallest Akaike information criterion (AIC) / consistent AIC (CAIC) was considered the best-fitting model.

Examination of the modification indexes revealed that the item “the session made me feel more hopeful/optimistic” may relate to both the session and expectations subscales. Only by deleting this item altogether did the model fit improve.

Models were rerun with the remaining 12 items. Results revealed that the original one-factor solution was not a good fit for the data. Consequently, the originally hypothesized three-factor solution was considered, along with other possibilities. Examination of the AIC and CAIC indexes showed that a three-factor model was the best fit to the data, but that a three-factor model with a higher order unidimensional construct of overall satisfaction was also a reasonable fit.

Note that the potential outcomes items included a “not applicable” option, as eheadspace clinicians felt that each of these outcomes may not be applicable for some clients. Consequently, to include participant responses to these items for the CFA (which cannot accept missing data), where only some items were not applicable, responses were substituted as neither agree nor disagree. Those who responded not applicable to all potential outcome items were excluded from analyses. Models were run with and without the substituted not applicable responses included, and all were a similar fit as shown in Table 1. Including those who chose not applicable to some questions, but still completed the full survey, reduces bias.

Cronbach alpha statistics showed high internal consistency for the total scale and all subscales (see Table 2). If any individual items were removed the internal reliability of the scales would be reduced. Item-total correlations were calculated, and all subscales showed strong interitem correlations: session satisfaction items (r=.67-.82), potential outcomes items (r=.70-.80), and service satisfaction items (r=.66-.82). Furthermore, session, outcomes, and service items were moderately to strongly correlated (r=.34-.82). This further supports a second-order factor of total satisfaction.

Table 2 shows that overall young people were satisfied with eheadspace; all subscale means were above the scale midpoint and negatively skewed. Young people were most satisfied with the eheadspace service followed by their session, and they were least satisfied with their potential outcomes.

Client characteristics were not shown to influence total satisfaction scores (see Table 3). There were no significant differences in total satisfaction between males and females, adolescents and young adults, those who were heterosexual and those who were LGBTIQ, those who were an Aboriginal and/or Torres Strait Islander person and those who were not, and between young people who lived in metropolitan or regional locations. Those who attended because of a mental health issue were significantly less satisfied than those who attended for other reasons, although the effect was small. There was no effect of clinical stage.

Many service contact characteristics were significantly associated with satisfaction. Table 4 shows that young people who received esupport were more satisfied than those who received a brief service provision. Those who accessed prior to 11 pm were more satisfied than those who accessed late at night. Young people with contact of more than 30 minutes were more satisfied than those with shorter sessions. Clients who conveyed more than 500 words were more satisfied than those who conveyed 300 or less. Similarly, greater satisfaction was evident when clinicians conveyed more than 500 words. Young people who had not previously accessed a headspace center were more satisfied with eheadspace than those who had previously been to a headspace center. Wait time before session, previous eheadspace access, and referral source did not influence total satisfaction.

The satisfaction measure showed good psychometric properties. There was high internal consistency for the overall satisfaction scale and its three subscales. A three-factor model of the eheadspace satisfaction feedback survey was shown to be the best fit to the data and the hypothesized subscales of session satisfaction, potential outcomes, and service satisfaction were evident. A three-factor model with a higher order unidimensional construct of overall satisfaction was also a reasonable fit. A higher order unidimensional overall satisfaction construct is supported by the literature. For example, Simmons et al [31] found support for a global construct of satisfaction in the development of a satisfaction scale for young people attending headspace centers.

Consistent with other service satisfaction research and literature, the measures of service satisfaction were negatively skewed, with most respondents indicating high satisfaction. The measure can help identify the characteristics of clients who are dissatisfied with the service, but it may be less useful in distinguishing among those with positive experiences [41,42]. Consequently, it may be difficult to demonstrate “improved satisfaction” when scores are already so high.

Young people were generally very satisfied with eheadspace, and this did not vary across any demographic characteristics. No demographic differences were evident in this study unlike the findings of Rickwood et al [34] from face-to-face headspace center services, which found that men, younger clients, and those influenced by others to attend were less satisfied. This suggests that the eheadspace service is experienced in a similarly positive way across the demographics measured. However, the demographics of eheadspace clients are quite different from those of the headspace centers, as they are much more likely to be female and present with a high level of psychological distress [25]. This was reflected in that almost 78% of our participants were female, similar to the 80% who are female who access the service. Although the young men who accessed the service were equally satisfied, they were much less likely to access in the first place. The few client characteristics that were measured also did not substantially affect satisfaction. Those who attended because of a mental health issue were less satisfied than those who attended for other reasons, although the effect was small. It did not vary according to the estimate of clinical stage, which suggests that clients were equally highly satisfied regardless of the complexity of their issue. However, it should be noted that the measure of reason for attending was broad, comprising only mental health versus other issues, and the estimate of clinical stage was not obtained via a comprehensive clinical interview. It may be that satisfaction varies according to different types of mental health problems, as the online environment may be better suited to some than others and may be most useful as an adjunct to face-to-face therapy for some more serious and complex conditions.

Service contact characteristics were associated with satisfaction. Overall, young people who had a more comprehensive engagement with eheadspace were more satisfied. They were more satisfied when they received the more substantial esupport service rather than brief service provision, had a longer session, used more words, did not access the service very late at night, and had not previously experienced services from a face-to-face headspace center. This is consistent with research showing that eheadspace clients who had greater engagement showed more progress and depth of counseling and greater alleviation of psychological distress [43].

Similar to the findings regarding satisfaction from face-to-face headspace center services, satisfaction was lowest for perceived outcomes [34]. Many young people only access eheadspace once, so expectations of longer-term outcomes are unlikely to be well-formed. It is possible that many people might not know how to answer, especially if it is one of their first visits or their only visit, and if they only received information and not therapy as part of their session. Further, earlier research revealed that eheadspace clients have high service expectations [44], which are unlikely to be met with the minimal engagement that is typical.

The results of this study should be interpreted in light of its limitations. Notably, providing service satisfaction was optional, and the response rate was less than 10%. Although this was expected, it indicates likely response bias. It may be that only the more satisfied clients chose to complete the questionnaire. Finding ways to encourage more clients to complete the measure, and ensuring that those who are dissatisfied and who have less service engagement respond, are challenges for service satisfaction measures. Other indicators of service satisfaction should also be monitored, such as the number of site visits and drop-off rates.

Some of the measures were limited by the brief online nature of the data collection. In particular, the measure of clinical staging was an estimate. This measure was included to gain some understanding of the potential impact of client complexity, but it is acknowledged that a brief, online minimum data collection procedure cannot provide a robust measure of clinical stage such as can be ascertained through clinical interview within mental health services that adopt a transdiagnostic clinical staging approach (see [21]).

The satisfaction results are regularly monitored and analyzed by the eheadspace service and evaluation staff and are used to guide service improvement. By identifying patterns in client satisfaction, the areas where the service needs to be tailored to better meet different client needs can be identified. Patterns over time can also be used to evaluate the impact of changes in practice.

Note that currently the satisfaction results are only analyzed in aggregate and not linked to individual clients. There is the potential to also examine session satisfaction monitored over time for individual clients through routine outcome monitoring. This is a repeated measurement process carried out therapeutically to improve treatment outcomes, which has been shown to improve treatment engagement and outcomes [45]. An even briefer measure may be required, however, to implement repeatedly, and clients would need to be informed that their individual feedback would be available to their service providers.

The satisfaction measure used in eheadspace solely assessed youth satisfaction as a form of feedback. Much youth mental health research has previously focused on parental and caregiver satisfaction [33]. Parents and caregivers often have different views about service satisfaction [41]. Their views are also important in assessing online youth mental health services, although this may be less relevant for online services where self-referral is predominant and young people can easily access without parental support. Nevertheless, parents and caregivers often play a primary role in whether younger people attend, engage with, and complete treatment, demonstrating the importance of their satisfaction [46]. Patel et al [1] suggested that both young person and parental feedback is essential in assessing satisfaction with mental health services and this should be explored in the online service domain.

Finally, the measure was adapted from the headspace center Youth Service Satisfaction Scale [34]. This scale has been successfully implemented in the face-to-face services, and it was of interest to have a comparative measure for the online service. Given the briefer and more transitory nature of the online service interactions, further adaptations may be required to make the measure most suitable to the online modality.

This paper has described a service satisfaction measure for an online mental health service for young people. The eheadspace satisfaction measure is brief, and it demonstrated sound psychometric properties with good internal consistency and a clear factor structure. The measure showed high levels of satisfaction overall that were responsive to greater levels of engagement with the service. The measure could be adapted for use with other online youth mental health services, which are emerging as a critical setting for mental health service provision as part of stepped-care approaches in mental health [18].