Published on 17.04.19 in Vol 6, No 4 (2019): April
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/12169, first published Sep 24, 2018.
Young People’s Satisfaction With the Online Mental Health Service eheadspace: Development and Implementation of a Service Satisfaction Measure
Background: Online youth mental health services are an expanding approach to meeting service need and can be used as the first step in a stepped-care approach. However, limited evidence exists regarding satisfaction with online services, and there is no standardized service satisfaction measure.
Objective: This study implemented an online youth mental health service satisfaction questionnaire within eheadspace, an online youth mental health service. The aims were to test the questionnaire’s psychometric properties and identify current levels of satisfaction among service users, as well as to identify client and service contact characteristics that affect satisfaction.
Methods: Data were collected from 2280 eheadspace clients via an online questionnaire advertised and accessed through the eheadspace service platform between September 2016 and February 2018. Client and service contact characteristics, potential outcomes, and session and service feedback data were collected.
Results: The service satisfaction questionnaire demonstrated high internal consistency for the overall satisfaction scale (alpha=.95) and its three subscales: session satisfaction, potential outcomes, and service satisfaction. A three-factor model was the best fit to the data, although including a higher order unidimensional construct of overall satisfaction was also a reasonable fit. Overall, young people were very satisfied with eheadspace (mean 3.60, SD 0.83). Service characteristics, but not client characteristics, were significantly associated with satisfaction. Young people were more satisfied with eheadspace when they had greater engagement as evident through receiving esupport rather than briefer service provision, having a longer session and greater interaction with the clinician, and not previously attending a face-to-face headspace center.
Conclusions: The online youth mental health service satisfaction questionnaire developed for and implemented in eheadspace showed good psychometric properties. The measure is brief, has good internal consistency, and has a clear factor structure. The measure could be adapted for use in other online youth mental health services. The young people using eheadspace and completing the feedback survey were highly satisfied. Greater engagement with the online service was shown to be associated with greater satisfaction. No specific client demographic groups were shown to be more or less satisfied.
JMIR Ment Health 2019;6(4):e12169
Mental health problems peak in young people between the ages of 12 and 25 years , and more than one in four young people experience a mental illness each year [ ]. Three-quarters of all mental health problems have their first onset before the age of 25 [ ]. The delay between first symptoms and access to effective treatment can extend to 15 years [ ]. Poor mental health in childhood and adolescence increases vulnerability to mental health problems in adulthood, revealing the importance of prevention and early intervention for this age group [ - ]. It has been estimated that up to half of lifetime mental illnesses may be prevented through early intervention and prevention methods [ ]. Yet, only 13% of young men and 31% of young women seek professional help [ ]. Although treatment rates have improved in Australia following the introduction of subsidized services, help-seeking remains substantially lower for mental health problems compared to physical health conditions [ ].
Online mental health services have emerged as an innovative way to promote self-referral, increase help-seeking behaviors, and meet the high level of unmet service demand, through accessible youth-friendly technology [, ]. The internet is the main source of information for young people [ ], and the Australian Bureau of Statistics reports that 98% of youth aged 15 to 17 years, and 97% of young adults aged 18 to 24 years, accessed the internet during 2014 to 2015 [ ]. The internet has escalated as a source of information for mental health issues in recent years [ - ]. A cross-sectional study of 2000 Australian young people aged 12 to 25 years in 2008 found that 30.8% seek online mental health information and that 21% of those aged 12 to 17 years and 34% of those aged 18 to 25 years reported searching online for mental health help [ ]. Factors associated with internet use for mental health information included being female and using the internet after 11 pm. Research also shows that young people feel comfortable accessing online information about mental health issues, which may improve help-seeking behaviors for those who may not otherwise seek help using traditional approaches [ ].
A growing range of online options are available, from self-help psychoeducation to interactive sessions with a mental health clinician . These can be incorporated within integrated systems of mental health care, including emerging stepped-care approaches, which are staged systems comprising a hierarchy of interventions from least to most intensive that are matched to an individual’s needs [ ]. Although such approaches need to be flexibly implemented so that they are appropriate to severity, need, and stage of presentation, online resources and services can comprise part of the spectrum of interventions [ ]. Importantly, e-therapy can be included in ways that complement and extend traditional therapeutic approaches. E-therapy has been shown to be particularly effective for mild to moderate presentations and for the common mental health problems of anxiety and depression, but applications in more complex and serious presentations are also emerging [ ]. Online services have many advantages, including increasing mental health literacy [ ], potentially improving help-seeking behaviors for face-to-face therapy [ ], and providing support that is easily and widely accessible, while reducing the costs to service providers as part of an integrated system of care [ ].
Nevertheless, the growth of mental health services available online provides a challenge for young people, because there is a wide range of choice that is difficult to navigate . Importantly, most young people self-refer to online services [ ], so it is important that they are directed to high-quality and evidence-based services [ ]. Service satisfaction feedback to measure young people’s experiences, the quality of services, and outcomes achieved is essential for service development, and to ensure that young people can access online services that meet their needs.
Service satisfaction is a broad term that can encompass a range of factors relating to care. Self-report feedback tools provide service providers with important information regarding clients’ progress . Regular feedback is recommended for all mental health practice, as clinicians are not always able to recognize treatment failure as it is happening [ , ]. However, there is a lack of formal feedback measures for child therapy, giving young clients “little voice in the services they receive” [ ]. Simmons et al [ ] emphasized the importance of satisfaction scales covering the relevant areas for youth mental health because much research to date in youth satisfaction has focused on parental or caregiver satisfaction [ ].
The few examples of youth-targeted service satisfaction measures include the Multidimensional Adolescent Satisfaction Scale  and the headspace Youth (mental health) Service Satisfaction Scale [ , ]. The Youth Service Satisfaction Scale is one of the few service satisfaction measures specifically developed for use in early intervention youth mental health services for young people aged 12 to 25 years. It has demonstrated good psychometric properties and revealed a high level of satisfaction with headspace center-based (face-to-face) services. To our knowledge no youth satisfaction questionnaires targeted toward online services exist; therefore, this study aimed to redress this gap.
This study involved the development and implementation of a service satisfaction questionnaire in an online youth mental health service—eheadspace—an online mental health support and counseling service for young Australians aged 12 to 25 years implemented as part of headspace, the National Youth Mental Health Foundation funded by the Australian Government’s Department of Health. The eheadspace service provides free and confidential access to mental health clinicians via Web chat, email, and phone from 9 am to 1 am AEST 7 days per week. This extends the reach of headspace centers, which provide face-to-face services through 110 centers across Australia. Like headspace centers, young people can access eheadspace for any reason, although the service is promoted as providing support and counseling. The eheadspace service began in 2011, in recognition of the high demand for youth mental health care that could not be met by face-to-face services alone. There has been a high level of uptake of the eheadspace service, and young people have been shown to access the service when they are feeling highly distressed . However, many young people only access once, and there has been little examination to date of young people’s experience of and their satisfaction with this service approach. Understanding user satisfaction is essential to avoid premature disengagement and to ensure that young people receive a service that they value and that meets their needs.
The aims of this study were to:
- Adapt the headspace center Youth Service Satisfaction Scale for use in eheadspace and examine its psychometric properties;
- Establish current levels of satisfaction among young people accessing eheadspace; and
- Determine whether satisfaction varies according to client characteristics or characteristics associated with the service contact.
Participants consisted of 2280 young people aged 12 to 25 years (mean 17.8, SD 3.2), who registered with eheadspace Web chat between September 20, 2016 and February 20, 2018, wanted help for themselves (rather than a friend), and completed at least one service satisfaction survey. Young people were included if they received a direct eheadspace service, which could be either (1) a brief service provision engaging the young person, identifying the presenting problems, and using brief therapeutic strategies to address these problems; or (2) an esupport-targeted strategy delivered following initial contact and engagement of the young person guided by a care plan and following a full assessment.
All young people registering for eheadspace Web chat during the survey period were invited to voluntarily participate. The survey response rate was 8.7%, from a total of 26,234 eligible young people who accessed the service during this period. The eheadspace users can complete a feedback survey each time they access the service; however, if users had completed more than one survey, only the first complete survey was analyzed in this paper.
Service satisfaction surveys were collected from eheadspace clients via an online questionnaire advertised and accessed through the eheadspace service platform. The platform provides other service options (eg, a link to the Digital Work and Study Service and support for family and friends), but only young people who used the eheadspace online service were offered the service satisfaction questionnaire. To access eheadspace online, a user clicks on a link to “online chat” and then provides required registration or log-in information. When a young person logged in and selected the online chat option, a blue box appeared on the side, asking: “Do you have a spare minute or two to complete our survey about your chat today? Thanks in advance if you do.” Young people were invited to click a link to access the questionnaire once they had finished their online chat, which subsequently directed them to the participant information page. Participation was voluntary.
Survey data were combined with demographic information, clinical characteristics, and service information that was extracted from the eheadspace administrative Minimum Data Set. All young people electronically enter data when they first log in to access the service, and each time they access the service and it has been 14 or more days since they last did so. All clinicians enter data at the end of each service session.
Ethics approval was obtained from Melbourne Health.
Demographic characteristics included gender (male, female, trans, intersex, and another gender [please specify]), age group (12-17 or 18-25 years), sexuality (heterosexual, lesbian, gay, bisexual, questioning, and other sexuality [LGBTIQ]), Aboriginal and/or Torres Strait Islander person (yes or no), and area of residence (metropolitan or regional). Note that gender was analyzed only as male or female due to the small sample size for the nonbinary categories.
Reason for attending was measured using clinician-rated responses to a list of options asking the main issue for which the young person presented. This was categorized into mental health or other issues. Other issues included situational issues, vocational assistance, alcohol or other drugs, physical health, sexual health and “other.”
Clinical stage was estimated for those presenting with mental health issues to provide a general indication of diagnostic complexity. Clinicians were asked to estimate an approximation of the clinical stage using responses from options based on the clinical staging heuristic . Stage 0 indicated none or few depressive or anxiety symptoms and little or no impairment in functioning; stage 1a indicated mild to moderate Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV) disorder of depression, generalized anxiety disorder (GAD), or impairment in functioning; stage 1b indicated stage 1a plus additional features falling short of stage 2; stage 2 indicated moderate to severe symptoms meeting threshold diagnosis; stage 3 indicated periods of remission (<6 months of symptom improvement) but no recovery; and stage 4 indicated ongoing severe symptoms with no asymptomatic periods. If the young person accessed eheadspace for a non-mental health-related issue, “not applicable” was selected; if the clinician was unable to adequately assess clinical stage, “not enough information available” was selected.
Service Contact Characteristics
Service characteristics included the type of direct service (brief service provision or esupport), wait time before an eheadspace session (the time between the user completing their registration and log-in for the online chat and the time the clinician responds, ranging from 0 to 106 minutes), session start time (9 am to 1 am), session duration (<15 minutes to ≥90 minutes), number of words conveyed by the young person (0 to 2599 words), number of words conveyed by the clinician (0 to 3583 words), and the number of times the young person had previously received a direct service (0 to 2 or more sessions). Young people were also asked if they had visited a headspace center prior to using eheadspace (yes or no) and referral information (self-referred or referred by other).
To develop a measure of eheadspace service satisfaction, relevant items were adapted from the headspace center Youth Service Satisfaction Scale  and supplemented by items from a review of existing satisfaction surveys and consultation with eheadspace staff and clients. This resulted in 13 items comprising statements that were each responded to on a 5-point Likert scale from strongly disagree to strongly agree, with higher scores indicating higher levels of satisfaction. The items focused on three different satisfaction domains:
- Session satisfaction: session focused on young person’s main concern, young person felt listened to and understood, session helped the young person to understand their situation more clearly, young person was provided with skills or resources to help them manage their situation going forward, session made young person feel more hopeful and optimistic.
- Potential outcomes: young person feels better day-to-day, can manage the things they do better, copes better, has improved relationships.
- Service satisfaction: young person feels comfortable sharing information with eheadspace clinicians, eheadspace is easy to use, young person would recommend eheadspace to a friend, eheadspace met the young person’s expectations.
Data were analyzed using IBM SPSS version 24 and AMOS version 25. An exploratory principal components factor analysis of the 13 original eheadspace feedback survey items was first conducted. Confirmatory factor analysis (CFA) with maximum-likelihood estimation was then used to further test the factor structure. Cronbach reliability analysis was conducted to confirm internal consistency.
Descriptive statistics were computed for all variables. Independent t tests and one-way analyses of variance were then used to determine group differences in satisfaction scores based on client characteristics (gender, age group, sexuality, Aboriginal and/or Torres Strait Islander status, areas of residence, reason for attending, clinical stage), and service contact characteristics (type of service, wait time before session, session start time, duration, prior access to a headspace center, number of previous direct service sessions, word count per client and clinician, referral information). Significance was set at P<.01 due to the high power from the large sample size.
Client and Service Contact Characteristics
Of the 2280 participants, 77.76% (n=1773) were female, and 55.04% (n=1255) were aged between 12 and 17 years, whereas 44.96% (n=1025) were between 18 and 25 years. There were 64.96% (1481/2280) who identified as heterosexual, 70.44% (1606/2280) who lived in a metropolitan area, and 3.77% (86/2280) who identified as an Aboriginal and/or Torres Strait Islander person.
For presenting issue, 58.82% (1341/2280) contacted eheadspace for mental health problems and 26.32% (600/2280) contacted for another issue (14.87%, 339/2280 missing). Of those with a clinical stage rating (ie, excluding “not applicable” and “not enough information available”), more than half (54.38%, 522/960) were rated stage 1a indicating mild to moderate DSM-IV disorder of depression, GAD, or impairment in functioning.
The majority of clients (85.13%, 1941/2280) had an esupport session, whereas 14.87% (339/2280) had a brief service provision session on the day of completing the survey. Half (50.39%, 1149/2280) the sessions lasted 31 to 60 minutes, but sessions ranged from less than 15 minutes to 90 or more minutes. More than half (57.96%, 1125/1941) of esupport sessions lasted 31 to 60 minutes, whereas only 7.08% (24/339) of brief service provisions lasted this long. Only 5.04% (115/2280) did not have to wait to access eheadspace when they visited, 41.49% (946/2280) had to wait less than 5 minutes, and 53.46% (1219/2280) had to wait more than 5 minutes (maximum wait time was 106 minutes). Most (83.29%, 1899/2280) sessions occurred before 11 pm, but 16.71% occurred between 11 pm to 1 am. On average, young people inputted a mean 397.01 (SD 290.05) words, compared to a mean 686.18 (SD 343.25) words from clinicians. The majority of young people (71.84%, 1638/2280) input fewer than 500 words per session, compared with only 30.04% of clinicians (685/2280). Most young people (67.72%, 1544/2280) self-referred to eheadspace, with 30.35% (692/2280) referred by others. Only 21.80% (497/2280) of participants had visited a headspace center prior to accessing eheadspace. Before the session on the day of survey completion, 64.04% (1460/2280) had not previously received esupport, 16.49% (376/2280) had previously received one, 7.54% (172/2280) had previously received two, and 11.84% (270/2280) had previously received three or more esupport sessions.
Factor Analysis of Satisfaction Scale
The 13 original items of the eheadspace feedback survey were subjected to principal components analysis (PCA) to identify the factorial structure. Inspection of the correlation matrix revealed all coefficients above 0.30. The Kaiser-Meyer-Olkin value observed was 0.95, which exceeded the recommended value of 0.60 , and the Bartlett test of sphericity [ ] was statistically significant, supporting the factorability of the correlation matrix.
The PCA revealed the presence of one component with an eigenvalue of 8.3, explaining 63.8% of the variance. Standardized factor loadings for 12 of 13 items were strong and positive (range 0.64-0.87), except for “eheadspace is easy to use,” which was 0.50. The eigenvalues and factor loadings supported a unidimensional solution, and an inspection of the scree plot revealed a clear break after the first factor .
A CFA was run to further test the factor structure (seeand ). The following fit indexes were employed to assess model fit: chi-square closest to zero, root mean square error of approximation less than 0.06 (with the lower bound of its 90% confidence interval less than 0.05 to indicate close fit), the comparative fit index greater than 0.90, and adjusted goodness of fit index greater than 0.90 [ ]. Based on the recommendation of Holmes-Smith et al [ ], the model with the smallest Akaike information criterion (AIC) / consistent AIC (CAIC) was considered the best-fitting model.
Examination of the modification indexes revealed that the item “the session made me feel more hopeful/optimistic” may relate to both the session and expectations subscales. Only by deleting this item altogether did the model fit improve.
Models were rerun with the remaining 12 items. Results revealed that the original one-factor solution was not a good fit for the data. Consequently, the originally hypothesized three-factor solution was considered, along with other possibilities. Examination of the AIC and CAIC indexes showed that a three-factor model was the best fit to the data, but that a three-factor model with a higher order unidimensional construct of overall satisfaction was also a reasonable fit.
Note that the potential outcomes items included a “not applicable” option, as eheadspace clinicians felt that each of these outcomes may not be applicable for some clients. Consequently, to include participant responses to these items for the CFA (which cannot accept missing data), where only some items were not applicable, responses were substituted as neither agree nor disagree. Those who responded not applicable to all potential outcome items were excluded from analyses. Models were run with and without the substituted not applicable responses included, and all were a similar fit as shown in. Including those who chose not applicable to some questions, but still completed the full survey, reduces bias.
|Model||χ2 (df)||P value||RMSEAa||AGFIb||CFIc||AICd||CAICe|
|One factor||3418.7 (56)||<.001||0.180||0.591||0.821||3462.7||3606.3|
|Two factor||2008.7 (54)||<.001||0.140||0.723||0.896||2056.7||2213.3|
|Three factor without total||533.6 (51)||<.001||0.071||0.926||0.974||533.6||763.8|
|Three factor without N/Af responses||880.9 (51)||<.001||0.094||0.884||0.949||934.9||1111.1|
|Final three-factor solution||787.8 (53)||<.001||0.086||0.897||0.961||837.8||1001.0|
aRMSEA: root mean square error of approximation.
bAGFI: adjusted goodness of fit index.
cCFI: comparative fit index.
dAIC: Akaike information criterion.
eCAIC: consistent Akaike information criterion.
fN/A: not applicable.
Reliability of Satisfaction Scale
Cronbach alpha statistics showed high internal consistency for the total scale and all subscales (see). If any individual items were removed the internal reliability of the scales would be reduced. Item-total correlations were calculated, and all subscales showed strong interitem correlations: session satisfaction items (r=.67-.82), potential outcomes items (r=.70-.80), and service satisfaction items (r=.66-.82). Furthermore, session, outcomes, and service items were moderately to strongly correlated (r=.34-.82). This further supports a second-order factor of total satisfaction.
shows that overall young people were satisfied with eheadspace; all subscale means were above the scale midpoint and negatively skewed. Young people were most satisfied with the eheadspace service followed by their session, and they were least satisfied with their potential outcomes.
Client characteristics were not shown to influence total satisfaction scores (see). There were no significant differences in total satisfaction between males and females, adolescents and young adults, those who were heterosexual and those who were LGBTIQ, those who were an Aboriginal and/or Torres Strait Islander person and those who were not, and between young people who lived in metropolitan or regional locations. Those who attended because of a mental health issue were significantly less satisfied than those who attended for other reasons, although the effect was small. There was no effect of clinical stage.
Many service contact characteristics were significantly associated with satisfaction.shows that young people who received esupport were more satisfied than those who received a brief service provision. Those who accessed prior to 11 pm were more satisfied than those who accessed late at night. Young people with contact of more than 30 minutes were more satisfied than those with shorter sessions. Clients who conveyed more than 500 words were more satisfied than those who conveyed 300 or less. Similarly, greater satisfaction was evident when clinicians conveyed more than 500 words. Young people who had not previously accessed a headspace center were more satisfied with eheadspace than those who had previously been to a headspace center. Wait time before session, previous eheadspace access, and referral source did not influence total satisfaction.
|Satisfaction domain||Mean (SD)||Skewness||SE||Cronbach alpha|
|Session satisfaction||3.77 (1.07)||–1.085||0.051||.93|
|Potential outcomes||3.16 (0.86)||–0.604||0.051||.92|
|Service satisfaction||3.88 (0.91)||–0.320||0.051||.91|
|Total satisfaction||3.60 (0.83)||–0.918||0.051||.95|
|Client characteristic||Satisfaction, mean (SD)||F (df1,df2)||P value||Partial η2|
|12-17 years||3.64 (0.79)|
|18-25 years||3.56 (0.88)|
|Aboriginal and/or Torres Strait Islander person||1.61 (2278,91)||.11||.032|
|Area of residence||–0.02 (2251,1262)||.98||.000|
|Reason for attending||–2.89 (1939,1208)||.004||–.062|
|Mental health issue||3.67 (0.76)|
|Clinical stage estimateb||1.80 (3,956)||.15||.006|
|Stage 0||3.72 (0.76)|
|Stage 1a||3.70 (0.70)|
|Stage 1b||3.61 (0.77)|
|Stage 2||3.51 (0.79)|
aLGBTIQ: Lesbian, Gay, Bisexual, Transgender, Intersex and Questioning.
bNote that no participants were rated in stages 3 or 4.
|Service characteristic||Satisfaction, mean (SD)||F (df1,df2)||P value||Partial η2|
|Type of service||11.40 (403,2278)||<.001||.282|
|Brief service||3.04 (1.03)|
|Session start time||17.87 (3,2276)||<.001||.023|
|9 am-5 pm||3.66 (0.78)|
|5 pm-8 pm||3.66 (0.81)|
|8 pm-11 pm||3.66 (0.81)|
|11 pm-1 am||3.32 (0.94)|
|Wait time before session||0.62 (3,2276)||.61||.001|
|<5 minutes||3.63 (0.83)|
|<10 minutes||3.59 (0.80)|
|≥10 minutes||3.58 (0.83)|
|Session duration||9.37 (3,1937)||<.001||.014|
|<15 minutes||3.52 (0.88)|
|15-30 minutes||3.57 (0.82)|
|31-60 minutes||3.73 (0.72)|
|>60 minutes||3.84 (0.68)|
|Word count per client||54.23 (3,2276)||<.001||.067|
|Word count per clinician||73.09 (3,2276)||<.001||.088|
|Number of previous direct support sessions||0.13 (3,2274)||.94||.000|
|Referral information||0.11 (1414,2234)||.92||.000|
|Referred by other||3.60 (0.79)|
|Prior access to a headspace center||–2.91 (790,1986)||.004||–.068|
This study examined the implementation of a service satisfaction measure adapted from the headspace center-based satisfaction measure in the online youth mental health service, eheadspace. The aim was to determine the psychometric properties of the adapted measure in the online service environment and establish the level of service satisfaction among young people using the service, while examining whether this varied by client characteristics.
Psychometric Properties of the Satisfaction Scale
The satisfaction measure showed good psychometric properties. There was high internal consistency for the overall satisfaction scale and its three subscales. A three-factor model of the eheadspace satisfaction feedback survey was shown to be the best fit to the data and the hypothesized subscales of session satisfaction, potential outcomes, and service satisfaction were evident. A three-factor model with a higher order unidimensional construct of overall satisfaction was also a reasonable fit. A higher order unidimensional overall satisfaction construct is supported by the literature. For example, Simmons et al  found support for a global construct of satisfaction in the development of a satisfaction scale for young people attending headspace centers.
Consistent with other service satisfaction research and literature, the measures of service satisfaction were negatively skewed, with most respondents indicating high satisfaction. The measure can help identify the characteristics of clients who are dissatisfied with the service, but it may be less useful in distinguishing among those with positive experiences [, ]. Consequently, it may be difficult to demonstrate “improved satisfaction” when scores are already so high.
Current Satisfaction of Young People Accessing eheadspace
Young people were generally very satisfied with eheadspace, and this did not vary across any demographic characteristics. No demographic differences were evident in this study unlike the findings of Rickwood et al  from face-to-face headspace center services, which found that men, younger clients, and those influenced by others to attend were less satisfied. This suggests that the eheadspace service is experienced in a similarly positive way across the demographics measured. However, the demographics of eheadspace clients are quite different from those of the headspace centers, as they are much more likely to be female and present with a high level of psychological distress [ ]. This was reflected in that almost 78% of our participants were female, similar to the 80% who are female who access the service. Although the young men who accessed the service were equally satisfied, they were much less likely to access in the first place. The few client characteristics that were measured also did not substantially affect satisfaction. Those who attended because of a mental health issue were less satisfied than those who attended for other reasons, although the effect was small. It did not vary according to the estimate of clinical stage, which suggests that clients were equally highly satisfied regardless of the complexity of their issue. However, it should be noted that the measure of reason for attending was broad, comprising only mental health versus other issues, and the estimate of clinical stage was not obtained via a comprehensive clinical interview. It may be that satisfaction varies according to different types of mental health problems, as the online environment may be better suited to some than others and may be most useful as an adjunct to face-to-face therapy for some more serious and complex conditions.
Service contact characteristics were associated with satisfaction. Overall, young people who had a more comprehensive engagement with eheadspace were more satisfied. They were more satisfied when they received the more substantial esupport service rather than brief service provision, had a longer session, used more words, did not access the service very late at night, and had not previously experienced services from a face-to-face headspace center. This is consistent with research showing that eheadspace clients who had greater engagement showed more progress and depth of counseling and greater alleviation of psychological distress .
Similar to the findings regarding satisfaction from face-to-face headspace center services, satisfaction was lowest for perceived outcomes . Many young people only access eheadspace once, so expectations of longer-term outcomes are unlikely to be well-formed. It is possible that many people might not know how to answer, especially if it is one of their first visits or their only visit, and if they only received information and not therapy as part of their session. Further, earlier research revealed that eheadspace clients have high service expectations [ ], which are unlikely to be met with the minimal engagement that is typical.
The results of this study should be interpreted in light of its limitations. Notably, providing service satisfaction was optional, and the response rate was less than 10%. Although this was expected, it indicates likely response bias. It may be that only the more satisfied clients chose to complete the questionnaire. Finding ways to encourage more clients to complete the measure, and ensuring that those who are dissatisfied and who have less service engagement respond, are challenges for service satisfaction measures. Other indicators of service satisfaction should also be monitored, such as the number of site visits and drop-off rates.
Some of the measures were limited by the brief online nature of the data collection. In particular, the measure of clinical staging was an estimate. This measure was included to gain some understanding of the potential impact of client complexity, but it is acknowledged that a brief, online minimum data collection procedure cannot provide a robust measure of clinical stage such as can be ascertained through clinical interview within mental health services that adopt a transdiagnostic clinical staging approach (see ).
Implications for Practice
The satisfaction results are regularly monitored and analyzed by the eheadspace service and evaluation staff and are used to guide service improvement. By identifying patterns in client satisfaction, the areas where the service needs to be tailored to better meet different client needs can be identified. Patterns over time can also be used to evaluate the impact of changes in practice.
Note that currently the satisfaction results are only analyzed in aggregate and not linked to individual clients. There is the potential to also examine session satisfaction monitored over time for individual clients through routine outcome monitoring. This is a repeated measurement process carried out therapeutically to improve treatment outcomes, which has been shown to improve treatment engagement and outcomes . An even briefer measure may be required, however, to implement repeatedly, and clients would need to be informed that their individual feedback would be available to their service providers.
The satisfaction measure used in eheadspace solely assessed youth satisfaction as a form of feedback. Much youth mental health research has previously focused on parental and caregiver satisfaction . Parents and caregivers often have different views about service satisfaction [ ]. Their views are also important in assessing online youth mental health services, although this may be less relevant for online services where self-referral is predominant and young people can easily access without parental support. Nevertheless, parents and caregivers often play a primary role in whether younger people attend, engage with, and complete treatment, demonstrating the importance of their satisfaction [ ]. Patel et al [ ] suggested that both young person and parental feedback is essential in assessing satisfaction with mental health services and this should be explored in the online service domain.
Finally, the measure was adapted from the headspace center Youth Service Satisfaction Scale . This scale has been successfully implemented in the face-to-face services, and it was of interest to have a comparative measure for the online service. Given the briefer and more transitory nature of the online service interactions, further adaptations may be required to make the measure most suitable to the online modality.
This paper has described a service satisfaction measure for an online mental health service for young people. The eheadspace satisfaction measure is brief, and it demonstrated sound psychometric properties with good internal consistency and a clear factor structure. The measure showed high levels of satisfaction overall that were responsive to greater levels of engagement with the service. The measure could be adapted for use with other online youth mental health services, which are emerging as a critical setting for mental health service provision as part of stepped-care approaches in mental health .
The headspace National Youth Mental Health Foundation is funded by the Australian Government Department of Health. The authors would like to thank the young people who contributed data for this paper.
Conflicts of Interest
All authors are employed by or directly involved with headspace National Youth Mental Health Foundation.
- Patel V, Flisher AJ, Hetrick S, McGorry P. Mental health of young people: a global public-health challenge. Lancet 2007 Apr 14;369(9569):1302-1313. [CrossRef] [Medline]
- Burgess PM, Pirkis JE, Slade TN, Johnston AK, Meadows GN, Gunn JM. Service use for mental health problems: findings from the 2007 National Survey of Mental Health and Wellbeing. Aust N Z J Psychiatry 2009 Jul;43(7):615-623. [CrossRef] [Medline]
- Australian Bureau of Statistics. 2007. National Survey of Mental Health and Wellbeing: Summary of Results URL: http://www.abs.gov.au/ausstats/abs@.nsf/Latestproducts/4326.0Main%20Features32007 [accessed 2018-12-13] [WebCite Cache]
- Burns J, Birrell E. Enhancing early engagement with mental health services by young people. Psychol Res Behav Manag 2014;7:303-312 [FREE Full text] [CrossRef] [Medline]
- Kirby M. Mental health in Canada: out of the shadows forever. CMAJ 2008 May 06;178(10):1320-1322 [FREE Full text] [CrossRef] [Medline]
- Rickwood DJ, Deane FP, Wilson CJ. When and how do young people seek professional help for mental health problems? Med J Aust 2007 Oct 1;187(7 Suppl):S35-S39. [Medline]
- Jenkins R, Baingana F, Ahmad R, McDaid D, Atun R. Social, economic, human rights and political challenges to global mental health. Ment Health Fam Med 2011 Jun;8(2):87-96 [FREE Full text] [Medline]
- Kim-Cohen J, Caspi A, Moffitt TE, Harrington H, Milne BJ, Poulton R. Prior juvenile diagnoses in adults with mental disorder: developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry 2003 Jul;60(7):709-717. [CrossRef] [Medline]
- Slade T, Johnston A, Teesson M, Whiteford H, Burgess P, Pirkis J, et al. Australian Government Department of Health. 2009. The Mental Health of Australians 2: Report on the 2007 National Survey of Mental Health and Wellbeing URL: http://www.health.gov.au/internet/main/publishing.nsf/content/mental-pubs-m-mhaust2 [WebCite Cache]
- Roxon N, Macklin J, Butler M. Australian Government. 2011. National Mental Health Reform-Ministerial Statement URL: https://www.budget.gov.au/2011-12/content/ministerial_statements/health/download/ms_health.pdf [accessed 2018-12-13] [WebCite Cache]
- Rickwood D. Entering the e-spectrum: An examination of new interventions for youth mental health. Youth Stud Aust 2012;31(4):18-27.
- Andersson G, Titov N. Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry 2014 Feb;13(1):4-11 [FREE Full text] [CrossRef] [Medline]
- Ivancic L, Perrens B, Fildes J, Perry Y, Christensen H. Youth Mental Health Report. Sydney: Mission Australia and Black Dog Institute; 2014. URL: https://www.google.com/search?q=Ivancic%2C+L.%2C+et+al.%2C+Youth+mental+health+report.+Mission+Australia+and+Black+Dog+Institute%2C+2014.&rlz=1C1GCEA_enAU821AU821&oq=Ivancic%2C+L.%2C+et+al.%2C+Youth+mental+health+report.+Mission+Australia+and+Black+Dog+Institute%2C+2014.&aqs=chrome..69i57.1044j0j4&sourceid=chrome&ie=UTF-8 [WebCite Cache]
- Australian Bureau of Statistics. 2015. Household use of information technology URL: http://www.abs.gov.au/AUSSTATS/abs@.nsf/Lookup/8146.0Explanatory%20Notes12014-15?OpenDocument [accessed 2018-12-13] [WebCite Cache]
- Burns JM, Davenport TA, Christensen H, Luscombe GM, Mendoza JA, Bresnan A, et al. Game On: Exploring the Impact of Technologies on Young Men’s Mental Health and Wellbeing. Findings From the First Young and Well National Survey. Melbourne: Young and Well Cooperative Research Centre; 2013. URL: https://cdn.movember.com/uploads/files/Our%20Work/game-on-movember-foundation.pdf [accessed 2018-12-13] [WebCite Cache]
- Griffiths KM, Christensen H. Internet-based mental health programs: a powerful tool in the rural medical kit. Aust J Rural Health 2007 Apr;15(2):81-87. [CrossRef] [Medline]
- Griffiths KM, Farrer L, Christensen H. The efficacy of internet interventions for depression and anxiety disorders: a review of randomised controlled trials. Med J Aust 2010 Jun 7;192(11 Suppl):S4-S11. [Medline]
- Burns JM, Davenport TA, Durkin LA, Luscombe GM, Hickie IB. The internet as a setting for mental health service utilisation by young people. Med J Aust 2010 Jun 7;192(11 Suppl):S22-S26. [Medline]
- Oh E, Jorm AF, Wright A. Perceived helpfulness of websites for mental health information: a national survey of young Australians. Soc Psychiatry Psychiatr Epidemiol 2009 Apr;44(4):293-299. [CrossRef] [Medline]
- Australian Government Department of Health. PHN Primary Mental Health Care Flexible Funding Pool Implementation Guidance: Stepped Care. Canberra: Commonwealth of Australia; 2016. URL: http://www.health.gov.au/internet/main/publishing.nsf/content/2126B045A8DA90FDCA257F6500018260/$File/1PHN%20Guidance%20-%20Stepped%20Care.PDF [accessed 2019-03-03] [WebCite Cache]
- Cross SP, Hickie I. Transdiagnostic stepped care in mental health. Public Health Res Pract 2017 Apr 27;27(2):1 [FREE Full text] [CrossRef] [Medline]
- Fletcher K, Foley F, Murray G. Web-based self-management programs for bipolar disorder: insights from the online, recovery-oriented bipolar individualised tool project. J Med Internet Res 2018 Oct 24;20(10):e11160 [FREE Full text] [CrossRef] [Medline]
- Collin PJ, Metcalf AT, Stephens-Reicher JC, Blanchard ME, Herrman HE, Rahilly K, et al. ReachOut.com: the role of an online service for promoting help-seeking in young people. Advances in Mental Health 2014 Dec 17;10(1):39-51. [CrossRef]
- Santor DA, Poulin C, LeBlanc JC, Kusumakar V. Online health promotion, early identification of difficulties, and help seeking in young people. J Am Acad Child Adolesc Psychiatry 2007 Jan;46(1):50-59. [CrossRef] [Medline]
- Rickwood D, Webb M, Kennedy V, Telford N. Who are the young people choosing web-based mental health support? Findings from the implementation of Australia's national web-based youth mental health service, eheadspace. JMIR Ment Health 2016 Aug 25;3(3):e40 [FREE Full text] [CrossRef] [Medline]
- Rickwood DJ. Promoting youth mental health through computer-mediated communication. Int J Ment Health Pr 2012 Jan 30;12(3):32-44. [CrossRef]
- Kornør H, Bergman H, Hanssen-Bauer K, Soares-Weiser K, Tollefsen T, Bjørndal A. Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database Syst Rev 2015;6:1-14. [CrossRef]
- Weisz JR, Ng MY, Bearman SK. Odd couple? Reenvisioning the relation between science and practice in the dissemination-implementation era. Clin Psychol Sci 2013 Dec 11;2(1):58-74. [CrossRef]
- Lambert MJ, Whipple JL, Hawkins EJ, Vermeersch DA, Nielsen SL, Smart DW. Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clin Psychol Sci Pract 2003;10(3):288-301. [CrossRef]
- Duncan B, Sparks J, Miller S, Bohanske RT, Claud DA. Giving youth a voice: a preliminary study of the reliability and validity of a brief outcome measure for children, adolescents, and caretakers. J Brief Ther 2006;5(2):71-88.
- Simmons MB, Parker AG, Hetrick SE, Telford N, Bailey A, Rickwood D. Development of a satisfaction scale for young people attending youth mental health services. Early Interv Psychiatry 2014 Nov;8(4):382-386. [CrossRef] [Medline]
- Biering P. Child and adolescent experience of and satisfaction with psychiatric care: a critical review of the research literature. J Psychiatr Ment Health Nurs 2010 Feb;17(1):65-72. [CrossRef] [Medline]
- Garland A, Saltzman M, Aarons G. Adolescent satisfaction with mental health services: development of a multidimensional scale. Eval Program Plann 2000;23(2):165-175 [FREE Full text]
- Rickwood D, Nicholas A, Mazzer K, Telford N, Parker A, Tanti C, et al. Satisfaction with youth mental health services: further scale development and findings from headspace - Australia's National Youth Mental Health Foundation. Early Interv Psychiatry 2017 Dec;11(4):296-305. [CrossRef] [Medline]
- McGorry PD, Purcell R, Hickie IB, Yung AR, Pantelis C, Jackson HJ. Clinical staging: a heuristic model for psychiatry and youth mental health. Med J Aust 2007 Oct 1;187(7 Suppl):S40-S42. [Medline]
- Kaiser HF. An index of factorial simplicity. Psychometrika 1974 Mar;39(1):31-36. [CrossRef]
- Bartlett M. A Note on the Multiplying Factors for Various χ2 Approximations. Journal of the Royal Statistical Society: Series B (Methodological) 2018 Dec 05;16(2):296-298. [CrossRef]
- Cattell RB. The Scree test for the number of factors. Multivar Behav Res 1966 Apr;1(2):245-276. [CrossRef]
- Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Modeling 1999 Jan;6(1):1-55. [CrossRef]
- Holmes-Smith P, Coote L, Cunningham E. Structural Equation Modeling: From the Fundamentals to Advanced Topics. Melbourne: School Research, Evaluation and Measurement Services; 2006.
- Athay MM, Bickman L. Development and psychometric evaluation of the youth and caregiver Service Satisfaction Scale. Adm Policy Ment Health 2012 Mar;39(1-2):71-77 [FREE Full text] [CrossRef] [Medline]
- Brannan A, Sonnichsen S, Heflinger C. Measuring satisfaction with children's mental health services: validity and reliability of the satisfaction scales. Eval Program Plann 1996;19(2):131-141 [FREE Full text]
- Dowling M, Rickwood D. Investigating individual online synchronous chat counselling processes and treatment outcomes for young people. Adv Ment Health 2015 Jan 30;12(3):216-224. [CrossRef]
- Dowling M, Rickwood D. A naturalistic study of the effects of synchronous online chat counselling on young people's psychological distress, life satisfaction and hope. Couns Psychother Res 2015 Jul 14;15(4):274-283. [CrossRef]
- Hart A, Saunders A, Thomas H. Attuned practice: a service user study of specialist child and adolescent mental health, UK. Epidemiol Psichiatr Soc 2005;14(1):22-31. [Medline]
- McNaughton D. Measuring parent satisfaction with early childhood intervention programs. Top Early Child Spec Ed 2016 Aug 17;14(1):26-48. [CrossRef]
|AGFI: adjusted goodness of fit index|
|AIC: Akaike information criterion|
|CAIC: consistent Akaike information criterion|
|CFA: confirmatory factor analysis|
|CFI: comparative fit index|
|DSM-IV: Diagnostic and Statistical Manual of Mental Disorders Version 4|
|GAD: generalized anxiety disorder|
|LGBTIQ: Lesbian, Gay, Bisexual, Transgender, Intersex and Questioning|
|PCA: principal components analysis|
|RMSEA: root mean square error of approximation|
Edited by J Torous; submitted 24.09.18; peer-reviewed by C Edridge, O Santesteban-Echarri; comments to author 22.11.18; revised version received 17.12.18; accepted 19.12.18; published 17.04.19
©Debra Rickwood, Alison Wallace, Vanessa Kennedy, Shaunagh O’Sullivan, Nic Telford, Steven Leicester. Originally published in JMIR Mental Health (http://mental.jmir.org), 17.04.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.