This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
Digital interventions have the potential to serve as cost-effective ways to manage occupational stress and well-being. However, little is known about the adoption of individual-level digital interventions at organizations.
The aim of this paper is to study the effects of an unguided digital mental health intervention in occupational well-being and the factors that influence the adoption of the intervention.
The intervention was based on acceptance and commitment therapy (ACT) and its aim was to teach skills for stress management and mental well-being. It was delivered via a mobile and a Web-based app that were offered to employees of two information and communication technology (ICT) companies. The primary outcome measures were perceived stress and work engagement, measured by a 1-item stress questionnaire (Stress) and the Utrecht Work Engagement Scale (UWES-9). The intervention process was evaluated regarding the change mechanisms and intervention stages using mixed methods. The initial interviews were conducted face-to-face with human resource managers (n=2) of both companies in August 2013. The participants were recruited via information sessions and email invitations. The intervention period took place between November 2013 and March 2014. The participants were asked to complete online questionnaires at baseline, two months, and four months after the baseline measurement. The final phone interviews for the volunteer participants (n=17) and the human resource managers (n=2) were conducted in April to May 2014, five months after the baseline.
Of all the employees, only 27 (8.1%, 27/332) took the app into use, with a mean use of 4.8 (SD 4.7) different days. In the beginning, well-being was on good level in both companies and no significant changes in well-being were observed. The activities of the intervention process failed to integrate the intervention into everyday activities at the workplace. Those who took the app into use experienced many benefits such as relief in stressful situations. The app was perceived as a toolkit for personal well-being that gives concrete instructions on how mindfulness can be practiced. However, many barriers to participate in the intervention were identified at the individual level, such as lack of time, lack of perceived need, and lack of perceived benefits.
The findings suggest that neither the setting nor the approach used in this study were successful in adopting new digital interventions at the target organizations. Barriers were faced at both the organizational as well as the individual level. At the organizational level, top management needs to be involved in the intervention planning for fitting into the organization policies, the existing technology infrastructure, and also targeting the organizational goals. At the individual level, concretizing the benefits of the preventive intervention and arranging time for app use at the workplace are likely to increase adoption.
Prevention of mental health problems is a topical issue. Psychosocial stress is a risk factor for mental health problems [
Wider adoption of workplace wellness programs could prove beneficial for financial and productivity outcomes as well as health outcomes [
Work stress interventions utilizing acceptance and commitment therapy (ACT) have been shown to reduce stress and increase well-being and job performance [
Recently, digital interventions have been presented as more cost-effective and scalable means to promote health and well-being compared to face-to-face interventions [
Adoption processes of digital interventions at organizations have received little attention. In the context of this paper, the term adoption refers to a process that ends in the appropriate and effective use of a technology or digital service. In other studies the term operationalization has been used instead, referring to the actual introduction, adoption, and employment of the technology in practice, including also training and education [
Adoption of digital interventions should not be considered only from the organization’s point of view, but also from the individual’s point of view. Employees’ adherence, including the extent to which individuals experience the content of the intervention [
The aim of this paper is to evaluate the adoption process and effects of an individual-level unguided digital mental health intervention at organizations. In addition, it explores users’ motivations and experiences related to the mobile and Web app used to deliver the intervention. An earlier pilot study showed that the app can be acceptable, useful, and engaging among stressed working-age adults [
Nielsen and Abildgaard’s intervention evaluation framework was used for intervention process and effects evaluation [
Intervention evaluation framework [
The intervention took place in two companies (Company A and Company B) working in the information and communication technology (ICT) industry (
Description of the companies.
|
Company A | Company B |
Number of employees | 230 in Finland | 102 in Finland, over 4000 worldwide |
Offices | 2 locations | 2 locations |
Age of employees | 35 years on average | 30-40 years on average |
Gender | 83% men | 28% men |
Most common job titles | Software designer, consultant, project manager | Technical writer, consultant, project manager |
Amount of remote work | Occasional remote work and travel | Technical writers often work at customers’ premises and consultants travel often |
Amount of sick leaves | 3-4% of working days, length of a sick leave usually 1-3 days | 5-6% of working days, also some long sick leaves |
Top 3 reasons for sick leaves | Common cold, stomach flu, musculoskeletal problems | Common cold, musculoskeletal problems, mental problems |
Occupational health care | All employees are within the occupational health care | All employees are within the occupational health care |
Work well-being assessments | Annual survey | Annual survey |
The intervention content was delivered through a digital training app which was available in a mobile (iOS and Android) and a Web-based version (
Screenshot of the mobile app (texts translated from Finnish).
Screenshot of the web app (texts translated from Finnish).
The participants of the study were the human resource (HR) managers and employees of the two ICT companies. The employees of both companies with sufficient computer literacy and fluency in Finnish were eligible to participate in the study. The HR managers of the companies invited the employees via email messages to an initial info session, which was held by researchers. The session consisted of a 20-minute presentation about the ACT approach, the app, and the aims and procedures of the study. In Company A, two info sessions were held at the two company offices. In Company B, the info session was arranged as an online meeting. In addition, the HR managers put information about the study in the company intranet and distributed paper flyers at the company offices. In all invitations and recruitment materials, the study was framed as a positive opportunity to improve one’s well-being and learn new skills to manage stress. The participants were instructed to use the digital app regularly, in brief sessions several times per week, but the intervention was otherwise unguided.
The intervention was designed to follow the stages outlined in
Intervention process design based on Nielsen and Abildgaard’s framework [
Stage | Tasks | Program design |
Initiation | Clarify roles of different actors; create a communication strategy | First contact with companies through their HR managers |
Screening | Assess organizations’ needs; select measures on individual and organizational level | Interviews with HR managers at the companies to recognize organization-specific needs and context; selection of suitable measures that complement existing assessment methods in the companies, with emphasis on measures focused on strengths and skills rather than problems and weaknesses |
Action plans | Clarify intervention activities, their purpose and timeline; select methods for evaluating success of actions | App available on Web and mobile platforms; baseline survey conducted online; approximate intervention duration 3 months; kick-off events for employees held by researchers on-site and/or through online meetings; email and calendar invitations, intranet announcements and flyers at worksite; if possible, events aligned with other events/trainings in the company |
Implementation | Document intervention activities; assign person who makes intervention happen in the organization | Regular contact with HR managers to keep track of the progress and activities inside the companies; mid-survey (online) to assess initial engagement and experiences among employees |
Evaluation of effects | Measure changes in health and well-being; measure changes in working conditions and organizational procedures | Follow-up survey (online); analysis of changes in different well-being measures from baseline to follow-up; interviews with volunteer employees, superiors and HR managers |
The study and all the related questionnaires were reviewed by the Ethics Committee of the Pirkanmaa Hospital District. Participants’ informed consent was obtained in the beginning of the baseline questionnaire online. The data was stored on a secure server. All the data was anonymized for reporting and publication.
Data was collected between August 2013 and May 2014. The study began in August 2013 with HR managers’ interviews, and the intervention took place between November 2013 and March 2014. The study ended with the final interviews with the participants in April to May 2014.
Electronic questionnaires were sent via LimeSurvey (version 2.00) in the beginning of the study for all employees, and at two and four months for those employees who completed the baseline questionnaire. Occupational health and well-being was assessed in the beginning and in the end using a 1-item Stress questionnaire (Stress) [
Secondary outcome measures were an 84-item work-related well-being questionnaire which assessed both personal and work community well-being (P-TyHy, TyHy) [
Details of the secondary outcome measure questionnaires and scales.
Name | Score range | Significance of higher scores | Internal consistency | Cronbach's alpha range |
Work-related well-being questionnaire (P-TyHy, TyHy) | 0-70 | Better occupational health and well-being | N/A | N/A |
Satisfaction With Life Scale (SWLS) | 5-35 | Higher satisfaction with life | Good | .79-.89 |
Mindfulness questionnaire (FMI) | 14-56 | Higher mindfulness skills | Good | .86-.93 |
Work-Related Acceptance and Action Questionnaire (WAAQ) | 0-42 | Higher psychological flexibility | Good | .81-.84 |
Self-Regulation Questionnaires (SRQ) | 1-7 | Higher motivation | N/A | N/A |
Qualitative data consisted of several semi-structured interviews. Before the intervention, HR managers (n=2) were interviewed separately, face-to-face for the companies’ characteristics, needs, and well-being challenges. In addition, the schedule for the intervention activities was outlined together with them. After the intervention, the HR managers were interviewed for their views on the success of the intervention, attitudes toward occupational health and mobile coaching in an organizational context, as well as intervention adoption in organizations. It should be noted that the HR manager of Company A changed during the course of the study.
After the intervention, a voluntary subset of employees (n=17) were interviewed, including 13 employees who had participated in the intervention study and 4 who had not participated. The interviewees were enquired about their experiences with the intervention process, motivations to learn stress management skills and participate in the study, attitudes toward occupational well-being, and mobile coaching in an organizational context. The intervention participants were also enquired about the motivations and user-experiences related to the app use. If an interviewee was working as a manager, such as team manager, additional questions were presented concerning the possibility of app use in team meetings and benefits of the app for managers. Interviews were conducted as 30 to 60 minute recorded phone interviews, after which notes were written and significant parts transcribed.
Data about the app usage was collected via app logs. When signing the informed consent, participants received a study identification code which was used to link the intervention usage logs to the participant. The participants were asked to input their code into the app settings so that their usage data could be identified. The usage log of the app, including time stamped user actions, was transmitted to a database on a secure server.
Baseline and final questionnaires were considered in the data analysis. The scales were analyzed, and their median values and inter-quartile ranges calculated. Statistical tests were conducted with IBM SPSS Statistics (version 20) software. Change in participants’ ratings of well-being was analyzed with a non-parametric Wilcox test and differences between the companies were analyzed with a non-parametric Mann-Whitney test. Well-being results were analyzed separately for both companies, and all together to evaluate the overall intervention. For the motivation items, the 7-point Likert scale was additionally dichotomized into agree or disagree for further analysis; persons answering 1-3 were considered disagreeing and 4-7 agreeing.
For the qualitative data, the overall evaluation was made by combining both populations due to the small sample size. The questionnaires’ open-ended questions and data from the final interviews were processed by putting the information from written documents into tables (MS Excel 2010, version 14) according to the original themes of the questionnaires and interviews. After this phase, common themes related to adoption as well as participants’ motivation and experiences were identified. In identifying motivations, themes arising from Self-Determination Theory (SDT) were used [
Usage logs were processed to calculate the usage duration in days and unique uses of the app. The duration was calculated as the time between the first and last timestamps. Actual use days, meaning the number of days when the app was used, were calculated.
In total, 13.0% (43/332) of the employees participated in the study (
From the identified 25 logs, the mean use of the app was used 4.8 (SD 4.7) different days (range 0-19) and the mean app use period was 65.7 (SD 60.3) days (range 0-154). The app use period means the time between the first use session and last use session. For 64% (16/25), the use period was over week.
Participant flow diagram.
Since the face-to-face interviews in the beginning of the study provided largely similar information from both companies with respect to the companies’ characteristics, needs, and well-being, the intervention was implemented in a similar manner in both companies. However, there were minor deviations in the informing of the intervention. In Company B, the employees were more thoroughly informed of the upcoming study by the monthly letters and the participation of a researcher in one of the company’s remote meetings. Furthermore, the company culture, motivational factors, and challenges were somewhat different, and therefore, the information sessions emphasized different benefits, namely personal well-being and new skills in Company A and occupational well-being and helping others in Company B.
The findings show the intervention process applied in this study did not lead to successful adoption of a digital intervention in occupational well-being: only 13.0% (43/332) filled in the baseline questionnaire and only 8.1% (27/332) of all employees started using the app. In the following sections, the assumptions which the intervention design was based on, the correct assumptions (italics), and a clarification of where the assumptions failed are described.
The assumptions in the beginning were that the HR managers (1)
The assumptions in the beginning were that (1)
The assumptions in the beginning were that (1) aligning the intervention activities with existing activities in the company makes the process simpler for the employee; and (2)
The assumptions in the beginning were that (1) the HR managers pay attention to intervention penetration; and (2)
The assumptions in the beginning were that (1) three months is a sufficient time to detect meaningful changes in well-being indicators; and (2) introduction of the intervention leads to changes in organizational procedures. The first assumption was not possible to test with such a small study population and low adherence. The second assumption was also incorrect. The intervention had no observable impact on organizational procedures.
This section describes findings regarding individuals’ motivations related to study participation, digital intervention use, and user experience. The findings suggest that autonomous regulations were strongly present in both motivations to participate in the study and in the motivations to use the digital intervention. This is important since it is the most beneficial type of motivations in terms of long-term behavior change.
Motivations to participate in the intervention were inquired in the beginning of the study with an ad hoc SRQ. The most common reasons to participate in the study among the respondents (n=42) were (1) desire to feel better at work (100%, 42/42); (2) desire to improve well-being (100%, 42/42), (3) desire to learn new skills (98%, 41/42); (4) enjoyment of trying something new (93%, 39/42), (5) desire to advance scientific research (86%, 36/42), (6) interest in participating in a research study (86%, 36/42), (7) interest in figuring out how the digital intervention works (76%, 32/42); and (7) feeling good about doing something for the good of the society (67%, 28/42). Interviews highlighted similar motivations.
Barriers for participating were inquired with a separate mini-questionnaire from the non-participants (n=62) and they reported not participating in the study mainly because they (1) did not see problems in their occupation well-being and thus, they had no need for the app (68%, 42/62); (2) did not have time (65%, 40/62); and (3) did not remember to participate, which implies that the study was not in high priority for them (48%, 30/62).
Motivations to use the app were inquired in the midterm questionnaire during the study with an ad hoc SRQ. The most common reasons to use the app among the respondents (n=18) were (1) importance of the personal well-being (100%, 18/18); (2) interest in making life changes (100%, 18/18); (3) desire to learn new things (100%, 18/18); (4) appreciation towards the app contents (94%, 17/18); (5) it was fun to do exercises with the app (83%, 15/18); and (6) it brought enjoyment to process everyday issues with the app (72%, 13/18).
Barriers for use were inquired in the midterm questionnaire from the participants who did not use the app. The most common reasons among the respondents (n=7) not to use were (1) not finding the time (100%, 7/7); and (2) not having a suitable phone (71%, 5/7) (eg, Windows phones were popular in Company B). Interviews confirmed that lack of time, having no need, and not remembering (ie, low priority) hindered app use. As one project manager from company B commented: "As a mother of small children, my time is limited."
Participants suggested several ways to support app use, such as group meetings, common use sessions at a workplace, reminders, follow-ups and provision of information on study progress.
User experiences were collected in the end of the study from the 13 interviewees who had used the app relatively actively.
The participants experienced that using the app helped mindfulness to become part of their routines, because the exercises gave concrete instructions how mindfulness can be practiced. They learned ways to perform breathing and relaxation exercises and the app made them aware of the importance of being present. As one team manager from Company B commented: "From that you remember those things and there are good phrases which are good to remember that ‘oh yeah, this is exactly how it goes’, kind of the awareness and reminding that these are good things."
However, participants, who were already familiar with mindfulness, experienced that the app did not bring anything new to them.
The app was seen as a toolkit for personal well-being. It helped in stress management, removed anxiety, brought relief in stressful situations, and helped to concentrate (eg, on themselves and their well-being) through improved mindfulness skills. As one project manager from Company A commented:
I got positive feelings (about this). It gave such an impression that if I would like to think about a small relaxation in the middle of the day, it would help and it could provide tools for this.
However, participants often stated they had used the app so little that it did have a noticeable influence on their well-being yet, despite feeling that intervention had a positive impact on the underlying factors of stress.
The app was mainly found to be easy to use. As a software designer from Company A commented:
I liked that the exercises were short and you were able to mark them as done. You didn’t have to remember where you were going. When you logged in it took you where you were going. So in that respect it was good (app) in my opinion.
However, sometimes the structure caused some confusion. For example, it was difficult to find the same exercise again or to understand where they were in the app. The short exercises were seen positively but there were varying opinions about the introduction videos. For example, the credibility of the videos was questioned and it was difficult to find time to watch them. Performing the exercises at one’s own pace was seen both as a possibility and as a risk. Because participants had to use the app on their own time, performing exercises occasionally at work gave them bad conscience.
Stress and work engagement (UWES-9) were the primary outcome measures in the effects evaluation. The changes between well-being measures were calculated for the participants who provided information for all the measures both in the beginning and in the end (
Intervention had no significant effects on well-being in the two companies, neither separately nor when considered as one population calculated by the Wilcoxon test.
Median effects on the well-being-related measures.
|
Company A (n=12) | Company B (n=13) | Both companies (n=25) | ||||
Beginning | End | Beginning | End | Beginning | End | ||
Stress |
|
|
|
|
|
|
|
|
MED | 3.0 | 3.0 | 2.0 | 3.0 | 3.0 | 3.0 |
|
IQR | 1.8 | 1.8 | 1.0 | 0.5 | 1.0 | 0.5 |
UWES-9 |
|
|
|
|
|
|
|
|
MED | 4.4 | 4.3 | 4.2 | 4.3 | 4.2 | 4.3 |
|
IQR | 2.4 | 2.3 | 2.1 | 1.6 | 2.2 | 1.8 |
P-TyHy |
|
|
|
|
|
|
|
|
MED | 51.5 | 51.0 | 53.3 | 52.5 | 51.7 | 51.3 |
|
IQR | 17.9 | 17.5 | 12.9 | 9.4 | 14.0 | 10.0 |
TyHy |
|
|
|
|
|
|
|
|
MED | 55.3 | 56.1 | 40.6 | 45.0 | 47.8 | 46.7 |
|
IQR | 16.8 | 16.5 | 12.5 | 8.6 | 18.9 | 19.2 |
SWLS |
|
|
|
|
|
|
|
|
MED | 24.0 | 24.0 | 21.0 | 21.0 | 24.0 | 23.0 |
|
IQR | 8.5 | 7.8 | 8.0 | 10.0 | 10.5 | 7.0 |
FMI |
|
|
|
|
|
|
|
|
MED | 34.0 | 30.0 | 35.0 | 39.0 | 35.0 | 36.0 |
|
IQR | 12.5 | 23.5 | 4.5 | 6.5 | 9.0 | 14.0 |
WAAQ |
|
|
|
|
|
|
|
|
MED | 26.5 | 23.0 | 31.0 | 31.0 | 29.0 | 29.0 |
|
IQR | 16.0 | 12.3 | 8.0 | 6.0 | 8.5 | 11.0 |
This study evaluated the adoption process and effects of a mental health app targeted to individual employees at organizations. Because the findings from a prior pilot study were positive [
The assessment of motivations and user experiences shows that employees participated in the intervention and used the app for the "right reasons" (eg, personally motivating reasons). However, the lack of time, the lack of perceived benefits, and the lack of perceived need prevented them from actively using the app and practicing its exercises to improve their stress management skills. Although the app was not targeted only for people under stress or with a diagnosed illness, it apparently failed to bring enough value for mostly healthy people. The app should be developed further in order to concretize its benefits as a preventive approach with immediate positive impact on well-being.
Based on our study, some specific factors that could have a positive influence on the adoption process were identified. Because time was not allocated for employees to use the app at work, its use did not become a part of organizations’ everyday practices. Thus, it would be beneficial to allocate time for app use at work. The HR managers stated that management has to make a decision in principle to adopt the app and only then they can suggest that all employees use the app at work. This finding is in line with earlier studies that have shown that managers’ attitudes influence the intervention success [
Management might have more positive attitudes toward the app if it would have proven benefits related to organization’s functions and overall performance of the employees, such as a possibility to measure, monitor, and manage workload or employee welfare. However, the HR managers pointed out that an action plan is needed to describe what to do in risk situations, such as if the employee welfare suddenly decreases. It is also important that the app has proven benefits for the employees and they are communicated properly.
In this study the intervention was delivered through a mobile and a Web app, but the mobile app was not available for Windows phones, which hindered the adoption. It would be important to provide the tools for using the app or develop an app for all the platforms that are used in the end-user organisation. Earlier studies have reported similar findings [
Overall, co-operation with HR managers was found to be useful in this study. They were a good source of information for the researchers. It would have been useful to co-operate with them in the concept design phase in order to get a more thorough understanding of the organization’s needs. Moreover, it should be considered to include top management in the intervention planning since they make the strategic decisions. Then it might be possible to make necessary changes in the organisation for the intervention or design them as part of the intervention
Before and during the intervention the employees should have been motivated to participate in the study and use the app, and arranging concrete activities or events at the workplace could have increased the reach and engagement. Earlier research suggests that lack of support outside training sessions can be a barrier for individual or group training in the workplace [
There are many possibilities for future research. For example, traditional training methods or well-being campaigns could be combined with digital interventions. Offering physical contact with other people as one intervention component could make also the digital component more engaging [
This study has some limitations. The small number of participants makes it difficult to assess the effects of the intervention on their well-being and on the organizational settings. Also, the appropriateness of the well-being measures can be questioned, because their sensitivity in the relatively short duration of the study may be suboptimal. Additionally, the open study setting and the uncontrolled participant selection increases the probability of biased results [
This article presents findings from a 4-month study of an individual-level digital mental health intervention at two organizations. Process and effects evaluation from the organization’s point of view was conducted together with the study of employees’ motivations and user experiences. The low number of participants and low intervention adoption show that both the intervention process as well as the digital app should be developed further in order to be successful in this context. Importantly, the study suggests that the intervention planning process should aim to involve the top management of the organization. Although the intervention was not successful as such, the process evaluation provides important insights into how digital interventions should be planned and conducted in the context of occupational well-being. At the organizational level, top management needs to be involved in the intervention planning for fitting into the organization policies, the existing technology infrastructure, and targeting the organizational goals. At the individual level, it has to be tackled how to create time for use and concretise the benefits of the preventive intervention.
Acceptance and Commitment Therapy
mindfulness questionnaire
human resource
information and communication technology
personal well-being questionnaire
Self-Regulation Questionnaire
Satisfaction with Life Scale
work community well-being questionnaire
Utrecht Work Engagement Scale
Technical Research Centre of Finland Ltd
Work-Related Acceptance and Action Questionnaire
This work was supported by the Finnish Work Environment Fund (Työsuojelurahasto) under Grant 113050. We wish to thank the Finnish Work Environment Fund for funding this research, and the participating companies and the Research and Education Centre Synergos for their support. Additionally, this work was supported by a grant from Tekes – the Finnish Funding Agency for Innovation as part of Digital Health Revolution Program. The multi-disciplinary program is coordinated and managed by Center for Health and Technology, University of Oulu, Finland. Also, we want to acknowledge the valuable contribution of the people who took part in the study.
None Declared.