Nielsen and Abildgaard’s intervention evaluation framework was used for intervention process and effects evaluation [22]. The framework, where we also highlight the central components including digital intervention, motivations and user experiences as part of the change mechanisms are illustrated in Figure 1. The results are structured first with process evaluation including change mechanisms and intervention process. Then the effects evaluation is done on the effects on well-being.

The intervention was designed to follow the stages outlined in Table 2 according to Nielsen and Abildgaard [22]. The delivery of the intervention was planned to be similar in both companies, with only small adaptations resulting from the information gathered during the initiation and screening stages.

The study and all the related questionnaires were reviewed by the Ethics Committee of the Pirkanmaa Hospital District. Participants’ informed consent was obtained in the beginning of the baseline questionnaire online. The data was stored on a secure server. All the data was anonymized for reporting and publication.

Data was collected between August 2013 and May 2014. The study began in August 2013 with HR managers’ interviews, and the intervention took place between November 2013 and March 2014. The study ended with the final interviews with the participants in April to May 2014.

Electronic questionnaires were sent via LimeSurvey (version 2.00) in the beginning of the study for all employees, and at two and four months for those employees who completed the baseline questionnaire. Occupational health and well-being was assessed in the beginning and in the end using a 1-item Stress questionnaire (Stress) [28], and a 9-item Utrecht Work Engagement Scale (UWES-9) [29,30] as primary outcome measures. The scores in the Stress questionnaire range from 1-5 with higher scores signifying a worse stress situation, whereas scores range from 0-6 for the UWES-9 test, where a higher score signifies a better work engagement. The Stress scale has shown satisfactory content, criterion, and construct validity for group level analysis [28]. The internal consistency of UWES-9 is good, with Cronbach's alpha varying between .85 and .92 across samples in 10 different countries [31,32].

Secondary outcome measures were an 84-item work-related well-being questionnaire which assessed both personal and work community well-being (P-TyHy, TyHy) [33], a 5-item Satisfaction With Life Scale (SWLS) [34], a 14-item Mindfulness questionnaire (FMI) [35], and a 7-item Work-Related Acceptance and Action Questionnaire (WAAQ) [10]. Motivations related to participation in the study and using the app were studied in the beginning, in the middle of the study, and in the end using ad hoc 16- and 14-item Self-Regulation Questionnaire (SRQ) [36]. Because an existing SRQ questionnaire for this specific context did not exist, questions were adaptations of the existing SRQ questionnaires. Motivations were inquired with additional open-ended questions. Details of the questionnaires and scales are shown in Table 3.

Qualitative data consisted of several semi-structured interviews. Before the intervention, HR managers (n=2) were interviewed separately, face-to-face for the companies’ characteristics, needs, and well-being challenges. In addition, the schedule for the intervention activities was outlined together with them. After the intervention, the HR managers were interviewed for their views on the success of the intervention, attitudes toward occupational health and mobile coaching in an organizational context, as well as intervention adoption in organizations. It should be noted that the HR manager of Company A changed during the course of the study.

After the intervention, a voluntary subset of employees (n=17) were interviewed, including 13 employees who had participated in the intervention study and 4 who had not participated. The interviewees were enquired about their experiences with the intervention process, motivations to learn stress management skills and participate in the study, attitudes toward occupational well-being, and mobile coaching in an organizational context. The intervention participants were also enquired about the motivations and user-experiences related to the app use. If an interviewee was working as a manager, such as team manager, additional questions were presented concerning the possibility of app use in team meetings and benefits of the app for managers. Interviews were conducted as 30 to 60 minute recorded phone interviews, after which notes were written and significant parts transcribed.

Data about the app usage was collected via app logs. When signing the informed consent, participants received a study identification code which was used to link the intervention usage logs to the participant. The participants were asked to input their code into the app settings so that their usage data could be identified. The usage log of the app, including time stamped user actions, was transmitted to a database on a secure server.

In total, 13.0% (43/332) of the employees participated in the study (Figure 4). The mean age of the participants was 37 years (range 25-54) and 67% (29/43) were female. It was found that 63% (27/43) of the participants, that is 8.1% (27/332) of all the employees started using the app, and 43 (13.0%, 43/332) answered the baseline questionnaire, 25 (58%, 25/43) the mid-term questionnaire, and 26 (60%, 26/43) the final questionnaire. Of those, 25 (58%, 25/43) participants answered all the questionnaires and were included to the longitudinal statistical analyses.

From the identified 25 logs, the mean use of the app was used 4.8 (SD 4.7) different days (range 0-19) and the mean app use period was 65.7 (SD 60.3) days (range 0-154). The app use period means the time between the first use session and last use session. For 64% (16/25), the use period was over week.

Stress and work engagement (UWES-9) were the primary outcome measures in the effects evaluation. The changes between well-being measures were calculated for the participants who provided information for all the measures both in the beginning and in the end (Table 4). The table demonstrates that the baseline level of the participants was quite good. Stress was significantly lower in Company B in the beginning (Mann-Whitney U=40.5, P=.03) and TyHy (work community well-being) was significantly higher in Company A both in the beginning (Mann-Whitney U=35.0, P=.02) and in the end (Mann-Whitney U=38.5, P=.03). The reliabilities of the measures were calculated using Cronbach’s alpha with the data from the preliminary and final questionnaires. For UWES-9 it was between .94 and .95, .95 for P-TyHy, .96 for TyHy, between .88 and .92 for SWLS, between .89 and .94 for FMI, and between .95 and .96 for WAAQ. For the single-item Stress scale, Cronbach’s alpha could be calculated but showed satisfactory content, criterion, and construct validity for group level analysis [28].

Intervention had no significant effects on well-being in the two companies, neither separately nor when considered as one population calculated by the Wilcoxon test.

This study evaluated the adoption process and effects of a mental health app targeted to individual employees at organizations. Because the findings from a prior pilot study were positive [14], it was expected that the app would be well received and it would have a positive impact on employees’ well-being. However, the adoption rate of the app was low as only a small number of employees (13.0%, 43/332) chose to take part in the intervention, and therefore, the effects of the intervention on the employees’ well-being could not be verified. However, the multiple item questionnaires were tested with Cronbach’s alpha and were proven reliable (alpha >.8).

The assessment of motivations and user experiences shows that employees participated in the intervention and used the app for the "right reasons" (eg, personally motivating reasons). However, the lack of time, the lack of perceived benefits, and the lack of perceived need prevented them from actively using the app and practicing its exercises to improve their stress management skills. Although the app was not targeted only for people under stress or with a diagnosed illness, it apparently failed to bring enough value for mostly healthy people. The app should be developed further in order to concretize its benefits as a preventive approach with immediate positive impact on well-being.

Based on our study, some specific factors that could have a positive influence on the adoption process were identified. Because time was not allocated for employees to use the app at work, its use did not become a part of organizations’ everyday practices. Thus, it would be beneficial to allocate time for app use at work. The HR managers stated that management has to make a decision in principle to adopt the app and only then they can suggest that all employees use the app at work. This finding is in line with earlier studies that have shown that managers’ attitudes influence the intervention success [37].

Management might have more positive attitudes toward the app if it would have proven benefits related to organization’s functions and overall performance of the employees, such as a possibility to measure, monitor, and manage workload or employee welfare. However, the HR managers pointed out that an action plan is needed to describe what to do in risk situations, such as if the employee welfare suddenly decreases. It is also important that the app has proven benefits for the employees and they are communicated properly.

In this study the intervention was delivered through a mobile and a Web app, but the mobile app was not available for Windows phones, which hindered the adoption. It would be important to provide the tools for using the app or develop an app for all the platforms that are used in the end-user organisation. Earlier studies have reported similar findings [38].

Overall, co-operation with HR managers was found to be useful in this study. They were a good source of information for the researchers. It would have been useful to co-operate with them in the concept design phase in order to get a more thorough understanding of the organization’s needs. Moreover, it should be considered to include top management in the intervention planning since they make the strategic decisions. Then it might be possible to make necessary changes in the organisation for the intervention or design them as part of the intervention

Before and during the intervention the employees should have been motivated to participate in the study and use the app, and arranging concrete activities or events at the workplace could have increased the reach and engagement. Earlier research suggests that lack of support outside training sessions can be a barrier for individual or group training in the workplace [38]. Moreover, it is important to avoid burdening employees with data collection activities and rather aim to use the data that is already being collected. In this study, it proved difficult to integrate the data collection activities with the existing ones, especially as the researchers were working outside the organization. With an open setting used in this study, it was challenging to engage employees in research procedures, especially final interviews.

There are many possibilities for future research. For example, traditional training methods or well-being campaigns could be combined with digital interventions. Offering physical contact with other people as one intervention component could make also the digital component more engaging [13,39]. In addition, other solutions on how to raise the participation among employees should be studied. Building incentives into wellness programs has been suggested as one solution [5].

This study has some limitations. The small number of participants makes it difficult to assess the effects of the intervention on their well-being and on the organizational settings. Also, the appropriateness of the well-being measures can be questioned, because their sensitivity in the relatively short duration of the study may be suboptimal. Additionally, the open study setting and the uncontrolled participant selection increases the probability of biased results [39]. Even though ACT-based exercises are suitable for everyone for preventing mental health problems and stress, it is not known whether the results would have been different if the intervention would have been targeted for people under stress or with a diagnosed illness.

This article presents findings from a 4-month study of an individual-level digital mental health intervention at two organizations. Process and effects evaluation from the organization’s point of view was conducted together with the study of employees’ motivations and user experiences. The low number of participants and low intervention adoption show that both the intervention process as well as the digital app should be developed further in order to be successful in this context. Importantly, the study suggests that the intervention planning process should aim to involve the top management of the organization. Although the intervention was not successful as such, the process evaluation provides important insights into how digital interventions should be planned and conducted in the context of occupational well-being. At the organizational level, top management needs to be involved in the intervention planning for fitting into the organization policies, the existing technology infrastructure, and targeting the organizational goals. At the individual level, it has to be tackled how to create time for use and concretise the benefits of the preventive intervention.