According to the World Health Organization, anxiety disorders currently affect approximately 3.6% of the world’s population, which translates to about 264 million individuals worldwide, making them one of the most common mental health conditions globally [1]. Moreover, in the United States, approximately 31.1% of adults experience some form of anxiety disorder during their lives [2]. These numbers do not include the distress experienced by individuals dealing with subclinical levels of anxiety daily. Anxiety often arises when people feel threatened or face stressful situations. Prolonged or intense stress can disrupt the body’s stress response system, leading to heightened anxiety levels [3,4]. By regulating the stress response system, individuals can learn to modulate their reactivity to stressors, reduce anxiety symptoms, and enhance overall well-being [5-7].

Ecological momentary interventions (EMIs) are digital interventions delivered in real time in real-world settings to help individuals integrate specific strategies into their everyday life [8]. The widespread use of smartphones and other mobile devices makes EMIs easily accessible and provides a broad reach. Therefore, EMIs offer a scalable, cost-effective, and flexible platform for delivering mental health interventions, including psychotherapy [9,10]. In particular, EMIs have great potential for helping individuals cope with stress in real time, because they can provide immediate support and resources to address stressors as they arise throughout the day. Consequently, EMIs present a promising avenue for enhancing therapeutic outcomes and promoting mental well-being by giving the general public access to important tools and strategies for psychological resilience and well-being.

One approach that is consistently emphasized in various psychotherapies, including metacognitive therapy [11], cognitive behavioral therapy (CBT) [12], acceptance and commitment therapy (ACT) [13], mindfulness-based stress reduction [14], and various mindfulness-based interventions [15], is the development of metacognitive skills. Metacognitive skills allow individuals to effectively navigate and regulate their cognitive and emotional processes, leading to improved coping strategies and overall well-being [6,16-18]. Therefore, combining the benefits of metacognitive skill training with the accessibility and effectiveness of EMIs is a promising avenue to helping individuals actively cultivate self-awareness, regulate their cognitive processes, and integrate specific coping strategies into their daily lives.

InsightApp is a gamified EMI and research tool designed to strengthen people’s metacognitive skills for coping with challenging real-life situations and embracing strong emotions [19]. The main strategies the app integrates are the activating event, belief system, consequences, disputation of beliefs, and effective new beliefs (ABCDE) method for cognitive restructuring from CBT; reflections on values and committed action from ACT; and emotion regulation strategies from ACT, CBT, and mindfulness-based interventions.

In addition to the benefits of helping individuals improve their mental well-being, EMIs facilitate running longitudinal studies with the help of ecological momentary assessment. This approach allows researchers to collect real-time data on the effectiveness of interventions over extended periods [20]. A longitudinal approach addresses methodological shortcomings, such as retrospective recall bias, and enables examination of both between-participant (personality level) differences and within-participant variations [21]. By incorporating longitudinal designs and daily assessments in real-life settings, researchers can gain valuable insights into the dynamic nature of stress and coping, contributing to the development of more effective and personalized interventions.

This experiment aimed to conduct a longitudinal randomized controlled trial to evaluate the effectiveness of InsightApp in enhancing individuals’ mental health and well-being as well as addressing their daily anxiety levels and their capacity to engage in valued actions when confronted with challenging situations. Our experiment evaluates the impact of the intervention on mental health and well-being before and after the intervention. Furthermore, through a longitudinal experimental design with a robust placebo control group, we gain insights into how InsightApp affects participants’ anxiety, their ability to cope with stress, and their engagement in valued actions in everyday life.

The experiment was conducted under protocol number 510/2020BO approved by the Independent Ethics Commission at the Medical Faculty of the University of Tübingen. The Max Planck Society Data Protection Office has granted authorization for the data protection and privacy policy protocol. This authorization confirms that the project proposal adheres to the internal data protection requirements of the Max Planck Society as well as the General Data Protection Regulation (EU 2016/679) and the German Federal Data Protection Act. The data protection and privacy policy agreements are incorporated in both the ethics protocol and the consent form.

To enroll in the experiment, participants provided informed consent to (1) their voluntary participation in the experiment and (2) the sharing of their nonidentifiable data for research purposes. By giving their consent, participants agreed to adhere to the requirements for participation, including aspects such as payments, bonuses, time commitment, privacy policy, data protection, data collection, data use, and the withdrawal process. To ensure privacy, all data collected during the study are stored locally on participants' devices and synchronized securely with Firebase (Google) cloud services, certified under major privacy and security standards. Only the participant and experimenters have access to these data. Participants can withdraw their participation at any time without penalty, with compensation provided pro rata to the time engaged in the study.

The experiment’s payment was split into 2 parts. The initial payment was given to participants upon downloading the app and finishing the 30-minute onboarding. The payment was adjusted to maintain an average wage of GBP £6 (US $7.58) per hour. After completing the experiment, participants were compensated for daily participation, offboarding tasks, and extra bonuses. Through active participation and successful completion of the experiment, participants could earn up to GBP £9.80 (US $10.59) per hour. Excluded participants were compensated for the work completed before their exclusion, using a rate of GBP £6 (US $7.58) per hour. The estimated average experiment time was 154 (SD 8.85) minutes for participants in the control group and 161 (SD 8.04) minutes for those in the treatment group.

The sample for this experiment comprised 228 participants, of whom 197 (86.4%) completed the experiment. Participants were recruited from a pool of approximately 39,000 English-speaking adults with iPhones. We used Prolific (Prolific Academic Ltd), a web-based platform used by >25,000 researchers, to recruit and compensate these individuals. Prolific offers access to a diverse pool of >130,000 potential participants, enabling efficient web-based research engagement.

In our study, participants were enrolled at 2 separate points. Initially, 228 participants were enrolled at the start of the study. To compensate for the participants who were excluded and those who dropped out before the experimental conditions diverged on day 3, as preregistered, we conducted a second round of recruitment to maintain the required sample size. Specifically, of the 228 participants, 22 (9.6%) were excluded for failing attention checks, 13 (5.7%) for not starting the experiment, and 2 (0.9%) due to technical issues. These 37 (16.2%) participants were replaced by new enrollees.

During the experiment, 30 (13.2%) of the 228 participants were lost to follow-up due to not meeting the participation criteria (n=22, 9.6%) or technical problems (n=8, 3.5%). In addition, 1 participant (0.4%) was excluded from the data analysis for failing midexperiment attention checks. Multimedia Appendix 2 includes a detailed description of the inclusion and exclusion criteria, along with the recruitment and data collection timeline (Table S1 in Multimedia Appendix 2).

The final sample of 197 participants comprised 97 participants (49.2%) in the experimental condition and 100 participants (50.8%) in the control condition. Most (192/197, 97.5%) of these participants shared demographic information (mean age 38, SD 11.50 years; range 20-80 years; female: n=103, 53.6%). The sample included participants who identified as Asian (7/192, 3.6%), Black (3/192, 1.6%), mixed race (5/192, 2.6%), White (175/192, 91.1%), or other (2/192, 1%). More detailed demographic characteristics, including descriptive statistics by group, are provided in Multimedia Appendix 3.

Participants were first directed to a web-based form, which provided instructions for downloading InsightAppExperiment from the App Store and starting the experiment. Participants were randomly assigned to the control or treatment group through an automated process within the app. Participants were blinded to their group allocation. Further details on the randomization and blinding process are provided in Multimedia Appendix 6. After logging into the app, participants provided informed consent, confirming their agreement to participate in the experiment and share their data. The experiment started with a brief 1-minute introductory video that explained the onboarding process, followed by a short survey about the participants’ sex, age, and nationality. To proceed, all participants completed the reactivity and values reflection modules. Participants then set reminder preferences and received payment instructions. Finally, they accessed the app’s main screen to initiate the preintervention phase. They then accessed the app’s main screen to begin the experimental timeline explained earlier.

During the study, participant engagement was actively monitored. Participants received daily reminders within the app to complete their morning routines and fill out their evening reports. In addition, we sent daily messages via Prolific to remind them to participate in the study. If participants did not complete an evening report, they were allowed to do so the following morning.

The Results section is structured into two main parts: (1) the intervention’s effect on mental health and well-being immediately after the intervention phase and 7 days later and (2) the longitudinal data on anxiety, emotional struggle, and valued action. We verified that there were no significant differences (all P>.30) between the control and treatment groups in terms of baseline demographic or psychological characteristics. Detailed baseline comparisons between groups across both categories are provided in Multimedia Appendix 7.

We provide additional details on the study’s dropout, compliance, engagement, and missing data and sparseness in Multimedia Appendix 8. This appendix outlines participant retention, compliance with study protocols, and an analysis of data sparsity, providing insights into participant engagement and potential data-related concerns.

The CONSORT diagram illustrates the participant flow throughout the study, including enrollment, allocation, follow-up, and analysis phases (Figure 4). The diagram shows the enrollment of 228 participants, with 114 (50%) randomly allocated to the treatment group and 114 (50%) to a placebo control group. Attrition included 17 participants lost to follow-up in the treatment group and 13 in the control group. The final analysis comprised 97 participants in the treatment group and 100 in the control group.

In this section, we present the results of the comparison between the treatment group and the control group at posttest and follow-up stages on participants’ scores related to mental health and well-being. This analysis examines the intervention’s effects on mental health and well-being while controlling for participants’ baseline scores before the intervention. Further details on the baseline anxiety sample characteristics, as well as the pre- and posttest means, SDs, medians, IQRs, please refer to Multimedia Appendix 9. This appendix provides a comprehensive overview of these metrics for each condition, time point, and anxiety-related variable.

The 2 groups’ posttest scores on measures of mental health neither significantly differed directly after the intervention (F_1,799=0.173, P=.68, partial η²=0.0002) nor during the follow-up period (F_1,799=0.235, P=.63, partial η²=0.0003). The posttest measures of mental health also did not significantly differ, neither directly after the intervention (F_1,799=1.515, P=.22, partial η²=0.002) nor during the follow-up period (F_1,799=0.719, P=.40, partial η²=0.0009).

In this randomized controlled trial, we examined the effectiveness of InsightApp in helping individuals cope with stress and anxiety in real-life settings. During the intervention period, the treatment group was significantly more resilient to stress in terms of the intensity of their anxiety and the degree to which they struggled with it. Moreover, the treatment group demonstrated a higher enactment of their valued intentions compared to the control condition. These effects were sustained in the postintervention period regarding the struggle with anxiety. However, the postintervention scores and follow-up scores on measures of well-being and mental health did not significantly differ between conditions (all P>.22).

The experiment had the following limitations. First, the sample was relatively homogeneous, lacking diversity in ethnicity, education, and location. This limits the generalizability of findings across different cultural backgrounds. In addition, as the sample was drawn from the general population, the intervention’s effectiveness may vary for individuals with higher anxiety levels. Participants were also compensated, potentially affecting their engagement and motivation, which could differ from those in the general population without such incentives.

Second, the relatively brief training and postintervention period (ie, 7 days) limit our understanding of the intervention’s long-term effectiveness. Extended use may yield more pronounced effects. The duration of sustained improvements beyond the 7-day postintervention phase is also uncertain. Furthermore, the results of our longitudinal data analysis should be taken with a grain of salt because some time series were weakly autocorrelated. Third, the experiment design has room for improvement. Presurvey measures may have unintentionally provided psychoeducation to the control group. Moreover, as the control condition included elements of the reflection modules, such as the reactivity and values modules, our results may underestimate the benefits of the complete InsightApp as a package intervention.

Two significant limitations impacted the pre-post measurement of anxiety in this study. First, the study was underpowered (power of 32.4%, f=0.10, α=.05) to detect small effects on posttest and follow-up measures of psychological traits, and because our intervention was short, small effects were to be expected. Therefore, the nonsignificant findings do not imply that the corresponding effects do not exist. Rather, our study was underpowered to conclusively determine the significance of long-term effects on psychological traits.

Furthermore, the administration of the HAM-A posed challenges. This scale was administered via an online self-report format rather than the traditional clinician-administered interview. The HAM-A is typically conducted by a trained clinician to ensure an accurate interpretation of the items, which may not be as intuitive for participants to self-assess. Although we provided clear instructions, the absence of direct clinician oversight may have affected the reliability and validity of the results. This deviation from the standard administration method should be considered when interpreting the findings, as it may introduce variability in how participants understood and responded to the scale.

Another limitation of this study is the brief duration of the intervention, which may have restricted the ability to detect changes in trait-level measures of anxiety that typically require longer periods to show. Extending the intervention beyond 7 days may enhance its impact, particularly on more stable, trait-like outcomes, and should be explored in future research.

Another potential concern that could affect the interpretability of the results is the evaluation of the app in a population that may not have fully endorsed anxiety symptoms. To address this concern, we conducted an analysis of baseline anxiety levels, as detailed in Multimedia Appendix 9, specifically in the Anxiety Sample Characteristics section. This analysis shows that even though we recruited our sample from the general population, it included a broad spectrum of anxiety levels, with >70% of participants reporting moderate to high anxiety traits. There was no evidence that the effectiveness of the intervention varied for participants with differing baseline levels of anxiety (P>.05; refer to Multimedia Appendix 9). However, because the app was not tested in a clinical sample, we cannot conclude that the intervention would be effective or have a similar size effect in a clinical population. This makes testing the intervention in a clinical sample an important direction for future research.

Finally, while we observed significant effects in the longitudinal data, we did not find such significant results in the comparison of pre- versus posttest measures. The lack of pre-post significant results could raise questions regarding the overall impact of the intervention on participants’ psychological well-being. In the subsequent section, we explore potential factors that could explain the lack of statistically significant pre-post effects.

The intervention effectively reduced daily anxiety levels and struggle with anxiety compared to the control group. However, no statistically significant differences were observed in postintervention self-report questionnaires. When interpreting these results, it is important to consider that this discrepancy may stem from the study’s short duration and its lack of power to detect small effects in pre-post measures. In addition, differences in outcome variables could also contribute to these findings. The experience sampling data showed anxiety reduction in specific stress situations, while postintervention questionnaires assessed general mental health and well-being regardless of stress. In addition, although the wording of the instructions for the self-report measures—originally designed to assess traits—was adjusted to reflect the past two weeks, it may still have lacked sensitivity to detect short-term intervention effects. Longer interventions may have a more substantial impact on traits, whereas shorter-term measures, such as ecological momentary assessment or daily diaries, are more sensitive to short-term fluctuations in psychological states [20,42]. Further investigation and extended interventions are needed for a comprehensive understanding of the intervention’s effectiveness in promoting psychological well-being.

Notably, participants’ improvement in anxiety struggle levels during the intervention was sustained and even reduced after the intervention. However, the effect on anxiety intensity did not persist. This suggests that the app’s focus on metacognitive skills may help users acknowledge and accept intense emotions, reducing their struggle with anxiety even without app support. However, participants might have relied on the app for anxiety intensity reduction rather than the app’s general strategies. Further investigation is required to understand how the app impacted anxiety intensity and develop strategies for users to experience these benefits independently of the app. Differential retention of effects between anxiety intensity and struggle underscores their distinct dimensions in an individual’s anxiety experience. Measuring and addressing both aspects is crucial for a comprehensive understanding of anxiety’s impact on well-being.

Another point worth noting when interpreting the results is that baseline anxiety traits did not significantly moderate the intervention’s outcomes. This indicates that there is no evidence to suggest that the intervention’s effectiveness varies across different levels of baseline anxiety. Although the app was tested on the general population, the sample exhibited a broad range of anxiety levels, with a substantial proportion of participants experiencing moderate to high anxiety traits, underscoring broad applicability.

In this section, we compare the results of our experiment with previous research in the field. InsightApp collects data using experience sampling and pre-post follow-up assessments of mental health and well-being. Most studies on EMIs for anxiety only rely on pre-post comparisons [9,43,44]. By contrast, our experiment additionally used experience sampling to track participants’ reactivity to stress on a daily basis during and after the intervention. To the best of our knowledge, there is currently no other experiment that examined how EMIs moderate the effects of stress on daily reports of anxiety, struggle with anxiety, and intention enactment. Furthermore, InsightApp incorporates multiple psychotherapeutic strategies from ACT, CBT, and mindfulness-based cognitive therapy into an integrated gamified approach. To the best of our knowledge, previous mental health apps for stress and anxiety have not integrated all of these components into a game for people to practice metacognitive skills with their real-life challenges. Therefore, we compare our findings with previous studies on digital interventions that include ≥1 component similar to those of InsightApp.

Mindful awareness and acceptance of emotions are key components actively trained by InsightApp. Previous studies in the literature on stress coping have provided evidence supporting the idea that being aware of the present moment can reduce emotional reactivity to stressors [6,7,45-47]. Our results indicate that when participants engage in a daily practice of noticing, accepting, and embracing their emotions, they experience significantly less anxiety when facing stressors and report fewer struggles with their feelings of anxiety compared to the control group. Our findings highlight the potential benefits of incorporating these practices into daily life beyond the controlled settings of a laboratory environment. In addition, our experiment supports the notion that training metacognitive skills with daily challenges leads to sustained improvements. The findings indicate that metacognitive skills for coping with stress and anxiety are trainable and can be effectively improved with a relatively small amount of practice. Importantly, the positive effects of this training on the degree to which participants struggle with anxiety were sustained after the training period ended. This suggests that metacognitive skills can be cultivated and applied in everyday life. This makes them valuable tools for managing stress and anxiety.

Some studies have also examined the interaction between mindfulness, coping with stress, and value-congruent action. These studies have found evidence supporting the hypothesis that individuals who possess higher levels of present-moment awareness are less emotionally reactive to stressors and are more likely to engage in value-congruent actions [17,46-48]. The results of our experiment consistently align with previous findings. Participants who practiced awareness and acceptance of their emotions using InsightApp demonstrated significantly lower emotional reactivity to stressors, experienced less anxiety, and exhibited a higher frequency of enacting their valued intentions compared to the control group.

As reported in the Results section, participants’ scores on postintervention measures of mental health and well-being did not show significant improvement compared to the control group, neither directly after the 7-day intervention phase nor in the follow-up assessments. The meta-analysis conducted by Versluis et al [43] covered 15 interventions targeting anxiety, indicating a small to medium effect size of Hedges g=0.47 on pre- and posttest anxiety measures.

When contrasting the effectiveness of InsightApp with these interventions, it is important to take into account factors such as the intervention duration, experimental design, the inclusion of control conditions, and the strength of those control conditions. As discussed in the Methods section, previous work has emphasized the significance and complexities involved in crafting robust control conditions that can effectively address the digital placebo effect [22,23]. This phenomenon involves improvements in outcome variables that are driven by their belief in the effectiveness of the digital intervention. To effectively address the digital placebo effect, control conditions should closely resemble the treatment condition. Hence, it is crucial for them to exhibit structural equivalence and similar aesthetics, quality, and usability as the experimental condition [23]. In addition, the control condition should feel credible to participants, and the description of the benefits that participants can expect from participating in the experiment should be identical. Waitlist controls fail to address the placebo effect, as participants do not anticipate receiving benefits between assessments. Consequently, studies with control conditions dissimilar to the treatment intervention or waitlist controls might overestimate the intervention’s impact on participants. Experiments with strong control conditions are still rare in research on EMIs for anxiety. For example, from the 15 interventions targeting anxiety reviewed by Versluis et al [43], only 7 included a control group. Of those interventions, only 4 incorporated a placebo control groupor an active control group. Therefore, the effect size of g=0.47 might overestimate the real effect of the digital interventions on participants’ pre-post measures of anxiety. Our experiment addressed this problem by using an active placebo control group that is highly isomorphic to the treatment group. Our experiment thereby isolated the active component under examination. Moreover, our experimental design was attuned to participants’ improvements in daily life. This strategy helped to prevent the potential overestimation of the intervention effects in pre-post outcome measures while also allowing for the capture of more nuanced and gradual improvements in participants’ mental health and well-being.

Finally, the intervention presented in this study was tested as an intensive, short-term treatment, with 1 session per day over a week. The complete study lasted 3 weeks, including a pre- and postintervention period. This contrasts with traditional psychotherapy, which typically spans several weeks or months but usually involves only 1 session per week. Unlike in therapy, short, intensive interventions are not uncommon in research on EMIs. For example, a systematic review of 19 papers on EMIs reported that study durations ranged from 2 to 15 weeks, with an average of 4 weeks [49]. Another study evaluating 26 EMI studies based on CBT principles reported that study durations ranged from 2 weeks to 6 months, with a median duration of 30 days [50].

In this experiment, we conducted a randomized controlled trial combining 2 distinct experimental designs: an experience sampling design and a pre-post follow-up design, each applied to different outcome measures. The experiment’s results showed significant improvements in participants’ anxiety, struggle with anxiety, and intention enactment as reactivity to stress on momentary daily measures. Moreover, the improvement in participants’ struggle with anxiety persisted after the end of the intervention, indicating that participants internalized at least some of the emotion regulation strategies InsightApp had directed them to use during the intervention. However, these positive changes did not lead to significant improvements in self-report measures of overall well-being and mental health. Further research is needed to resolve the apparent contradictions between the observed improvements during the training and the lack of improvements in pre-post measures of general mental health. Future studies should explore intervention durations longer than those used in this study and be adequately powered to detect small improvements, ensuring that subtle effects are not missed due to insufficient sample sizes.