RCT participants randomly assigned to the intervention arm regularly received a dashboard for 2 months. Digital dashboards (Figure S1 in Multimedia Appendix 1) were generated from consenting patients’ data from social media posts, online searches, and phone sensor data. The dashboards included 4 sections: late-night phone activity, physical movement, trending digital activity, and digital communication. Late-night phone activity was derived from AWARE (version 2; Open-source Context Instrumentation Framework For Everyone) and included the amount of time participants’ phone screens were on from midnight to 4 AM. We labeled the duration between midnight and 2 AM as nighttime and 2 AM to 4 AM as late-night activity. The physical movement section was derived using an AWARE pedometer plugin and included miles walked per week, days most walked, and days least walked. Trending digital activity was derived from status updates from Facebook and search queries from Google and YouTube. This section included the top 5 most frequent words from each platform within a 3-month period. The digital communication represented content from a digital source (eg, text message) that patients self-identified as wanting to include on the dashboard. These data were accompanied by responses to 2 prompts, “Why did you share this?” and “How did this make you feel?” (Figure S1 in Multimedia Appendix 1). Prior to receiving the first dashboard, both the patient and therapist received a brief 5-minute training video that described the dashboard, potential uses, and privacy or security risks.

All participants who were enrolled in the RCT (NCT04011540) conducted by Merchant et al [10] (ie, 57 patients and 47 therapists) were randomly assigned to the intervention arm (ie, regularly received a dashboard for 2 months) [10] and were asked to complete a postintervention debrief survey after the 2-month period. The survey data was de-identified and upon completion, participants received a US $50 gift card as compensation. The study protocol was reviewed and approved by the University of Pennsylvania’s Institutional Review Board (831246). The sample size allowed for thematic saturation. The study followed the Consolidated Criteria for Reporting Qualitative Research (COREQ) reporting guideline [11].

Two semistructured debriefing surveys (1 for patients and 1 for therapists) were developed based on a literature review of social media and digital data and psychotherapy, experience with the dashboard, and consumer service and experience questions like the net promoter score (NPS; Table S1 in Multimedia Appendix 1) [12]. Self-report surveys were administered via the Way to Health, Health Insurance Portability and Accountability Act (HIPAA) Compliant survey platform. Participants had the option to complete the survey via videoconference. Participants who selected an interview were conducted via videoconference with the first author (LS), a cis female researcher with more than 10 years of experience in qualitative interviewing. Participants who elected videoconference were asked survey questions verbatim and were audio-recorded. Interviews were approximately 30 minutes long.

We used descriptive statistics (eg, frequencies) to quantify study participants’ responses to close-ended questions. In line with the NPS scoring methods, scores were calculated by subtracting the “detractors” (scores of 0‐6) from “promoters” (scores of 9‐10) [12]. We conducted a post-hoc analysis of therapist NPS using pairwise comparisons.

Open-ended questions were analyzed using NVivo (version 10; Lumivero). A thematic analysis of responses was conducted using the constructivist paradigm. Thematic codes were developed through line-by-line reading of 20% of patient and therapist responses randomly by 2 independent readers (nonauthors). The independent readers met regularly to create a codebook and were supervised by a nonauthor. Each code was given an explicit definition and examples to ensure coding accuracy and to improve intercoder reliability. After the codebook was created, each reader then independently coded 5 deidentified transcripts. The κ score was above 0.75 and discrepancies in coding were resolved through a constant comparison approach [13].

Open-ended questions relating to dashboard experience were also analyzed using latent Dirichlet allocation (LDA) [14] to identify themes and keywords. LDA is a probabilistic topic modeling technique used to identify clusters of co-occurring words in text data. We used MALLET implementation [15] of LDA to generate topics in an unsupervised manner. We first removed noncontent words such as a, an, and is, using the Natural Language ToolKit (NLTK) Python library [16] along with the study-specific words such as dashboards. In addition, highly frequent words such as data and phone were also added to the stopwords list (Table S2 in Multimedia Appendix 1). We generated 3 sets of topics by varying the number of topics to 3, 4, and 5 with α=.5 and β=.01. Two coauthors (LS and MS) qualitatively examined the coherence of topics to select the best set and picked the final set of 3 topics.

Of the 104 intervention participants, 96.1% participated in the debriefing survey or interview from December 2021 to January 2022. Our sample includes 56 patients and 44 therapists. Of which, most (n=89, 89%) were selected to complete the debrief via survey form and 7 patients and 4 therapists elected to complete the survey via videoconference (n=11, 11%). The study sample’s sociodemographics were similar to that of the total RCT sample [10]. Patients’ mean age was 31 years; 89.2% (n=50) were women and 83.9% (n=47) had been in therapy for more than a year. Therapists were mostly female (n=36, 81.8%) and White (n=36, 81.8%), who had a master’s degree (n=33, 75.0%). They worked in mostly private practices (n=31.8, 70.4%) and practiced cognitive behavioral therapy (n=14, 31.8%), eclectic psychotherapy (n=8, 18.2%), psychodynamic (n=5, 11.3%), family systems (n=4, 9.0%), family systems (n=4, 9.0%), and “other” forms of therapy (n=13, 29.5%)

Most patients and therapists (n=48) described their experience with the digital health dashboard as “positive” (Table 1). Overall, most participants felt “very comfortable” sharing digital data and “very comfortable” reviewing and discussing the dashboard (Table S3 in Multimedia Appendix 1).

Patients and therapists most often used the dashboard at the beginning of the therapy session (Table 1). A total of 60% (n=26) of therapists noted that they initiated showing or discussing the dashboard in session whereas 32% (n=18) of patients reported “both the patient and therapist” initiated showing or discussing the dashboard (Table 1). The therapists reported a 14 NPS when asked if they would recommend the dashboard to a colleague and 0 to a patient. A pairwise comparison was −38 for therapists. Patients reported a −22 when asked to recommend using a dashboard to a friend in psychotherapy.

The results from LDA and directed content analysis were organized into three themes (with nine subcategories): (1) dashboard experience (positive, neutral or negative, and comfortable); (2) perception of the dashboard’s impact on enhancing therapy (accountability, increased awareness over time, and objectivity); and (3) dashboard refinements (additional sources, tailored content, and ethics) (Table 2).

The dashboard provided several modules highlighting objective data like time spent on one’s phone late at night (eg, midnight to 4 AM) and average miles walked per week (Figure S1 in Multimedia Appendix 1). A patient elaborated on how they

Another patient shared, “I enjoyed seeing the information and I would try to make sure I spent less time on my phone during the night.” Therapists noted that the dashboard helped identify progress or specific concerns in a more objective way than self-reported disclosures. Specifically, a therapist revealed that a “patient gained insight into their digital behavior and made choices about how to modify it when it becomes problematic.” Similarly, another therapist said, “It was helpful to have data regarding a client’s phone usage other than self-report to increase interventions related to sleep hygiene, avoidance behavior, and increased behavioral activation.”

This mixed methods analysis provides insight on patients’ and therapists’ experience regularly receiving and incorporating dashboards in mental health therapy. Using LDA and directed content analysis, we identified 3 themes and 9 subcategories to characterize the RCT participant experience. We found that the two methods provide complementary strengths; LDA offers automated topic discovery and highlights keywords while the directed content analysis allows for nuanced interpretations. To our knowledge, this is the first study to capture both patient and therapist experiences using the same dashboard over a period of time. Our findings must be interpreted with the understanding that participants were those who had, in the underlying study, already consented to use these data in therapy. Individuals who enrolled in this study may be inherently different from the larger groups of individuals enrolled in mental health therapy.

Given that previous work by Merchant et al [10] found that a digital data dashboard into therapy did not statistically impact patient self-report health-related quality of life, depression or anxiety symptoms, or therapeutic alliance over the 2-month study period the interviews provide additional insight into the RCT null findings. The surveys revealed that most patients and therapists described their experience with the digital health dashboard as positive or neutral (Table 1). The proportion of neutral responses is noteworthy and may relate to dashboard differences by patient. Participants also shared several ways in which they perceived the dashboard to enhance therapy (Table 2). Most themes centered around awareness, objectivity, and accountability and future research should incorporate additional patient-centered validated measures such as self-awareness and patient activation to accurately capture the dashboard’s use. Indeed, patients and therapists were comfortable using the dashboard themselves and reported that the dashboard helped them stay “accountable.” Yet, in every case, they were less comfortable recommending the same approach for a friend or colleague. This is noteworthy as recommending a service or product is a hallmark metric for commercial success [12].

A prominent finding was that patients enjoyed seeing trends over time and having this longitudinal data in the short term may have facilitated behavior changes such as spending less time on social media or on their phone at night. Therapists noted how the late-night module provided objective data to potentially tailor interventions. For instance, a therapist said, “It was helpful to have data regarding a client’s phone usage other than self-report to increase interventions related to sleep hygiene, avoidance behavior, and increased behavioral activation.” This quote exemplifies key measurement-based care (MBC) principles of being data-driven, objective, continuous monitoring, and enhanced personalization. This finding is similar to Hobbs et al [9] where psychotherapists noted a moderate or slight improvement in providing effective treatment due to accessing their patients’ digital data. Regularly receiving digital data metrics during the RCT can be compared to evidence-based practice of MBC, where client data are collected throughout treatment to individualize clinical decision-making [17-19]. MBC often involves regularly and systematically administering validated survey instruments such as symptom rating scales or ecological momentary assessments [18,19] to collect and monitor patient processes over time. In the RCT, rather than completing surveys over time, the digital data displayed on the dashboard were a proxy for key mental health and well-being domains, such as physical activity and time spent online from midnight to 4 AM (Figure S1 in Multimedia Appendix 1). Future work is needed to test if digital data metrics can be effective like MBC in mental health therapy.

Patients and therapists recommended improvements to the dashboard. Patients wanted more control of the data shared. This feedback is critical as there is limited research on regularly receiving digital data in psychotherapy. Yoo et al [20] found that patient autonomy and agency are critical when they asked clinical and human-computer interaction researchers about the design features of a clinician-facing tool for using insights from patients’ social media activity. In addition to design features, both patients and therapists also noted privacy concerns regarding social media data generally and specifically sharing it with one’s therapist [20]. This sentiment was underscored in previous research where patients undergoing mental health therapy thought sharing data with their therapist could be “creepy” [7]. This finding could be related to the national discussion of data privacy on social media platforms [21]. As such participants enrolled in the RCT were comfortable sharing and discussing data and due to the nature of the research, patients and their therapists both provided informed consent and the patients had full control of what type of data they shared and for how long.

Our study has several limitations. First, the RCT and survey were conducted during the COVID-19 pandemic, where most psychotherapy sessions occurred via videoconferencing and introducing the dashboard during this period might have been challenging. For instance, during the COVID-19 pandemic, therapists were managing substantially increased workloads and adding new tasks for some clients can create logistical challenges. Previous literature suggests that this issue can be mitigated by introducing a separate nonclinician professional or digital navigator to assist with technological concerns [22]. Although the American Psychological Association has shared that in 2022 “telehealth is here to stay” [23], telemental health therapy is not the most common modality of therapy. Previous research found that the transition to online telemental therapy affected therapeutic relationships as therapists shared feeling a lack of personal connection with their patients [24]. Second, most were conducted via survey, which limited follow-up questions. Third, participants enrolled in this study were willing to jointly review dashboards in psychotherapy and were heterogenous in terms of mental health conditions, time enrolled in therapy, educational attainment, race-ethnicity, and access to technology [10]. Fourth, given the iterative nature of qualitative analysis, we worked with trained coders to conduct the analysis and the coauthors reviewed the thematic analysis.

Future research is needed to determine if digital data in psychotherapy can improve patient clinical outcomes. If it is useful, as noted in the American Psychological Association’s “Guidelines for the Optimal Use of Social Media in Professional Psychological Practice,” training therapists on how to use social media data safely and ethically in therapy is paramount [25]. This training should also incorporate what therapists should look for in an app or digital tool such as data sharing, privacy, and confidentiality terms [26]. Similarly, the training should focus on how to introduce digital tools and obtain informed consent from the patient. To prepare both therapists and patients for using this novel data, clearly defined guidelines, policies, and best practices should be established and disseminated widely. In the context of receiving informed consent, privacy and confidentiality were highlighted by both stakeholders; future research should focus on a continuous process of informed consent [27].

Our findings show that overall study participants enjoyed receiving a digital dashboard over time and elaborated on how the dashboard was used in therapy to hold them accountable. Therapists found the dashboard to provide insight and to be useful in augmenting care. Both groups highlighted ethical considerations and provided feedback on future dashboard design.