Published on in Vol 7, No 11 (2020): November

Preprints (earlier versions) of this paper are available at, first published .
Leveraging the Power of Nondisruptive Technologies to Optimize Mental Health Treatment: Case Study

Leveraging the Power of Nondisruptive Technologies to Optimize Mental Health Treatment: Case Study

Leveraging the Power of Nondisruptive Technologies to Optimize Mental Health Treatment: Case Study

Authors of this article:

Shiri Sadeh-Sharvit1, 2 Author Orcid Image ;   Steven D Hollon3 Author Orcid Image

Industry Perspective

1Eleos Health, Cambridge, MD, United States

2Center for m2Health, Palo Alto University, Palo Alto, CA, United States

3Vanderbilt University, Nashville, TN, United States

Corresponding Author:

Shiri Sadeh-Sharvit, PhD

Eleos Health

1 Broadway

Cambridge, MD, 02141

United States

Phone: 1 5109848132


Regular assessment of the effectiveness of behavioral interventions is a potent tool for improving their relevance to patients. However, poor provider and patient adherence characterize most measurement-based care tools. Therefore, a new approach for measuring intervention effects and communicating them to providers in a seamless manner is warranted. This paper provides a brief overview of the available research evidence on novel ways to measure the effects of behavioral treatments, integrating both objective and subjective data. We highlight the importance of analyzing therapeutic conversations through natural language processing. We then suggest a conceptual framework for capitalizing on data captured through directly collected and nondisruptive methodologies to describe the client’s characteristics and needs and inform clinical decision-making. We then apply this context in exploring a new tool to integrate the content of therapeutic conversations and patients’ self-reports. We present a case study of how both subjective and objective measures of treatment effects were implemented in cognitive-behavioral treatment for depression and anxiety and then utilized in treatment planning, delivery, and termination. In this tool, called Eleos, the patient completes standardized measures of depression and anxiety. The content of the treatment sessions was evaluated using nondisruptive, independent measures of conversation content, fidelity to the treatment model, and the back-and-forth of client-therapist dialogue. Innovative applications of advances in digital health are needed to disseminate empirically supported interventions and measure them in a noncumbersome way. Eleos appears to be a feasible, sustainable, and effective way to assess behavioral health care.

JMIR Ment Health 2020;7(11):e20646




Psychotherapy is on the edge of a transformation driven by technology. More than 50% of adults in high- and middle-income countries will experience a mental disorder in their lifetimes [1]. However, while psychological treatments for most mental disorders have demonstrated efficacy, the quality and effectiveness of mental health care delivery remain inadequate due to multiple reasons, including access, cost, paucity of clinicians trained in empirically-supported models, and the absence of objective and systematic methods for assessing treatments during their delivery [2,3]. Further, health insurance models often limit the number of sessions patients can receive, thereby requiring sustainable outcomes within a relatively short period of time. Treatment delivery models need to be improved to bridge this gap.

One avenue to improving the efficacy and cost-effectiveness of mental health care and increasing patient satisfaction and engagement is integrating technology into clinical practice through patient evaluations and monitoring of the treatment process [4]. Recent digital developments can provide clinicians with nuanced, real-time information to assist their decision-making capacity. Cutting edge technologies can be incorporated into clinical practice for this purpose. Data can be presented to the provider prior to each session to inform them of the patient’s progress, summarize key points from the last session, and prepare them for the upcoming meeting [5]. These data can also inform supervisors, program directors, and other stakeholders to guide clinical decisions, resource allocation, and training.

The goal of this paper is to examine and provide an update on state-of-the-art techniques and methodologies to assess behavioral interventions. We also provide a conceptual framework for collecting and integrating client and treatment data, summarizing, analyzing, and visualizing information to accurately capture the client’s progress and needs. We then demonstrate the utility of a platform incorporating such instruments in the treatment for anxiety and depression.

Standardized Assessments of Treatment Effects

To formulate a case and assess the impacts of a prescribed intervention, one must rely on data [6]. Technology can be successfully leveraged to provide measurement-based care (MBC), defined as the practice of grounding clinical care in patient data collected throughout treatment [7]. MBC is superior to usual care because it offers several benefits that optimize treatments, such as providing insight into treatment progress, early detection of symptom relapse, and improving outcomes [8]. Evidence indicates that patients whose providers use MBC achieve greater and faster treatment response and symptomatic remission [9]. For instance, 2 validated symptom assessment measures, The Patient Health Questionnaire-9 (PHQ-9 [10]) and the Generalized Anxiety Disorder-7 (GAD-7 [11]), are widely used in practice and have been utilized in thousands of studies and clinical settings globally. As their names suggest, they are very brief, totaling 16 items. They provide an estimation of the patient's depression and anxiety levels and whether these are within the clinically severe, moderate, or normal ranges. Such self-report tools can help clarify the patient’s experience and progress, and provide a real-time signal as to whether the treatment helps them get where they want to be [12].

Despite being the simplest, most cost-effective way of assessing patients’ clinical status, self-report measures in clinical practice have several caveats. Patient compliance with self-assessment is limited, with frequent complaints that questionnaires are cumbersome to complete and repetitive [13,14]. Patients may exaggerate or minimize reports to continue receiving services or avoiding potential consequences of their condition. Therefore, a combination of self-report instruments, clinician ratings, and data from additional resources could likely depict the client’s progress more accurately [15]. In order to make treatments more effective, precise, and relevant to the individuals seeking them, additional measures need to complement surveys and questionnaires. Mental health care must develop innovative technologies that could transform behavioral treatment without disrupting their delivery (eg, consuming time or interrupting the conversation flow) [16,17].

Machine Learning and Human-Centered Designs in Behavioral Medicine

Timely data are crucial for planning treatment and assessing its effects. Is the client occasionally preoccupied with troubling ideas or does she meet criteria for an obsessive-compulsive disorder? Has the veteran in treatment for insomnia been able to sleep better following the intervention? Does the teen in adolescent-focused therapy feel connected and valued enough in treatment to share their suicidal ideation with their therapist? Clinicians use various methods to collect these data and gauge their predictive value. However, new models should integrate numerous data sources to provide health care that is person-centered, efficient, and targeted to meet patients’ unique needs [18]. Technological developments can ensure information collecting is contextualized, optimized, and translated into clinical insights and actionable decisions by providers and stakeholders [19].

Machine learning (ML) provides unparalleled precision and accuracy in predicting treatment outcomes based on data collected early in treatment, and determining the most successful targets for interventions [20]. ML algorithms can integrate many sources of information to predict the client’s functioning in treatment, such as text used in the therapeutic conversation, the proportion of time each participant talked, conversation turn-taking, and the client’s self-report measures over time [21]. ML can also be used to assess treatment fidelity and the therapeutic relationship, which until recently relied on exhaustive manual work, mostly coded in research trials but not applied to community settings [22]. These capabilities enable timelier identification of trends in patients’ symptoms and the issues troubling them, as well as changes in the therapeutic alliance and the therapeutic relationship [23]. Greater awareness of nuanced changes over the course of treatment, both in the content expressed during the sessions and changes in symptoms throughout the treatment period, has the potential to better inform and better prepare the therapist to provide effective interventions, seek out consultation and support, or complement the current intervention with additional treatment modules as needed.

Increased adherence to evidence-based care predicts improved treatment outcomes [24]. However, ongoing evaluation of treatment progress remains a challenge [25]. Nondisruptive measures were introduced in behavioral medicine over half a century ago. There is documentation as early as the 1970s of audio recordings or videotapes of treatment sessions being routinely used in training, supervision, practice, and consultation [26]. Recordings are widely used in behavioral medicine [27] as video or audio recordings have been mandated in many clinical training programs and routine care [28,29]. Most patients report positive attitudes toward the use of recordings in their treatment [30]. For example, 71% of patients in a recent study were open to considering audio or video recordings of their treatment sessions. The patients’ comfort with recording was not associated with treatment refusal, duration, or outcomes [31].

Sophisticated algorithms for voice analysis through natural language processing (NLP) allow for the detection of trends in the conversation’s sentiment, content, and synchrony of participants. Insights from session recordings can inspire behavior change, informing the clinician of metrics relevant for both the process and the content of the intervention. Of note, data collected passively during the regular course of treatment in a nondisruptive manner can inform with respect to therapist variables as well. These data provide a broader, more nuanced consideration of how patient, treatment, and clinician variables interact to achieve treatment effects [32]. As such, methods that passively assess and integrate multiple variables at the same time can significantly affect treatment engagement, adherence, and outcomes. When ML models integrate several data resources and are scientifically robust, they can be implemented in a platform offering artificial intelligence (AI) information and prediction [33].

A plethora of empirical evidence suggests that within-session process variables predict the patient-therapist connection, the patient’s mood and anxiety, and the content of the patient’s interests, strengths, concerns, and dilemmas [34,35]. As such, data derived from session recordings can serve as objective markers of the treatment process and inform the clinician where to head next [36]. The patient/therapist listening ratio, number of cross-talks, and silences all reflect the nature of the clinical relationship, therapeutic alliance, and the patient’s engagement with the treatment. Evidence shows that treatments in which patients speak about themselves and are engaged in the conversation are particularly likely to maintain momentum [35]. Patient involvement in the session and therapist active listening can be observed by conversational interaction when the patient and the therapist take turns speaking, when there are few extended pauses, and when neither party overrides the other. Silences are effective when they are used in later sessions for brief periods. Long silences by the patient reflect a lesser sense of connection, affecting attrition, adherence, and outcomes [34].

Within-session content variables, such as data on most commonly used themes and affective tone, can inform the therapist, stakeholders, and policymakers in addressing underlying perpetuating factors. Integrating sentiment and the themes discussed can further inform stakeholders regarding treatment progress [37]. Patients that express less emotional content tend to rate the therapy as less helpful and their connection with their therapist as weaker [35]. Further, greater therapist insight into the interests, concerns, and experiences of the patient predicts whether they will reach the outcomes desired by each party [38,39].

Current Implementation of Digital Tools to Augment Behavioral Medicine Outcomes

Although their potential role in optimizing treatment delivery has been proposed, in-session and between-session data are not collected regularly, nor have they been integrated into mental health care services as of yet. Barriers often cited are limited clinician time to administer, collect, and analyze data, and concerns that the administration of measures would interfere with rapport and the therapeutic alliance [40]. For a digital tool to be maximally effective, it needs to collect information passively, without increasing therapist burden or reducing face-to-face communication [41]. Additionally, these data should be provided to therapists via a platform that is straightforward and easy to use, with clear visualization and comparison to earlier sessions. Such information presented in a timely fashion can inform decisions regarding resource allocation, such as increasing treatment dose, revisiting the level of care, and team consultation [7]. Further, aggregated data on treatment progress and outcomes can be used by providers and clinic directors for quality assessment.


Eleos Health is a therapy intelligence engine designed to provide intervention insights and inform clinical decision-making. The platform collects key metrics from treatment sessions and integrates them with standardized assessment scales, leveraging insights developed through ML and NLP analysis of large treatment datasets. The Eleos platform integrates subjective and objective measures of the treatment process, the patient-therapist communication, and outcomes into AI software. In the following case illustration, we demonstrate how objective measures of treatment process and content derived from session recordings can be integrated with patient self-assessments in real time to shape clinical insights and decisions in a positive direction. Eleos Health is a digital platform designed to integrate multiple patient data points to present providers with a comprehensive picture of the client’s progress in treatment. The platform is used in an app that can be used on a mobile device or desktop computer for in-person meetings or embedded within teleconferencing programs. The platform applies voice analysis to describe and summarize events throughout the treatment meeting, including the language used by the therapist and the patient. These data are complemented with weekly outcome monitoring through self-report assessments.

Case Example

The following case example illustrates the pilot use of an AI platform hosted by Eleos Health to collect and analyze the content of treatment sessions. The patient described signed consent for using de-identified treatment data prior to beginning the treatment and gave permission for the following text. All identifying information has been changed.

Kyle was a 24-year-old Latino man. He was born and raised in a small suburb next to a big metropolis. The youngest of 4 children, he described a very warm and strongly connected family growing up. He had graduated from a liberal arts college the previous year and found a job in a small startup company. He moved to a new city, where he lived with 3 roommates. Kyle described that in the past 18 months, since his junior year in college, he felt concerned about his future and unsure about which career path to choose. His friends and family recommended that he seek counseling, but after calling several therapists who did not have a slot, he did not begin treatment. Instead, he focused on his eating and physical activity, and thought that his new routine supported his transition into his new job. Kyle recently decided to seek treatment again after receiving a promotion at work. His boss moved out of town and the CEO of the company offered Kyle her position. Kyle accepted the promotion but became very anxious and had a hard time concentrating at work. He also reported sleep problems and that the healthy lifestyle he had worked hard to develop had been derailed.

When Kyle reached out to Dr. Davis, the therapist suggested incorporating a platform in his treatment that could record and analyze their treatment sessions. The therapist also suggested that the system send Kyle weekly assessments to enhance the therapist’s understanding of how best to help him. Kyle was skeptical that a digital platform could inform his therapist beyond the treatment session per se, but decided he had little to lose by trying it out. He signed a consent form that the therapist had sent him, which included an authorization to use a HIPAA-compliant platform named Eleos. Treatment was conducted in a blended fashion, integrating in-person and remotely delivered meetings. The Eleos platform provides voice analysis of the sessions, regardless of their delivery method.

Kyle received a text message from the system before the first session, requesting that he complete the PHQ-9 and the GAD-7. He completed the 16 items on his phone within a few minutes, which served as his baseline scores. When he sat in the therapist’s office in the first session, she pointed at a mobile device that she would use to record and analyze their meeting. Kyle’s therapist was very welcoming and helped him solidify his goals, which were to feel less anxious at work and happier after work hours. The therapist mentioned that Kyle’s PHQ-9 score was 18 and that his GAD-7 score was 15, indicating that his depression and anxiety were both in the severe range [10,11]. Since Kyle’s insurance covered only 12 sessions, the therapist explained that she would provide cognitive behavioral therapy, focusing on his interpretations of ongoing events and how these beliefs affected his emotions and behaviors. Given the severity of his symptoms, she also referred him for medication evaluation.

After their second session, Dr. Davis reviewed the reports she received from Eleos, the therapeutic intelligence platform she had been using. She was surprised to see that in their last session, she spoke in only 20% of the session and that she was speaking much slower than Kyle (Figure 1). The speech rate difference indicated to Dr. Davis that Kyle was speaking hurriedly and reflected her attempt to “slow him down” to reflect on his maladaptive assumptions and ingrained interactional patterns. However, she was wondering whether she could share these observations with Kyle more explicitly. These data helped the therapist understand that Kyle was experiencing significant mood, anxiety, and stress symptoms, but that in order to help Kyle achieve his goals, their synchrony during the session would have to be substantially improved [35]. A review of the process metrics provided insight into the issues troubling Kyle and the inaccurate beliefs and maladaptive information processing strategies that precipitated and maintained his cognitions [42]. A review of his most frequently used words indicated that he tended to use verbs, adjectives, and expressions associated with negative self-esteem, such as “failed,” “not as good as,” “disappointed,” and “messed it up,” when he talked about his work. However, when he talked about his relationships, he tended to use anxiety words, such as “stressed,” “pressured,” “overwhelmed,” and “toxic.” The system flagged these words and phrases, as they approximate depressive and anxiety symptoms and may reflect Kyle’s subjective experience, thereby enriching the self-report data collected between sessions. In addition, 3 of Kyle’s most used phrases were “shoulds” (eg, “have to,” “I must,” “should have known better”), which the system automatically analyzed and flagged. The therapist realized that Kyle was experiencing 2 distinct phenomena: At work, he felt like an imposter and worried about his functioning, whereas outside of work, he was distressed by blurred boundaries in his interpersonal relationships. Dr. Davis also realized that she was using little Socratic questioning, which may have reduced Kyle’s ability to re-examine his assumptions and information-processing skills [43]. However, she did observe her use of reflective listening methods often, mirroring and reframing what Kyle had said, which she was content with [44,45]. Dr. Davis brought the case to her weekly group consultation meeting and received feedback and advice from her peers that she intended to implement in the next few sessions.

Figure 1. Screenshots illustrating some of the Eleos Health platform process features. CBT: cognitive behavioral therapy; Speaking Ratio: proportion (%) of time spent speaking during the session; Speech Rate: speed of words per minute; Techniques Used: intervention strategies employed by the therapist during the session and automatically identified by the platform. Techniques used 3 times or more are indicated with a black checkmark, techniques used once or twice are denoted with a grey checkmark, and interventions not employed in the session do not have a checkmark.

In the following meeting, Dr. Davis shared with Kyle the insights she had gleaned from the platform. She described the 2 clinical issues—his depression regarding work and his anxiety regarding relationships—and asked him to prioritize his goals. Dr. Davis’s presentation of his challenges helped Kyle reflect on them, and he decided to first focus on his self-esteem at work. He learned how to observe, identify, and challenge negative perceptions of himself, and reported very quick improvements in his mood. Next, he was able to undertake the same process regarding several relationships that he felt had not been reciprocal and gratifying. He was pleased with his progress in treatment and felt happier and more relaxed. Kyle also liked that he could complete short assessments on his phone and got into the habit of doing this on his commute home from work when he had a few minutes to spare. Similarly, Dr. Davis appreciated the symptom-tracking feature, which let both her and her patient easily see how he was doing symptom-wise (Figure 2). In the session analytics reports, Dr. Davis also observed an improvement in the therapeutic alliance: She was able to incorporate more open-ended questions, Kyle was more receptive of her questions and comments, and their speech rate was more in sync with one another (Figure 1).

Figure 2. Patient self-monitoring data graphed on the Eleos platform. GAD-7: Generalized Anxiety Disorder-7; PHQ-9: Patient Health Questionnaire-9.

After 10 sessions, Kyle was not only feeling less depressed and anxious but was also receiving praise from his coworkers and his boss. He was able to strengthen his friendships and practice new skills for boundary-setting and interpersonal effectiveness. As treatment termination neared, the last sessions focused on brainstorming strategies for maintaining his progress, particularly since the workload at his company continued to increase.

Psychological interventions for mental disorders were found effective in numerous research trials. However, gaps in training, availability, access, dissemination, and cost impede the successful delivery of these interventions [46]. These are enhanced by privacy and data storage regulations, which are key for maintaining clients’ rights and trust [47]. The ever-growing demand for mental health care requires optimizing clinician decision-making using data collected passively [48]. Our goal was to highlight key issues for optimizing available mental health services and to demonstrate how a therapeutic intelligence platform can support this process.

This paper presents a novel approach to collecting comprehensive data on treatment progress. The implementation of ML models and AI in behavioral health care is a rapidly moving and innovative field, with the potential to significantly improve screening and clinical outcomes [49,50]. Accurate data can provide more information about the patient and can be translated into clinical decisions faster. Experts strive to base their decisions on data. Primary physicians, heart surgeons, physiotherapists, and other health professionals all function better when they review the patient's most recent tests and laboratory results prior to the appointment. Therefore, using real-time data is equally important in enhancing the work of mental health professionals [51]. Technologies that are scalable, cost-effective, and that enhance quality without burden can help therapists harness their efforts into providing more effective interventions with increased fidelity to data-informed treatments. Setting out clear performance targets in the training, provision, and implementation of evidence-enhanced treatments will enable health care services to continually improve. Nondisruptive measures are poised to ensure nearly effortless data collection, and innovative methods that inform clinicians and other stakeholders of the patient’s progress will likely make treatments more relevant and engaging [52].

Some caveats to the model described here should be mentioned. The Eleos platform was illustrated through its use in an outpatient setting, with a client who was relatively high functioning and a therapist that was tech-savvy and interested in using novel digital programs. Usability testing, user reviews, and long-term engagement with any product are key to realizing the practicality and helpfulness of new tools over time [53]. Further, evidence from more case reports, randomized controlled trials, and meta-analyses is needed to render these technologies pertinent, empirically supported, and easily applied in clinical settings.

The effective implementation of mental health care requires new approaches for developing, implementing, and evaluating interventions. A person-centered approach that capitalizes on greater data insights will certainly enhance the therapeutic process. Technology can help make efforts in this direction scalable and more efficient, thereby increasing the effects of behavioral interventions and reducing the burden of mental health problems worldwide.


The authors would like to thank the clinicians and patients who have been piloting the Eleos platform. We also thank patient “Kyle,” who read and approved this case illustration.

Conflicts of Interest

SSS is the Chief Clinical Officer of the commercial entity Eleos Health Inc that created the platform that is the subject of this case report. SDH is an unpaid advisor to Eleos Health Inc.

  1. Trautmann S, Rehm J, Wittchen H. The economic costs of mental disorders: Do our societies react appropriately to the burden of mental disorders? EMBO Rep. Sep 04, 2016;17(9):1245-1249. [FREE Full text] [CrossRef] [Medline]
  2. Ewbank MP, Cummins R, Tablan V, Bateup S, Catarino A, Martin AJ, et al. Quantifying the Association Between Psychotherapy Content and Clinical Outcomes Using Deep Learning. JAMA Psychiatry. Jan 01, 2020;77(1):35-43. [FREE Full text] [CrossRef] [Medline]
  3. Kazdin A. Addressing the treatment gap: A key challenge for extending evidence-based psychosocial interventions. Behav Res Ther. Jan 2017;88:7-18. [FREE Full text] [CrossRef] [Medline]
  4. Sadeh-Sharvit S. Use of Technology in the Assessment and Treatment of Eating Disorders in Youth. Child Adolesc Psychiatr Clin N Am. Oct 2019;28(4):653-661. [CrossRef] [Medline]
  5. Topol E. Deep medicine: how artificial intelligence can make healthcare human again. New York, NY. Basic Books; 2019.
  6. Scott K, Lewis CC. Using Measurement-Based Care to Enhance Any Treatment. Cogn Behav Pract. Feb 2015;22(1):49-59. [FREE Full text] [CrossRef] [Medline]
  7. Fortney JC, Unützer J, Wrenn G, Pyne JM, Smith GR, Schoenbaum M, et al. A Tipping Point for Measurement-Based Care. PS. Feb 2017;68(2):179-188. [CrossRef]
  8. Nasrallah H, Aboraya A. It's time to implement measurement-based care in psychiatric practice. Current Psychiatry. 2019;18(6):6-8. [FREE Full text]
  9. Guo T, Xiang Y, Xiao L, Hu C, Chiu HFK, Ungvari GS, et al. Measurement-Based Care Versus Standard Care for Major Depression: A Randomized Controlled Trial With Blind Raters. Am J Psychiatry. Oct 2015;172(10):1004-1013. [CrossRef] [Medline]
  10. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [FREE Full text] [CrossRef] [Medline]
  11. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. May 22, 2006;166(10):1092-1097. [CrossRef] [Medline]
  12. Darcy A, Sadeh-Sharvit S. Mobile Device Applications for the AssessmentTreatment of Eating Disorders. In: Agras SW, Robinson A, editors. The Oxford Handbook of Eating Disorders, Second Edition. Oxford, UK. Oxford University Press; 2017:492-504.
  13. Anatchkova M, Donelson SM, Skalicky AM, McHorney CA, Jagun D, Whiteley J. Exploring the implementation of patient-reported outcome measures in cancer care: need for more real-world evidence results in the peer reviewed literature. J Patient Rep Outcomes. Dec 27, 2018;2(1):64. [FREE Full text] [CrossRef] [Medline]
  14. Haun JN, Alman AC, Melillo C, Standifer M, McMahon-Grenz J, Shin M, et al. Using Electronic Data Collection Platforms to Assess Complementary and Integrative Health Patient-Reported Outcomes: Feasibility Project. JMIR Med Inform. Jun 26, 2020;8(6):e15609. [CrossRef]
  15. Cuijpers P, Li J, Hofmann SG, Andersson G. Self-reported versus clinician-rated symptoms of depression as outcome measures in psychotherapy research on depression: a meta-analysis. Clin Psychol Rev. Aug 2010;30(6):768-778. [CrossRef] [Medline]
  16. Briffault X, Morgiève M, Courtet P. From e-Health to i-Health: Prospective Reflexions on the Use of Intelligent Systems in Mental Health Care. Brain Sci. May 31, 2018;8(6):98. [FREE Full text] [CrossRef] [Medline]
  17. Crowe J. Reform, revolution and disruption in mental health care: a consumer's perspective. Public Health Res Pract. Apr 27, 2017;27(2). [FREE Full text] [CrossRef] [Medline]
  18. Piette JD, Krein SL, Striplin D, Marinec N, Kerns RD, Farris KB, et al. Patient-Centered Pain Care Using Artificial Intelligence and Mobile Health Tools: Protocol for a Randomized Study Funded by the US Department of Veterans Affairs Health Services Research and Development Program. JMIR Res Protoc. Apr 07, 2016;5(2):e53. [FREE Full text] [CrossRef] [Medline]
  19. Spanakis EG, Santana S, Tsiknakis M, Marias K, Sakkalis V, Teixeira A, et al. Technology-Based Innovations to Foster Personalized Healthy Lifestyles and Well-Being: A Targeted Review. J Med Internet Res. Jun 24, 2016;18(6):e128. [FREE Full text] [CrossRef] [Medline]
  20. Horgan D, Romao M, Morré SA, Kalra D. Artificial Intelligence: Power for Civilisation - and for Better Healthcare. Public Health Genomics. 2019;22(5-6):145-161. [FREE Full text] [CrossRef] [Medline]
  21. Atkins DC, Steyvers M, Imel ZE, Smyth P. Scaling up the evaluation of psychotherapy: evaluating motivational interviewing fidelity via statistical text classification. Implement Sci. Apr 24, 2014;9:49. [FREE Full text] [CrossRef] [Medline]
  22. Xiao B, Imel ZE, Georgiou PG, Atkins DC, Narayanan SS. "Rate My Therapist": Automated Detection of Empathy in Drug and Alcohol Counseling via Speech and Language Processing. PLoS One. 2015;10(12):e0143055. [FREE Full text] [CrossRef] [Medline]
  23. Imel ZE, Steyvers M, Atkins DC. Computational psychotherapy research: scaling up the evaluation of patient-provider interactions. Psychotherapy (Chic). Mar 2015;52(1):19-30. [FREE Full text] [CrossRef] [Medline]
  24. Gyani A, Shafran R, Layard R, Clark DM. Enhancing recovery rates: lessons from year one of IAPT. Behav Res Ther. Sep 2013;51(9):597-606. [FREE Full text] [CrossRef] [Medline]
  25. Clark DM, Fairburn CG, Wessely S. Psychological treatment outcomes in routine NHS services: a commentary on Stiles et al. (2007). Psychol Med. May 2008;38(5):629-634. [FREE Full text] [CrossRef] [Medline]
  26. Macaskill ND. Improving clinical outcomes in REBT/CBT: The therapeutic uses of tape-recording. J Rational-Emot Cognitive-Behav Ther. Sep 1996;14(3):199-207. [CrossRef]
  27. Barnett JE. Utilizing technological innovations to enhance psychotherapy supervision, training, and outcomes. Psychotherapy (Chic). Jun 2011;48(2):103-108. [CrossRef] [Medline]
  28. Haggerty G, Hilsenroth M. The use of video in psychotherapy supervision. British Journal of Psychotherapy. 2011/05/. 2011;27(2):01-210. [CrossRef]
  29. Shafranske E, Falender C. Clinical supervision. APA handbook of clinical psychologyducation and profession, Vol 5. American Psychological Association. APA handbooks in psychology®. 2016:175-196. [CrossRef]
  30. Shepherd L, Salkovskis PM, Morris M. Recording therapy sessions: an evaluation of patient and therapist reported behaviours, attitudes and preferences. Behav Cogn Psychother. Mar 2009;37(2):141-150. [CrossRef] [Medline]
  31. Briggie A, Hilsenroth M, Conway F, Muran J, Jackson J. Patient comfort with audio or video recording of their psychotherapy sessions: Relation to symptomatology, treatment refusal, duration, and outcome. Professional Psychology: Research and Practice. 2016;47(1):66-76. [CrossRef]
  32. Singla DR, Hollon SD. The importance of implementation science in scaling up psychological treatments. Behav Res Ther. Jul 2020;130:103614. [CrossRef] [Medline]
  33. Kalmady SV, Greiner R, Agrawal R, Shivakumar V, Narayanaswamy JC, Brown MRG, et al. Towards artificial intelligence in mental health by improving schizophrenia prediction with multiple brain parcellation ensemble-learning. NPJ Schizophr. Jan 18, 2019;5(1):2. [FREE Full text] [CrossRef] [Medline]
  34. Hill CE, Thompson BJ, Ladany N. Therapist use of silence in therapy: a survey. J Clin Psychol. Apr 2003;59(4):513-524. [CrossRef] [Medline]
  35. Sexton H, Littauer H, Sexton A, Tømmerås E. Building an alliance: Early therapy process and the client–therapist connection. Psychotherapy Research. Jan 2005;15(1-2):103-116. [CrossRef]
  36. Hollon S. Developing clinical practice guidelines to enhance clinical decision making. In: Magnavita JJ, editor. Clinical decision making in mental health practice. Arlington, VA. American Psychological Association; 2016:125-146.
  37. McCabe R, Howes C, Purver M. Linguistic indicators of severity and profess in online text-based therapy for depression. 2014. Presented at: Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality; 2014:7-16; Baltimore, Maryland, USA. URL: [CrossRef]
  38. Funk B, Sadeh-Sharvit S, Fitzsimmons-Craft EE, Trockel MT, Monterubio GE, Goel NJ, et al. A Framework for Applying Natural Language Processing in Digital Health Interventions. J Med Internet Res. Feb 19, 2020;22(2):e13855. [FREE Full text] [CrossRef] [Medline]
  39. Plateau C, Brookes F, Pugh M. Guided Recovery: An Interpretative Phenomenological Analysis of Service Users’ Experiences of Guided Self-Help for Bulimic and Binge Eating Disorders. Cognitive and Behavioral Practice. May 2018;25(2):310-318. [FREE Full text] [CrossRef]
  40. Garland AF, Kruse M, Aarons GA. Clinicians and outcome measurement: what's the use? J Behav Health Serv Res. 2003;30(4):393-405. [CrossRef] [Medline]
  41. Fernandes J, Brunton I, Strudwick G, Banik S, Strauss J. Physician experience with speech recognition software in psychiatry: usage and perspective. BMC Res Notes. Oct 01, 2018;11(1):1-5. [FREE Full text] [CrossRef] [Medline]
  42. Hollon S, DeRubeis R. Outcome studies in cognitive therapy. In: Leahy RL, editor. Science and Practice in Cognitive Therapy: Foundations, Mechanisms, and Applications. New York, NY. Guilford Press; 2018:48-61.
  43. Clark GI, Egan SJ. The Socratic Method in Cognitive Behavioural Therapy: A Narrative Review. Cognitive Therapy and Research. Jul 25, 2015;39(6):863-879. [CrossRef]
  44. Can D, Marín RA, Georgiou PG, Imel ZE, Atkins DC, Narayanan SS. "It sounds like...": A natural language processing approach to detecting counselor reflections in motivational interviewing. J Couns Psychol. Apr 2016;63(3):343-350. [FREE Full text] [CrossRef] [Medline]
  45. Thwaites R, Bennett-Levy J. Conceptualizing Empathy in Cognitive Behaviour Therapy: Making the Implicit Explicit. Behav. Cognit. Psychother. Jul 6, 2007;35(05). [CrossRef]
  46. Firth J, Torous J, Carney R, Newby J, Cosco TD, Christensen H, et al. Digital Technologies in the Treatment of Anxiety: Recent Innovations and Future Directions. Curr Psychiatry Rep. May 19, 2018;20(6):44. [FREE Full text] [CrossRef] [Medline]
  47. Lattie EG, Nicholas J, Knapp AA, Skerl JJ, Kaiser SM, Mohr DC. Opportunities for and Tensions Surrounding the Use of Technology-Enabled Mental Health Services in Community Mental Health Care. Adm Policy Ment Health. Jan 2020;47(1):138-149. [CrossRef] [Medline]
  48. Delgadillo J, Gonzalez Salas Duhne P. Targeted prescription of cognitive-behavioral therapy versus person-centered counseling for depression using a machine learning approach. J Consult Clin Psychol. Jan 2020;88(1):14-24. [CrossRef] [Medline]
  49. Sadeh-Sharvit S, Fitzsimmons-Craft EE, Taylor CB, Yom-Tov E. Predicting eating disorders from Internet activity. Int J Eat Disord. Sep 2020;53(9):1526-1533. [CrossRef] [Medline]
  50. Chien I, Enrique A, Palacios J, Regan T, Keegan D, Carter D, et al. A Machine Learning Approach to Understanding Patterns of Engagement With Internet-Delivered Mental Health Interventions. JAMA Netw Open. Jul 01, 2020;3(7):e2010791. [FREE Full text] [CrossRef] [Medline]
  51. Colombo D, Suso-Ribera C, Fernandez-Álvarez J. Exploring Affect Recall Bias and the Impact of Mild Depressive Symptoms: An Ecological Momentary Study. In: Cipresso P, Serino S, Villani D, editors. Pervasive Computing Paradigms for Mental Health. Cham. Springer; 2019:A.
  52. Jacobson NC, Chung YJ. Passive Sensing of Prediction of Moment-To-Moment Depressed Mood among Undergraduates with Clinical Levels of Depression Sample Using Smartphones. Sensors (Basel). Jun 24, 2020;20(12). [FREE Full text] [CrossRef] [Medline]
  53. Fuller-Tyszkiewicz M, Richardson B, Klein B, Skouteris H, Christensen H, Austin D, et al. A Mobile App-Based Intervention for Depression: End-User and Expert Usability Testing Study. JMIR Ment Health. Aug 23, 2018;5(3):e54. [FREE Full text] [CrossRef] [Medline]

AI: artificial intelligence
GAD-7: Generalized Anxiety Disorder-7
MBC: measurement-based care
ML: machine learning
NLP: natural language processing
PHQ-9: Patient Health Questionnaire-9

Edited by G Eysenbach; submitted 24.05.20; peer-reviewed by K McKay, C Kaylor-Hughes, A Baumel; comments to author 15.06.20; revised version received 24.07.20; accepted 28.10.20; published 26.11.20.


©Shiri Sadeh-Sharvit, Steven D Hollon. Originally published in JMIR Mental Health (, 26.11.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.