Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 26.02.20 in Vol 7, No 2 (2020): February

Preprints (earlier versions) of this paper are available at, first published Oct 23, 2019.

This paper is in the following e-collection/theme issue:


    Workshop on the Development and Evaluation of Digital Therapeutics for Health Behavior Change: Science, Methods, and Projects

    1Center for Technology and Behavioral Health, Geisel School of Medicine, Dartmouth College, Lebanon, NH, United States

    2Department of Psychiatry, Geisel School of Medicine, Dartmouth College, Lebanon, NH, United States

    3Department of Biomedical Data Science, Geisel School of Medicine, Dartmouth College, Lebanon, NH, United States

    4Clinical Research Grants Branch, National Institute on Drug Abuse, National Institutes of Health, Rockville, MD, United States

    5Department of Psychiatry, Washington University School of Medicine, St Louis, MO, United States

    6Department of Computer Science, Dartmouth College, Hanover, NH, United States

    7Department of Psychology, University of Florida, Gainesville, FL, United States

    8Center for Behavioral Intervention Technologies, Feinberg School of Medicine, Northwestern University, Chicago, IL, United States

    9Department of Psychiatry, New York State Psychiatric Institute, Columbia University, New York, NY, United States

    Corresponding Author:

    Alan J Budney, PhD

    Center for Technology and Behavioral Health

    Geisel School of Medicine

    Dartmouth College

    Suite 315

    46 Centerra Parkway

    Lebanon, NH, 03755

    United States

    Phone: 1 603 646 7000



    The health care field has integrated advances into digital technology at an accelerating pace to improve health behavior, health care delivery, and cost-effectiveness of care. The realm of behavioral science has embraced this evolution of digital health, allowing for an exciting roadmap for advancing care by addressing the many challenges to the field via technological innovations. Digital therapeutics offer the potential to extend the reach of effective interventions at reduced cost and patient burden and to increase the potency of existing interventions. Intervention models have included the use of digital tools as supplements to standard care models, as tools that can replace a portion of treatment as usual, or as stand-alone tools accessed outside of care settings or direct to the consumer. To advance the potential public health impact of this promising line of research, multiple areas warrant further development and investigation. The Center for Technology and Behavioral Health (CTBH), a P30 Center of Excellence supported by the National Institute on Drug Abuse at the National Institutes of Health, is an interdisciplinary research center at Dartmouth College focused on the goal of harnessing existing and emerging technologies to effectively develop and deliver evidence-based interventions for substance use and co-occurring disorders. The CTBH launched a series of workshops to encourage and expand multidisciplinary collaborations among Dartmouth scientists and international CTBH affiliates engaged in research related to digital technology and behavioral health (eg, addiction science, behavioral health intervention, technology development, computer science and engineering, digital security, health economics, and implementation science). This paper summarizes a workshop conducted on the Development and Evaluation of Digital Therapeutics for Behavior Change, which addressed (1) principles of behavior change, (2) methods of identifying and testing the underlying mechanisms of behavior change, (3) conceptual frameworks for optimizing applications for mental health and addictive behavior, and (4) the diversity of experimental methods and designs that are essential to the successful development and testing of digital therapeutics. Examples were presented of ongoing CTBH projects focused on identifying and improving the measurement of health behavior change mechanisms and the development and evaluation of digital therapeutics. In summary, the workshop showcased the myriad research targets that will be instrumental in promoting and accelerating progress in the field of digital health and health behavior change and illustrated how the CTBH provides a model of multidisciplinary leadership and collaboration that can facilitate innovative, science-based efforts to address the health behavior challenges afflicting our communities.

    JMIR Ment Health 2020;7(2):e16751




    Over the last 25 years, scientists in the fields of addiction science and mental health have studied how to enhance treatment outcomes, how to improve the dissemination and implementation of effective treatments, and how to reach a greater proportion of the population experiencing behavioral health challenges. Barriers to success have included substandard screening and identification of cases, limited numbers of trained personnel, high burden associated with training, poor fidelity and integrity of treatment delivery, low pay for providers, high costs for delivering the most effective interventions, stigma and burden associated with seeking treatment, and limited individualization and personalization of treatment.

    Broadly speaking, the health care field has integrated numerous advances in digital technology at an accelerating pace to improve health behavior, health care delivery, and cost-effectiveness of care. This area of science, often referred to as digital health, has been embraced within the realm of behavioral science, allowing for an exciting roadmap for advancing care by addressing the many challenges to the field via technological innovations [1-4]. Technology-delivered treatments, commonly referred to as digital therapeutics, offer the potential to extend the reach of evidence-based interventions at reduced cost and patient burden and to increase the potency of existing interventions. Delivering on these promises has substantial public health implications, given that the majority of people with behavioral health needs are not receiving treatment of any kind, and only the minority of those who do receive treatment are provided with an evidence-based intervention. Multiple studies have demonstrated that treatment models assisted by computer-, Web-, or smartphone-based delivery of evidence-based interventions improve access to care, reduce costs, and improve the efficiency of treatment delivery [1-4]. Intervention models have included the use of digital tools as supplements to standard care models, as tools that can replace a portion of treatment as usual, or as stand-alone tools accessed outside of care settings or direct to the consumer. Indeed, digital therapeutics have progressed to such an extent that the US Food and Drug Administration (FDA) recently launched an initiative to incorporate digital health into mainstream medical practice [5]. This significant step acknowledges the potential of and the growing evidence for digital therapeutics. The FDA has further defined a class of digital therapeutics as mobile medical applications, software apps that meet the definition of a medical device, that is, performing patient-specific analysis and providing patient-specific diagnosis or treatment recommendations, and at least one mobile app has received FDA permission for marketing [6,7].

    However, to further advance the potential public health impact of this promising line of research, several areas warrant development and investigation. Although some digital therapeutics for health behavior problems have been deemed effective, effect sizes to date have been generally small to medium, and treatment mechanisms have been understudied. To address this problem, it is essential to focus more research on identifying and understanding specific mechanisms relevant to health behavior change. Importantly, direct comparisons between digital therapeutics and well-understood traditionally delivered interventions may provide the clues necessary for achieving this goal and developing more potent interventions [8]. Given their on-demand availability and opportunities for personalization, digital therapeutics may impact the rate and nature by which health behavior change occurs. An increased understanding of the active ingredients and mediators of outcome from such interventions can greatly enhance future efforts to optimize development efforts. Such progress could move the digital health field beyond direct adaptations of existing face-to-face interventions to better harness the dynamic potential of technology in collecting data about individuals in unprecedented ways (and in real time) and in intervening at the exact moments when individuals may be maximally motivated and receptive [9].

    Optimization and acceleration of the development and evaluation of digital therapeutics also require advances in the measures and methods used to study treatment processes, mechanisms, and outcomes. Understanding how an intervention produces changes in cognitive, behavioral, or environmental factors that are then, in turn, causally related to health behavior change (eg, reduction in drug use) allows for the identification of more specific treatment targets, which, in turn, permit refinement and optimization of subsequent iterations of an intervention [8]. Continued development of more granular, objective, and valid measures of important mechanism constructs and processes and treatment outcome variables is also essential for the advancement of health behavior change research and development. Technological and data analytic advances offer exciting new approaches for obtaining and interpreting more meaningful and ecologically valid data that can be readily collected in the natural environment over substantial periods. Finally, the pace of science in the realm of digital health can greatly benefit from explicit methodological frameworks to guide research efforts and the application of innovative experimental designs. Several novel and underused methodological approaches are relevant to developing and evaluating digital therapeutics that can supplement the findings from traditional randomized controlled trials (RCTs) [10-13]. Such approaches include flexible, rigorous, and efficient methods, which can help optimize interventions and bridge the gap between clinical trials and community care.

    Center for Technology and Behavioral Health—Workshop

    The Center for Technology and Behavioral Health (CTBH; a P30 Center of Excellence supported by the National Institute on Drug Abuse [NIDA] at the National Institutes of Health) is an interdisciplinary research center at Dartmouth College focused on the goal of harnessing existing and emerging technologies to effectively develop and deliver evidence-based interventions for substance use and co-occurring disorders [14]. In total, 3 primary cores, treatment development and evaluation (TDE), emerging technologies and data analytics, and dissemination and implementation, bring together a diverse team with expertise in addiction science, behavioral health intervention, technology development, computer science and engineering, digital security, health economics, and implementation science.

    The TDE core of the CTBH organized a workshop, “Development and Evaluation of Digital Therapeutics for Behavior Change: Science, Methods, and CTBH Projects,” which was recently held on the campus of Dartmouth College. This was one of the series of workshops launched by the CTBH to encourage and expand collaboration among Dartmouth scientists and international CTBH affiliates across the diverse disciplines that engage in research related to digital technology and behavioral health. The goals of the workshop were to (1) highlight key conceptual and scientific issues vital to the proliferation of research on digital therapeutics, (2) stimulate new collaborations and scientific developments, and (3) showcase the work of CTBH-TDE core investigators. The workshop theme was the Science of Behavior Change. More than 50 faculty, students, trainees, and staff attended this all-day meeting in person, and collaborative partners from a number of US institutions and ETH Zürich’s Center for Digital Health Interventions attended remotely. This paper summarizes the workshop sessions, which addressed (1) principles of behavior change; (2) conceptual frameworks for optimizing applications in mental health, addictive behavior, and health behavior change; and (3) the diversity of experimental methods and designs that are essential to the successful development and testing of digital therapeutics for behavior change. Most sessions also provided examples of ongoing projects focused on identifying and improving the measurement of health behavior change mechanisms and the development and evaluation of digital therapeutics. Our goal here was to use the knowledge gleaned during this workshop to provide a broad overview of the wide-ranging research topics that must be addressed to advance the field of digital health and its impact on health behavior problems and to illustrate how our CTBH model can facilitate progress in meeting the vast health challenges facing behavioral health research and service delivery models.

    Science of Behavior Change

    WA, the Director of the NIDA’s Behavioral Therapy Development Program in the Clinical Research Grants Division, provided opening comments and highlighted NIDA’s keen interest in research on behavior change and mechanisms of change in research on addictions. He emphasized how science that targets potential change mechanisms, such as self-regulation, stress, resilience, and intra- and interpersonal processes, can have a ubiquitous impact on our ability to provide more focused and parsimonious treatments, that is, personalized medicine. WA concluded with a discussion of how the integration of technology-based treatments can uniquely target and facilitate change in these mechanisms and thereby advance the development of effective interventions for substance use and associated psychiatric disorders.

    As the workshop audience comprised a diverse group of multidisciplinary scientists who affiliate themselves with the CTBH, AB began with an overview of key principles of behavior change, from basic theories of learning (eg, Skinnerian principles of reinforcement) to current behavioral economic concepts that are highly relevant to identifying and targeting effective behavioral health change mechanisms. This presentation illustrated how understanding the basic elements of behavior change principles can be translated into personalized, effective treatment elements. As examples, AB described how novel translations of behavior analytic principles of reinforcement and the construct of temporal discounting have been applied to inform the development of innovative treatment strategies that can enhance treatment outcomes for substance use disorders and other health behavior problems [15-17]. He also shared other heuristic behavioral economic principles that are currently being transformed into tools designed to optimize health behavior change [17] and discussed the many ways that technology can be leveraged to facilitate and accelerate the creation and dissemination of effective behavior change interventions.

    JD then initiated a series of presentations that underscored the importance of using diverse and innovative methodologies and experimental designs to advance research on digital technology–based treatments. He introduced and discussed the key role that single-case experimental designs (SCEDs) can play in stage 1 testing of technology-based interventions to promote health behavior [18,19]. He illustrated how SCEDs can rigorously and efficiently answer questions about the preliminary efficacy and acceptability of an intervention. He reviewed the essential methodological elements of SCEDs and used examples from his published research on technology-based methods to promote smoking cessation to illustrate these elements [20]. In addition, he discussed how SCEDs can be employed to evaluate potential mechanisms underlying intervention efficacy. Specifically, mechanisms can be assessed by obtaining a time series of changes in measures of the mechanism construct and the outcome and by experimentally manipulating the mechanism using SCEDs. JD concluded by noting some of the practical and scientific advantages of SCEDs and how their use encourages data intimacy, scientific rigor, and innovation.

    CS’s presentation alerted the audience to several emerging research methods and designs that may be particularly useful in evaluating and understanding the impact of technology-based tools on health behavior and that can supplement the findings from RCTs (eg, the Multiphase Optimization Strategy and Sequential Multiple Assignment Randomized Trial [SMART]) [21-23]. CS focused on SMART designs, which involve random assignment of individuals to conditions more than once during a study, based on their response to conditions experienced earlier in the study. As SMART designs involve multiple randomizations over time, they test the impact of applying different intervention approaches at critical decision points based on intervention response. CS presented an example of a recently completed SMART trial for adolescent cannabis use [24]. Using a SMART design, this study tested the impact of an initial intervention (adding working memory training to contingency management [CM] for youth with cannabis use disorder at treatment onset) and an adaptive intervention (switching youth who did not show a positive response after the first month of the intervention to higher magnitude CM incentives) on clinical outcomes. CS summarized how the SMART design fits particularly well with digital interventions because rapid assessment of changing (dynamic) predictors of intervention response and adaptive changes in intermediate outcomes in response to interventions are facilitated by the technology. This effort may lead to technology-based tools that can be readily tailored to optimally meet the needs of an individual [25].

    Mechanism of Action and Measurement

    The workshop shifted focus with LM’s presentation on targeting mechanisms of action with digital therapeutics, in this case, self-regulation. She first raised awareness of the ubiquitous contribution and challenges that health risk behavior and poor adherence to medical regimens impart on chronic disease development and its management. Self-regulation, a person’s ability to manage cognitive, motivational, and emotional resources to act in accordance with his or her long-term goals, was introduced as an important causal mechanism of health behavior, and deficient self-regulation was proposed as a key target for interventions addressing health risk behavior in chronic diseases. LM outlined existing but disparate literature that highlights the clear promise of interventions that target self-regulation to improve health and the challenges that must be addressed if we are to most effectively address this transdisease process of self-regulation [12]. Examples were provided of how digital technology and data analytics have created unprecedented opportunities to assess and modify self-regulation and thus accelerate scientific understanding and application. Intensive and continuous individual data collection with mobile devices (or passive sensors and digital footprints of Web-based social media) can provide rich personal and environmental data in real time, which sophisticated data analytics can turn into meaningful insights about individual health and inform person-centered adaptive interventions.

    LM discussed an ongoing National Institutes of Health funded ontology of self-regulation project that involves the development of optimal measures of self-regulation using the aforementioned digital technology and data analytic methods and evaluation of how the mechanism of self-regulation relates to behavior change across 2 clinical populations (heavy smokers and persons with binge eating disorder) [12]. Participants will use a mobile health self-regulation platform that provides science-based behavior change tools to promote improved self-regulation and health behavior and facilitate real-time data collection.

    ES provided more detail on the first stage of this project, that is, the application of data analytics to the development of a momentary measure of self-regulation that can be used to effectively evaluate change constructs in naturalistic settings via mobile devices in trials using digital therapeutics to assess and treat health behavior problems. She described the iterative process and innovative data analytics involved in empirically extracting the most informative items derived from the multitude of items contained within the many existing self-regulation assessments to efficiently capture momentary self-regulation status with a brief measure in longitudinal studies. ES showed data from an initial piloting of this 20-item measure using ecological momentary assessment (3 times per day surveys for 2 weeks). Her team used multilevel factor analysis to select 12 items that represented 4 valid self-regulation subscales (perseverance, self-judgment, sensation seeking, and mindfulness). CTBH affiliates have included this momentary measure to assess changes in self-regulation in 4 clinical studies.

    The next 2 presentations illustrated the novel advances in measurement that passive sensing can bring to the study of behavioral health by providing examples of how mobile sensing can be used to identify temporal fluctuations in high-risk emotional states (eg, depression and stress). AC described his ongoing project focused on the high rates of depression and anxiety among college students. His study used passive sensing features of a smartphone and a commodity wrist-worn wearable (Microsoft Band2; Microsoft, WA) to monitor multiple behavioral and physiological correlates of depression over time (eg, activity level, conversation frequency, and sleep duration) [26]. These devices monitored location, phone usage, light and sound detection, activity duration (steps), heart rate, skin temperature, and galvanic skin response over 10 weeks, and text messages prompted collection of weekly patient health questionnaire depression scores and other experience sampling responses. Data analytic models using all these data and pre-post surveys that assessed stress, anxiety, productivity, and other relevant behavior or emotions were combined to determine how the passive sensing measures related to the students’ mental health functioning across the semester.

    In an excellent example of cross-discipline collaboration, SL presented results from a proof-of-concept pilot study conducted with members of Dartmouth’s Computer Science and Engineering Sciences Departments to evaluate the reliability and acceptability of a wearable sensor system (called the Amulet Sensor System, a wrist-worn device developed at Dartmouth) to passively identify stress in a college student population. The sensor system included a commodity chest strap heart rate monitor to collect heart rate variability data, an electrodermal activity sensor to assess skin conductance data, and the Amulet that acted as a data hub for data from the heart rate monitor. The Amulet included an accelerometer and software to conduct an ecological momentary assessment of stress constructs as well as an event mark feature to benchmark high-stress periods. SL discussed results about the feasibility of using commodity products for passive detection of stress and the potential of passive sensing techniques as the first line of assessment to trigger just-in-time interventions [27]. The discussion highlighted the importance and roles of multidisciplinary team of scientists from the fields of behavioral health, computer science, and engineering in the development and evaluation of digital approaches to behavioral health.

    Clinical Application

    The workshop then shifted to a more clinical focus as EN discussed the problem of intervention adherence in clinical trials using digital health tools, an issue similar to that which occurs with traditional behavioral and pharmacological interventions. He provided examples about how some technology-based interventions can reduce dropout compared with traditional therapist-delivered interventions for substance use disorders but noted that both approaches still have high rates of attrition. EN highlighted that more attention to increased production value, that is, more consumer-friendly user interface and user design, or gamification, could help address this vital adherence issue with behavioral health populations. He provided 2 examples of efforts in this direction, one that included high-quality video demonstrations to enhance development of coping skills in the context of treating substance use disorders [28] and one that is using gamification within a digital therapeutic for treating opioid use disorder [29], but he stressed the need to empirically test the assumption that these innovations positively impact adherence. EN showed data illustrating how a digital therapeutic tool had improved adherence to a buprenorphine medication regimen in the treatment of opioid dependence [30] and concluded with a discussion of other ideas for how technology could enhance treatment adherence, for example, remote camera (smartphone) monitoring and confirmation, facial recognition software combined with pill ingestion, automated messaging, and motivational prompts.

    The following series of presentations described ongoing digital health projects of the CTBH faculty and affiliates. Each of them highlighted behavior change methods, experimental design features, and potential next steps in their development efforts. First, MB described the development and testing of a mobile intervention to deliver motivational education for smoking cessation tailored for young adults with serious mental illness. She discussed the importance of reducing and quitting smoking to prevent the disparate chronic diseases in this vulnerable population and how design features of digital therapeutics need to be tailored to the special needs of the clinical population to optimize potential efficacy [31]. Her initial pilot study demonstrated increased quit attempts and increased biologically verified smoking abstinence compared with 2 control groups [32].

    Next, WT described an international project called Detection and Integrated Care for Depression and Alcohol Use in Primary Care, which is funded by the Research Partnerships for Scaling Up Mental Health Interventions in Low- and Middle-Income Countries (Scale-Up Hubs) program of the National Institute of Mental Health. The project seeks to build sustainable research capacity and science-based programs in Latin America while simultaneously creating new knowledge to inform science-based approaches to scaling up mental health implementation research [33,34]. In Latin America, the burden of mental health problems is high, and services for mental health care are low. Expanding access to mental health care in a way that can be quickly scaled and have a substantial impact on the population is a significant global challenge. This project involves training the Latin American primary care workforce to use digital technology to enhance screening and diagnosis and to deliver treatment for depression and substance use via a science-based mobile digital therapeutic tool.

    The next set of presentations illustrated how social media can be leveraged for conducting digital epidemiological studies, developing and testing of digital therapeutics, and in-person recruitment for traditional clinical trials. JB reported on a particularly cost-effective means of using social media to recruit adult and adolescent populations with unique characteristics. He highlighted existing literature demonstrating that such methods can be used to recruit clinically relevant populations (eg, persons with depression or HIV, electronic cigarette users, and those with alcohol use problems). He then described a series of studies conducted at CTBH using Facebook advertising mechanisms to recruit large and diverse samples of individuals who use cannabis and discussed the rich clinical epidemiological datasets that were obtained [35,36]. Such studies have provided valuable insights into policy impact, cannabis use phenomenology, and cannabis use benefits and consequences. He further highlighted how social media can also be leveraged to locate individuals interested in receiving treatment and for developing and delivering interventions remotely. AK described her innovative project that used social media, that is, private Facebook groups, to recruit adolescents at risk for anxiety and to remotely elicit their feedback on the design of a digital anxiety intervention [37]. She discussed how the observed active engagement and participation of teens in the Web-based focus group suggest that social media platforms may be an effective tool to engage and elicit feedback from youth in the early stages of the intervention design process.

    CS concluded the series of demonstration project presentations by describing her Web-delivered intervention (WebRx) for teens with type 1 diabetes (T1D) and their parents. Across several small iterative pilot studies, she developed a novel multicomponent intervention that targeted adherence to self-management behaviors necessary to manage T1D [38]. Adherence-focused strategies include incentives for youth for objectively defined and tracked adherence behaviors, incentives for parents for daily monitoring of youth adherence, and Web-based health coaching to promote effective use of diabetes device data, and digitally delivered working memory training for the teens. CS described results from an RCT that demonstrated the superiority of WebRx, which engendered higher self-monitoring of blood glucose, better visual-spatial working memory and inhibition, lower hemoglobin A1c levels than those receiving usual care, more frequent parent review of the adolescent’s glucometer, and reduced family conflict [38]. The presentation concluded by focusing on how this highly disseminable intervention focused on promoting the effective use of technology and software to monitor and improve medical outcomes.

    Conclusions and Future Directions

    This workshop brought together a collaborative group of junior and senior scientists from a diverse array of disciplines, including clinical and experimental psychology, data science and analytics, psychiatry, computer science, and population health, all interested in the same goal, that is, advancing digital therapeutics to improve health behavior. The presentations and discussions illustrated how scientists across these disciplines can learn from each other and jointly expedite the potential of technological innovation for improving public health and health care and how the science of behavior change is essential for maximizing the impact of this endeavor. Workshop participants endorsed finding additional meaningful ways to bring these seemingly disparate groups of scientists together more frequently. Many acknowledged that this was necessary to accelerate this area of science because of the multitude of phenomena involved in optimizing health and health research, for example, complexity of behavior, behavior change, assessment, measurement, big data analytics, intervention design, experimental design, technology capabilities and limitations, motivation for change, adherence to treatment regimens, and for optimal communication, dissemination, implementation, and sustainability of effective discoveries.

    Attendees expressed an eagerness to participate in future workshops or alternative interdisciplinary activities that would better connect them to their colleagues and promote more collaborative project proposals. This workshop, the second of a planned series of 3 CTBH workshops corresponding to the Center’s 3 scientific cores [39], confirmed and clearly illustrated the value of these types of structured activities for fostering greater collaboration among our interdisciplinary faculty and scientists. Complimentary ideas for the future included more in-depth workshops focused on deep dives into specific topics, organization of cross-disciplinary student and trainee journal clubs, use of alternative formats for workshops with built in time for small interdisciplinary group discussion, and the continued development of CTBH pilot funding opportunities that require specified cross-discipline collaborations.

    In summary, our workshop on the development and evaluation of digital therapeutics for health behavior change showcased the myriad research targets that will be instrumental in promoting and accelerating progress in the field of digital health and health behavior change. The CTBH at the Geisel School of Medicine at Dartmouth College provides a model of multidisciplinary leadership and collaboration that can facilitate innovative, science-based efforts to address the health behavior challenges afflicting our communities by engaging and guiding teams of scientists to conduct research that will provide the knowledge and tools to inform more effective public health programming.


    The preparation of this paper was partially supported by a P30 Center of Excellence grant, P30DA029926 (LM, principal investigator), from the NIDA at the National Institutes of Health. The authors gratefully acknowledge the contributions of the other speakers at the workshop and thank Sonia Oren and Michelle Gleason for their efforts and assistance in organizing the workshop and preparing this summary.

    Conflicts of Interest

    LM is affiliated with Square2 Systems, Inc; Healthsim, LLC; and Pear Therapeutics. These relationships are extensively managed by LM and her academic institution. Dr. Brunette had a research grant from Alkermes, Inc.


    1. Budney AJ, Borodovsky JT, Marsch LA, Lord SE. Technological innovations in addiction treatment. In: Danovitch I, Mooney LJ, editors. The Assessment and Treatment of Addiction: Best Practices and New Frontiers. St Louis, Missouri: Elsevier; 2019:75-90.
    2. Marsch LA, Borodovsky JT. Digital health interventions for substance use disorders: the state of the science. In: Miller SC, Feillin DA, Rosenthal RN, Saitz R, Ries RK, editors. The American Society of Addiction Medicine: Principles of Addiction Medicine. Sixth Edition. Philadelphia: Lippincott Williams & Wilkins; 2018:1049-1056.
    3. Naslund JA, Bartels SM, Marsch LA. Digital technology, including telemedicine, in the management of mental illness. In: Atkinson K, Mabey D, editors. Revolutionizing Tropical Medicine: Point-of-Care Tests, New Imaging Technologies and Digital Health. Hoboken, NJ: Wiley-Blackwell; 2019:505-530.
    4. Naslund JA, Aschbrenner KA, Araya R, Marsch LA, Unützer J, Patel V, et al. Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature. Lancet Psychiatry 2017 Jun;4(6):486-500 [FREE Full text] [CrossRef] [Medline]
    5. US Food and Drug Administration. 2018 Mar 23. Digital Health Criteria   URL: [accessed 2019-12-06]
    6. US Food and Drug Administration. 2019 Jul 18. Digital Health Software Precertification (Pre-Cert) Program   URL: https:/​/www.​​medical-devices/​digital-health/​digital-health-software-precertification-pre-cert-program [accessed 2019-12-06]
    7. US Food and Drug Administration. 2017 Sep 14. FDA Permits Marketing of Mobile Medical Application for Substance Use   URL: https:/​/www.​​news-events/​press-announcements/​fda-permits-marketing-mobile-medical-application-substance-use-disorder [accessed 2019-12-06]
    8. Dallery J, Jarvis B, Marsch L, Xie H. Mechanisms of change associated with technology-based interventions for substance use. Drug Alcohol Depend 2015 May 1;150:14-23 [FREE Full text] [CrossRef] [Medline]
    9. Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychol 2015 Dec;34S:1209-1219 [FREE Full text] [CrossRef] [Medline]
    10. Dallery J, Kurti A, Erb P. A new frontier: integrating behavioral and digital technology to promote health behavior. Behav Anal 2015 May;38(1):19-49 [FREE Full text] [CrossRef] [Medline]
    11. Marsch LA. Envisioning the future: transformation of health care systems via technology. In: Marsch LA, Lord SE, Dallery J, editors. Behavioral Healthcare and Technology: Using Science-based Innovations to Transform Practice. New York, NY: Oxford University Press; 2014.
    12. Eisenberg IW, Bissett PG, Canning JR, Dallery J, Enkavi AZ, Whitfield-Gabrieli S, et al. Applying novel technologies and methods to inform the ontology of self-regulation. Behav Res Ther 2018 Feb;101:46-57 [FREE Full text] [CrossRef] [Medline]
    13. Marsch LA. Opportunities and needs in digital phenotyping. Neuropsychopharmacology 2018 Jul;43(8):1637-1638 [FREE Full text] [CrossRef] [Medline]
    14. Center for Technology and Behavioral Health.   URL: [accessed 2019-12-06]
    15. Higgins ST, Silverman K, Sigmon SC, Naito NA. Incentives and health: an introduction. Prev Med 2012 Nov(55 Suppl):S2-S6 [FREE Full text] [CrossRef] [Medline]
    16. Dallery J, Meredith SE, Budney AJ. Contingency management in substance abuse treatment. In: Walters ST, Rotgers F, editors. Treating Substance Abuse: Theory and Technique. Third Edition. New York: The Guilford Press; 2012:81-112.
    17. Bickel WK, Moody L, Higgins ST. Some current dimensions of the behavioral economics of health-related behavior change. Prev Med 2016 Nov;92:16-23 [FREE Full text] [CrossRef] [Medline]
    18. Dallery J, Raiff BR. Optimizing behavioral health interventions with single-case designs: from development to dissemination. Transl Behav Med 2014 Sep;4(3):290-303 [FREE Full text] [CrossRef] [Medline]
    19. Dallery J, Cassidy RN, Raiff BR. Single-case experimental designs to evaluate novel technology-based health interventions. J Med Internet Res 2013 Feb 8;15(2):e22 [FREE Full text] [CrossRef] [Medline]
    20. Jarvis BP, Dallery J. Internet-based self-tailored deposit contracts to promote smoking reduction and abstinence. J Appl Behav Anal 2017 Apr;50(2):189-205 [FREE Full text] [CrossRef] [Medline]
    21. Dallery J, Riley J, Nahum-Shani I. Research designs to develop and evaluate technology-based health behavior interventions. In: Marsch LA, Lord SE, Dallery J, editors. Behavioral Healthcare and Technology: Using Science-Based Innovations to Transform Practice. New York, NY: Oxford University Press; 2014:168-186.
    22. Collins LM. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). New York, NY: Springer; 2018.
    23. Collins LM, Nahum-Shani I, Almirall D. Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART). Clin Trials 2014 Aug;11(4):426-434 [FREE Full text] [CrossRef] [Medline]
    24. Stanger C, Scherer EA, Vo HT, Babbin SF, Knapp AA, McKay JR, et al. Working memory training and high magnitude incentives for youth cannabis use: a SMART pilot trial. Psychol Addict Behav 2019 Jun 27. [CrossRef] [Medline]
    25. Boruvka A, Almirall D, Witkiewitz K, Murphy SA. Assessing time-varying causal effect moderation in mobile health. J Am Stat Assoc 2018;113(523):1112-1121 [FREE Full text] [CrossRef] [Medline]
    26. Wang R, Wang W, daSilva A, Huckins JF, Kelley WM, Heatherton TF, et al. Tracking depression dynamics in college students using mobile phone and wearable sensing. Proc ACM Interact Mob Wearable Ubiquitous Technol 2018;2(1):1-26. [CrossRef]
    27. Mishra V, Pope G, Lord S, Lewis S, Lowens B, Caine K, et al. Continuous detection of physiological stress with commodity hardware. ACM Transactions on Computing for Healthcare 2020 (forthcoming).
    28. Kiluk BD, Nich C, Buck MB, Devore KA, Frankforter TL, LaPaglia DM, et al. Randomized clinical trial of computerized and clinician-delivered CBT in comparison with standard outpatient treatment for substance use disorders: primary within-treatment and follow-up outcomes. Am J Psychiatry 2018 Sep 1;175(9):853-863 [FREE Full text] [CrossRef] [Medline]
    29. The Small Business Innovation Research (SBIR). A Game Based Intervention for Opioid Use Disorder   URL: [accessed 2019-12-06]
    30. Christensen DR, Landes RD, Jackson L, Marsch LA, Mancino MJ, Chopra MP, et al. Adding an internet-delivered treatment to an efficacious treatment package for opioid dependence. J Consult Clin Psychol 2014 Dec;82(6):964-972 [FREE Full text] [CrossRef] [Medline]
    31. Brunette MF, Feiron JC, Aschbrenner K, Colctti D, Devitt T, Greene MA, et al. Characteristics and predictors of intention to use cessation treatment among smokers with schizophrenia: Young adults compared to older adults. J Subst Abus Alcohol 2017;5(1):pii: 1055 [FREE Full text] [Medline]
    32. Brunette MF, Ferron JC, Robinson D, Coletti D, Geiger P, Devitt T, et al. Brief web-based interventions for young adult smokers with severe mental illnesses: A randomized, controlled pilot study. Nicotine Tob Res 2018 Sep 4;20(10):1206-1214 [FREE Full text] [CrossRef] [Medline]
    33. Project DIADA (Detection and Integrated Care for Depression and Alcohol Use in Primary Care). Project DIADA   URL: [accessed 2019-12-06]
    34. NIH: Project Reporter. Scaling Up Science-Based Mental Health Interventions in Latin America   URL: https:/​/projectreporter.​​project_info_description.​cfm?aid=9705920&icde=46275019&ddparam=&ddvalue=&ddsub=&cr=15&csb=default&cs=ASC&pball= [accessed 2019-12-06]
    35. Borodovsky JT, Marsch LA, Budney AJ. Studying cannabis use behaviors with Facebook and web surveys: methods and insights. JMIR Public Health Surveill 2018 May 2;4(2):e48 [FREE Full text] [CrossRef] [Medline]
    36. Knapp AA, Lee DC, Borodovsky JT, Auty SG, Gabrielli J, Budney AJ. Emerging trends in cannabis administration among adolescent cannabis users. J Adolesc Health 2019 Apr;64(4):487-493. [CrossRef] [Medline]
    37. Knapp AA, Mohr D, Budney AJ, Lord SE. Utilization of social media to recruit and engage teens in online focus groups. Poster. 2019 May 28 Presented at: Annual Meeting of the Society of Prevention Research; 2019; San Francisco, CA.
    38. Stanger C, Lansing AH, Scherer E, Budney AJ, Christiano AS, Casella SJ. A web-delivered multicomponent intervention for adolescents with poorly controlled Type 1 diabetes: a pilot randomized controlled trial. Ann Behav Med 2018 Nov 12;52(12):1010-1022 [FREE Full text] [CrossRef] [Medline]
    39. Kotz D, Lord SE, O'Malley AJ, Stark L, Marsch LA. Workshop on emerging technology and data analytics for behavioral health. JMIR Res Protoc 2018 Jun 20;7(6):e158 [FREE Full text] [CrossRef] [Medline]


    CM: contingency management
    CTBH: Center for Technology and Behavioral Health
    FDA: Food and Drug Administration
    NIDA: National Institute on Drug Abuse
    RCT: randomized controlled trial
    SCEDs: single-case experimental designs
    SMART: Sequential Multiple Assignment Randomized Trial
    T1D: type 1 diabetes
    TDE: treatment development and evaluation
    WebRx: Web-delivered intervention

    Edited by G Eysenbach; submitted 23.10.19; peer-reviewed by S Acquilano, H Xie; comments to author 30.11.19; revised version received 11.12.19; accepted 15.12.19; published 26.02.20

    ©Alan J Budney, Lisa A Marsch, Will M Aklin, Jacob T Borodovsky, Mary F Brunette, Andrew T Campbell, Jesse Dallery, David Kotz, Ashley A Knapp, Sarah E Lord, Edward V Nunes, Emily A Scherer, Catherine Stanger, William C Torrey. Originally published in JMIR Mental Health (, 26.02.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.