This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
A national priority at the US Department of Veterans Affairs (VA) is to increase the availability and accessibility of evidence-based psychotherapies (EBPs) across all VA medical facilities. Yet many veterans, particularly those who use remote outpatient VA clinics, still do not receive much needed evidence-based treatment. Strategies are needed for supporting mental health providers at rural VA community-based outpatient clinics (CBOCs) as they translate their clinical training to routine practice. The Coordinated Anxiety Learning Management (CALM) program is a computer-delivered program that supports the delivery of cognitive behavioral therapy (CBT) by providers in outpatient settings to patients with depression and anxiety, including posttraumatic stress disorder.
The objectives of our study were to (1) adapt an existing computer-based program to rural VA CBOCs through feedback from key stakeholder focus groups; (2) develop a prototype of the adapted program; and (3) determine the adapted program’s acceptability and feasibility. Mental health stakeholders included VA leaders (n=4) in the implementation of EBPs, VA experts (n=4) in CBT, VA CBOC mental health providers (n=8), and veterans (n=8) diagnosed with a mental health condition treated using the CALM program and receiving treatment in a VA CBOC.
An iterative approach comprising 3 waves of focus group discussions was used to develop a modified prototype of CALM. Following each wave of focus group discussions, template analysis was used to rapidly communicate stakeholder recommendations and feedback to the design team. The original program was first adapted through a process of data collection, design modification, and product development. Next, a prototype was developed. Finally, the redesigned program was tested for acceptability and feasibility through a live demonstration.
Key stakeholders suggested modifications to the original CALM program that altered its modules’ appearance by incorporating veteran-centric content. These modifications likely have no impact on the integrity of the original CALM program, but have altered its content to reflect better the demographic characteristics and experiences of rural veterans. Feedback from stakeholder groups indicates that changes will help VA patients identify with the program content, potentially enhancing their treatment engagement.
The development model was effective for economically gathering actionable recommendations from stakeholders to adapt a computer-based program, and it can result in the development of an acceptable and feasible computer-delivered intervention. Results have implications for developing computer-based programs targeting behavior change more broadly and enhancing engagement in EBP.
Ample evidence indicates the effectiveness of evidence-based psychotherapies (EBPs), particularly cognitive behavioral therapy (CBT) [
Despite this effort, broadly implementing CBT and other EBPs in VA treatment settings has been a challenge [
An additional challenge at VA CBOCs is ensuring that treatments are delivered with fidelity (ie, faithfulness to the treatment model). Without adequate fidelity, patients are unlikely to experience optimal outcomes from EBPs, even where they are accessible. Although the VA has made great strides in providing EBP-specific training to providers of mental health care, training alone is not sufficient to ensure fidelity [
Over the last decade, the National Institute of Mental Health-funded Coordinated Anxiety Learning Management (CALM) study [
Modified ADDIE (Analysis, Design, Development, Implementation, and Evaluation) model and study aims. CALM: Coordinated Anxiety Learning Management, CBT: cognitive behavioral therapy.
In this study, we sought to adapt the original CALM program for use in VA outpatient settings, thus, supporting VA CBOC mental health providers in delivering CBT to veterans with anxiety, PTSD, and depression. To achieve this objective, we modified a method common to the field of instructional design and technology (IDT) to guide the redesign of the CALM computer program for use in VA CBOCs. IDT models are prescriptive models that describe a set of activities involved in the planning, implementation, and evaluation of instructional programs [
As part of this process, 3 waves of focus group discussions were conducted with 4 groups of key stakeholders to redesign and test the CALM program iteratively for use in VA CBOCs. Stakeholder groups included (1) veterans receiving recent mental health care in a VA CBOC and diagnosed with anxiety disorders and depression, (2) CBOC mental health providers, (3) expert CBT VA clinicians, and (4) VA Central Office leaders with expertise in the implementation of EBPs within VA. In this paper, we describe the iterative process used to redesign and test the modified CALM program, as well as modifications used to adapt the program for use at VA CBOCs.
We recruited participants from 4 key stakeholder groups: veterans (n=11) with a recent (≥1 visit in the prior 6 months) mental health care visit at a VA CBOC and with a diagnosis of one or more of the following: PD, GAD, SAD, PTSD or depression; CBOC mental health providers (n=11) in the southern region of the United States, including nurses, psychiatrists, social workers, and psychologists; VA expert CBT clinicians (n=6); and VA Central Office leaders (n=5) with expertise in implementing EBPs within VA. We conducted 3 waves of focus group discussions with each stakeholder group.
Veteran participants were recruited from one CBOC in Arkansas. Mental health providers were recruited using a general call distributed via email to providers at CBOCs in Arkansas, Louisiana, Texas, and Mississippi. Multiple CBOCs were needed because most CBOCs have only one mental health provider on site. CBT experts were recruited nationally from within the VA, and leaders in implementation of EBPs within VA were recruited from the VA Central Office in Washington, DC CBT experts and leaders in implementation were individually recruited through convenience sampling based on their level of expertise and availability [
The purpose of recruiting veteran stakeholders was to help ensure that the modified CALM content was acceptable to the target patient population and reflected veterans’ illness experiences. We included CBOC mental health providers to help ensure that the modified content was acceptable for this group of providers (who serve a largely rural veteran population), images and case studies were appropriate for the target patient population, and the navigation and flow of CALM treatment material met the needs of CBOC providers. Inclusion of VA expert CBT clinicians helped ensure that the empirical support underlying the content of the CALM program was not compromised during the modification process. Finally, inclusion of VA Central Office leaders with expertise in EBP implementation within VA helped the study team ensure that CALM was consistent with prior CBT training efforts within VA.
Each group of key stakeholders participated in 3 waves of separate focus group discussions. Focus group discussions are a standard method used to garner diverse feedback efficiently on novel and existing products for new product development. CBOC mental health providers, CBT experts, and VA Central Office leaders with expertise in EBP implementation within VA participated in focus group discussions via teleconference with the assistance of Lync, a Web-based meeting portal that allows participants to view the same material simultaneously. In addition to providing feedback via telephone, participants could leave written feedback on the website chat room and email content for the focus group facilitators. Veteran focus groups met in person at a VA CBOC in Hot Springs, Arkansas.
All focus group discussions were co-moderated by the principal investigator (MAC) and a co-investigator (THA). Both researchers have extensive experience in qualitative interviewing and facilitating focus group discussions. An interview guide developed for the study was used to ensure that the discussion remained relatively consistent across stakeholder groups and that groups addressed all relevant topics. The same stakeholders were invited to participate in all 3 waves of focus group discussions to maximize the relevance of feedback (ie, participants knew whether recommendations had been correctly incorporated into the program). Focus group discussions were audiorecorded with permission from participants. The study was approved by the Central Arkansas Veterans Healthcare System Human Research Protections Program.
Rapid analytic techniques informed by Sobo et al [
To analyze focus group discussions, the lead analyst (THA) listened to the audio recording of the first focus group discussion and systematically populated template categories with data. These data consisted of paraphrased content reflecting stakeholder recommendations, observations, reactions, concerns, and questions. The goal was to capture the full range of responses to questions and comments from focus group discussions. After the lead analyst completed summarizing all content from the focus group discussions in the template, she met with the second analyst (MAC) to discuss analytic findings. Discrepancies in how template content was summarized or categorized were resolved through discussion. This analytic process was repeated for each focus group discussion. For focus group discussions held via teleconference, analysts also incorporated written feedback from the website and emails to facilitators into the template. After analyses were finalized for all 4 focus group discussions, findings were compiled and summarized. Analysts each reviewed the summary to ensure the validity of findings. This analytic process was followed for all 3 waves of focus group discussions.
A panel of experts reviewed the summary template before it was submitted to the design and development team, including study team members with expertise in CBT, the relevant mental health diagnosis, and software development. The purpose of this panel was to prioritize and determine the feasibility of each recommendation (eg, cost and time needed to complete the modification). The summary template was subsequently reduced to actionable recommendations for modification.
The CALM computerized program was created to guide and train mental health providers in delivering a course of CBT [
During the first wave of focus group discussions, an overview of CALM was presented to each stakeholder group. This also included screenshots from the program, as well as a detailed description of the functioning of CALM for each disorder. Following the overview, stakeholders were given time to ask questions about CALM; then, they were asked to provide initial reactions or recommendations for modifying and adapting the program for use within VA CBOCs.
Recommended modifications to the original program were largely “look and feel” changes pertaining to the images and illustrations in the modules. Stakeholders generally reported that the images and illustrations contained in the original program should be more representative of veterans. They recommended reducing the number of images and illustrations depicting college-aged women and men in white-collar occupations (eg, wearing suits) as well as incorporating imagery better reflecting the gender, age, and economic and ethnic or racial diversity of veteran patients. Veteran stakeholders did not like the background color of the CALM program and overall website template; they wanted a new logo developed specifically for the modified VA program.
Stakeholders generally also recommended modifying the images in the original program to make it more “culturally congruent” because the original program was not designed to acknowledge veterans’ military service. One expert CBT clinician noted that this can be “very invalidating” for some patients. Stakeholders suggested replacing some existing images with images of people in uniform; however, a veteran stakeholder cautioned that while incorporating images of people in uniform, “We don’t need to see a Vet on every single picture we look at.” Stakeholders also suggested using images of individuals with physical limitations in the Behavioral Activation module to encourage veterans with prosthetics or other physical limitations to engage in physical activities.
Furthermore, stakeholders recommended revisions designed to enhance the degree to which veterans could identify with the program content. To achieve this, they suggested incorporating videos into the modules that speak to the unique mental health concerns of veterans. Stakeholders also suggested using existing VA resources, such as the National Center for PTSD’s “About Face” and the VA’s “Make the Connection” websites [
Finally, stakeholders were generally concerned that some slides seemed too “content heavy” to be engaging. Several expert CBT clinicians and CBOC mental health providers thought that having to read a large amount of material during an hour-long therapy session might be difficult, and expert CBT clinicians were concerned that younger veterans with traumatic brain injuries would be overwhelmed by busy slides. This concern was validated by feedback from veteran stakeholders, who not only recommended reducing the number of words, but removing excessive images and illustrations. One veteran noted: “Pictures: unless it means something, what’s the point?”
A summary of stakeholder recommendations was presented to the expert panel for review, and actionable modifications were identified and prioritized. The study team then collaborated with the software development group to identify images and illustrations better reflecting the gender, age, and economic and ethnic or racial diversity of veterans. They also identified appropriate images of people in military uniform. The images below are examples of CALM module content before (
The team identified already available Web-based resources with videos of veterans describing their mental health concerns, their experiences with a condition and/or seeking mental health care, and ways that treatment helped them (
Finally, the team followed stakeholder suggestions in identifying links to Web-based resources for veterans and reduced the amount of text on slides identified by stakeholders as too content heavy. These modifications to the original program were submitted to the software development team, which then developed mock-ups, or wireframes, of the initial modifications to show stakeholders in Wave 2.
The wireframes were presented during a second wave of focus group discussions with veterans (n=6), CBOC mental health providers (n=10), expert CBT clinicians (n=5), and VA leaders with expertise in the implementation of EBPs within VA (n=5). Wave 2 participants were reminded of the initial version of CALM using graphics and were provided a side-by-side comparison of all modifications. After reviewing the wireframes, stakeholders provided feedback regarding modifications from the first round of focus groups, and a few suggested additional changes to the CALM program.
Stakeholders generally agreed that the modified program appeared more “inclusive of veterans that will be using CALM” and “user-friendly” for the veteran population. Although agreeing that the initial modifications were an improvement, one veteran stakeholder requested more images of older males to be more inclusive of Vietnam-era veterans. Stakeholders, particularly veterans, also responded positively to reductions in the amount of text on many slides. One CBOC mental health provider thought that reducing the amount of text would “leave more room for interaction” between providers and veteran patients. Stakeholders also responded positively to the inclusion of videos and links to resources. One younger participant in the veteran focus group who had completed VA treatment for PTSD suggested replacing a video in which veterans described only their treatment experiences with the one in which they explain that treatment is difficult at first, but helps. Finally, one CBOC mental health provider noted that many of his veteran patients were uncomfortable with writing and suggested the program allow veterans the option of verbally describing and recording trauma memories for use in the Exposure module, in addition to writing about them.
The expert panel again reviewed a summary of recommendations and prioritized actionable modifications from Wave 2 focus group discussions. Following this, the study team identified images and videos per stakeholder recommendations. The principal investigator (MAC) collaborated with one CALM developer (MAC) to incorporate an option allowing veterans who are uncomfortable with writing to verbally describe and record their traumatic experiences (
Original Coordinated Anxiety Learning Management (CALM) module content. (Screenshot taken by KMM; Source: University of California, Los Angeles).
Modified Coordinated Anxiety Learning Management (CALM) module content. (Screenshot taken by KMM; Source: US Department of Veterans Affairs).
Video featuring veterans describing their experiences with posttraumatic stress disorder (PTSD); used with permission from the National Center for PTSD “About Face” website. (Screenshot taken by KMM; Source: US Department of Veterans Affairs).
Option allowing veterans to describe and record traumatic experiences. (Screenshot taken by KMM; Source: US Department of Veterans Affairs).
During a final wave of focus group discussions, the prototype was pilot-tested by MGC during a live teleconferenced demonstration to 3 stakeholder groups: CBOC mental health providers (n=6), expert CBT clinicians (n=4), and VA leaders in clinic operations and implementation (n=3). Minor problems that arose during the pilot test were recorded in written notes by the study team and rapidly communicated to the design team following the demonstration. Veteran stakeholders (n=4) reviewed modified CALM modules presented in person by the principal investigator (MAC) and looked at an overview of all modifications by the lead qualitative analyst (THA). No remaining concerns about the program were elicited through this final wave of focus groups.
In this study, we used an iterative approach to adapt the original CALM program for use within the VA and, particularly, at rural, CBOCs. Overall, key stakeholders suggested modifications to the original CALM program that were largely “look and feel” adaptations that altered the appearance of the modules by incorporating veteran-centric content. This included relatively simple changes, such as replacing images and developing a new template with the VA logo, as well as more significant adaptations, such as embedding videos of veterans describing their treatment and illness experiences and modifying the case studies to better reflect experiences common to rural veterans. These modifications likely have no impact on the integrity of the original CALM program, while altering the content to better reflect the demographic characteristics and experiences of rural veterans.
An additional adaptation was to customize the treatment content to allow veterans the option of orally recounting (as opposed to only writing) their trauma experiences. This modification, along with the overall flexibility of the CALM program to meet the treatment needs of veterans with a wide variety of mental health conditions, is consistent with a long-standing cultural shift at the VA toward the provision of patient-centered health care consisting of a menu of treatment options that can be tailored to each individual veteran’s needs and goals for health and well-being [
Feedback from stakeholder groups—and, in particular, rural veterans—indicates that these modifications will help VA patients identify with the program content. This could potentially improve patient-provider communication during therapy sessions and increase veteran engagement in EBP for anxiety, depression, and PTSD. Enhancing engagement is important because attrition from EBPs is high among veterans, particularly those with PTSD [
CALM has the added benefit of being both patient and provider facing; thus, providers—not patients—directly interact with the program. The use of CALM during treatment sessions does not require patients to possess either technological proficiency or access to the internet at home. Future research could explore whether the program helps older patients gain a sense of familiarity with the use of technology in health care, thus, reducing the potential for a deepening digital divide among older adults.
Feedback, comments, and suggestions obtained from the 4 stakeholder groups, in tandem with results from similar studies [
The study design draws upon numerous strengths. An iterative design that included inviting the same stakeholders to participate in 3 waves of focus group discussions helped ensure the relevance and consistency of recommended modifications. The strength of this approach was reinforced by hosting the veteran focus group last, which allowed veterans to respond to the recommendations from the other stakeholder groups. Additionally, collecting feedback from stakeholders who are familiar with rural veterans, EBPs, and clinical practice at VA CBOCs may have increased the acceptability and feasibility of the modified program for use in this population. This approach was aligned with a person-based approach for developing and tailoring technology-based interventions [
One limitation of our study was that focus group participation declined throughout the study, particularly among veteran stakeholders. Because we were unable to obtain feedback from every stakeholder who participated in the first wave of focus group discussions, it is possible that we may have omitted or failed to implement suggestions as originally envisioned. Additionally, veteran and mental health provider stakeholders were recruited from one geographical region in the southern United States (Arkansas, Louisiana, and Texas). Thus, stakeholder feedback may not be generalizable to other locations. Although a necessary first step in this direction, this study also does not provide data regarding fidelity to the CBT model or treatment outcomes. A next step in this line of research is to determine whether the modified version of CALM improves VA CBOC mental health providers’ fidelity to the CBT model and improves veteran outcomes. The evidence already indicates that the original CALM program achieves these objectives [
We modified the CALM program for use in rural VA clinics based on feedback from 2 waves of focus group discussions with 4 key stakeholder groups. The results of pilot testing the modified program during a third and final wave suggest that the adaptations increased the relevance and acceptability of CALM content for rural veterans and other key VA stakeholders, such as mental health providers at VA CBOCs. It will be important to assess whether using the program as an interface between providers and patients during sessions enhances veterans’ engagement in EBPs. The iterative approach used in this study holds promise for economically gathering actionable recommendations from stakeholders to enhance the acceptability and feasibility of computer-based programs in health care settings.
Coordinated Anxiety Learning Management
community-based outpatient clinics
cognitive behavioral therapy
evidence-based psychotherapies
generalized anxiety disorder
instructional design and technology
panic disorder
posttraumatic stress disorder
social anxiety disorder
US Department of Veterans Affairs
This work was supported by funding from the US Department of Veterans Affairs Health Services Research CREATE initiative, Development Service to MAC, and the use of facilities and resources of the VA HSR&D Center for Innovations in Quality, Effectiveness and Safety (Cin13-413). The views expressed are those of the authors and do not represent the views of the US Department of Veterans Affairs, the US government, or Baylor College of Medicine.
THA and MAC co-wrote the introduction and methods, and THA wrote the results and discussion. All authors reviewed, edited, and approved the final manuscript. THA, MAC, KMM, and MBM collected all data. All authors were involved in developing the modified program. All authors were involved in modification of CALM for use in VA CBOCs.
None declared.