Published on 31.01.19 in Vol 6, No 1 (2019): January
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11973, first published Aug 19, 2018.
Recommendations for Methodology of Virtual Reality Clinical Trials in Health Care by an International Working Group: Iterative Study
Background: Therapeutic virtual reality (VR) has emerged as an efficacious treatment modality for a wide range of health conditions. However, despite encouraging outcomes from early stage research, a consensus for the best way to develop and evaluate VR treatments within a scientific framework is needed.
Objective: We aimed to develop a methodological framework with input from an international working group in order to guide the design, implementation, analysis, interpretation, and communication of trials that develop and test VR treatments.
Methods: A group of 21 international experts was recruited based on their contributions to the VR literature. The resulting Virtual Reality Clinical Outcomes Research Experts held iterative meetings to seek consensus on best practices for the development and testing of VR treatments.
Results: The interactions were transcribed, and key themes were identified to develop a scientific framework in order to support best practices in methodology of clinical VR trials. Using the Food and Drug Administration Phase I-III pharmacotherapy model as guidance, a framework emerged to support three phases of VR clinical study designs—VR1, VR2, and VR3. VR1 studies focus on content development by working with patients and providers through the principles of human-centered design. VR2 trials conduct early testing with a focus on feasibility, acceptability, tolerability, and initial clinical efficacy. VR3 trials are randomized, controlled studies that evaluate efficacy against a control condition. Best practice recommendations for each trial were provided.
Conclusions: Patients, providers, payers, and regulators should consider this best practice framework when assessing the validity of VR treatments.
JMIR Ment Health 2019;6(1):e11973
Therapeutic virtual reality (VR) is an innovative treatment modality to manage a broad range of health conditions and is gaining considerable attention [- ]. Users of VR wear a head-mounted display (HMD) with a close-proximity screen that creates a sense of being transported into life-like, three-dimensional worlds. VR has been used to assess and treat a wide variety of medical, surgical, psychiatric, and neurocognitive conditions including pain [ , , , , , ], addiction [ - ], anxiety disorders [ , , , - , - ], schizophrenia [ , , , - ], eating disorders [ , , - ], stroke rehabilitation [ , , - , - ], vestibular disorders [ ], and movement disorders [ ]. One of the first published uses of HMD-based therapy was the treatment of acrophobia in 1995 [ ]. There have also been functional magnetic resonance imaging studies demonstrating the effect of VR on the brain during receipt of a painful stimuli [ , ]. VR is thought to work through a combination of distraction, extinction learning, cognitive-behavioral principles, mindful meditation, stress reduction, gate-control theory, and the spotlight theory of attention [ , ]. Importantly, VR has become increasingly portable, immersive, and vivid, which has enabled the technology to be used in a broad range of inpatient and outpatient applications.
As the use of therapeutic VR expands, it is essential that guidelines are established to ensure scientific rigor in the development and evaluation of VR applications, similar to established standards for pharmacotherapies [, ]. VR developers would benefit from systematic guidance on best practices for designing and conducting VR clinical trials. To fulfil this unmet need, we garnered input from an international working group, called the Virtual Reality Clinical Outcomes Research Experts (VR-CORE) committee. This paper presents the resulting best practice framework informed by expert input, along with specific recommendations on ways to conduct high-quality VR treatment trials. Although the focus of this paper is VR, the framework also applies to other emerging “XR” technologies, including augmented reality and mixed reality, as the methodologic considerations for clinical trials are largely similar across XR platforms.
Identifying Virtual Reality Clinical Outcomes Research Experts
We performed a systematic review of randomized controlled trials (RCTs) using therapeutic VR to help identify eligible VR-CORE committee members through review of author lists. To cover the largest breadth of studies, the literature search focused on existing meta-analyses of therapeutic VR RCTs identified through search of PubMed, Google Scholar, and the Cochrane Database of Systematic Reviews using a combination of keywords: (“virtual reality” OR “VR”) AND (“review [pt]” OR “systematic review [pt]” OR “meta-anal*” OR “metaanaly*”). Based on our literature search, and supplemented by recommendations from established experts, we developed a multidisciplinary group for the VR-CORE, including experts in fields relevant to developing and testing VR treatments such as user-centered design principles, software design, epidemiology, statistics, and clinical trial methodology. The committee was formulated to balance expertise across clinical disciplines (medicine, pediatrics, surgery, psychology, psychiatry, neuroscience, anesthesia, nursing, and rehabilitation) and reflect multinational perspectives.
Collecting Input From the Virtual Reality Clinical Outcomes Research Experts
To obtain systematic feedback from the committee, a series of electronic meetings were held to collect and synthesize structured input. An iterative approach was modeled after similar processes were employed by our previous working groups in other fields of health care [, ]. Using an online meeting platform that allows users to view and react to each other’s comments [ ], committee members initially responded to open-ended “think aloud” prompts [ ] (eg, “When you think about the current state of the clinical VR research, what comes to your mind?”), followed by increasingly specific probes prepared by the moderators (eg, “What should be the role of human centered design principles in developing VR treatments?”). The full set of questions and responses is listed in . The active members of the VR-CORE at the time of this discussion are listed in the Acknowledgments section. Emergent themes and proposed methodologic best practices were culled from the online dialogue, and the resulting recommendations were distributed to the members for synthesis and iterative rephrasing.
Emergent Themes from Virtual Reality Clinical Outcomes Research Experts Meetings
provides excerpted transcripts of the VR-CORE responses to discussion topics. Key themes drawn from the online dialogue are summarized in the following sections.
Perceptions Regarding the Current State of Clinical Virtual Reality Research
Committee members described the current state of clinical VR research as the “Wild West” with a “lack of clear guidelines and standards.” The state of current VR research was described as “heterogeneous,” often focused “more on the tech rather than the theories behind it.” Committee members expressed concern that much of the current research is “merely descriptive” in nature, often insufficiently powered, focused on small case reports and retrospective analyses, and often does not employ experimental designs.
Perceptions About Ways to Improve Virtual Reality Literature
The committee believed it is vital to “include the patients’ voice early and often in the development of VR treatments” and that developers must “carefully, systematically, and meticulously seek the patients’ feedback” through participatory research and design thinking that involves multidisciplinary collaboration. The committee acknowledged the importance of including the voice of providers as well. The committee also called for better definitions and standardization of therapeutic VR study designs.
Most Important Considerations for Designing and Standardizing Clinical Virtual Reality Trials
The committee described various stages for developing and validating VR treatments, beginning with content development in partnership with end-users, progressing through initial clinical testing and safety evaluation, and ending with properly powered RCTs. The committee outlined a wide range of considerations for each stage (), including the importance of standardizing control groups, selecting clinically relevant outcome measures, reporting which equipment was used in the trial, accounting for dropouts and disqualified participants, and allowing for pragmatic features of each study design.
Clinical Trial Framework of the Virtual Reality Clinical Outcomes Research Experts
Although there are fundamental best practices in study design that apply to all biomedical intervention trials, the committee identified VR-specific attributes that are unique considerations for VR trials. Using the Food and Drug Administration Phase I-III pharmacotherapy model as guidance  and combining the results of literature synthesis with VR-CORE input, a framework emerged to support three phases of VR clinical study designs, named VR1, VR2, and VR3.
VR1 studies focus on content development by working with patient and provider end-users through principles of human-centered design. VR2 trials conduct early testing with a focus on feasibility, acceptability, tolerability, and initial clinical efficacy. VR3 trials are RCTs that compare clinically important outcomes between intervention groups and a control condition. Each study should undergo ethical review before initiation.summarizes each phase of the VR-CORE model. Best practice recommendations for each trial design are described below.
The committee strongly believes that therapeutic VR applications should be designed with direct input from patient and provider end-users. Lack of patient involvement, poor requirement definitions, and nonadaptation to user feedback are some of the common factors that explain failures of digital interventions . Incorporating patients into the design process enables developers to increase the relevance and effectiveness of VR treatments. The committee stresses that VR treatments should be created with acknowledgment of patients’ knowledge, attitudes, beliefs, preferences, and expectations of therapeutic VR. VR-CORE refers to a VR1 study as one that results in the development of VR treatment in partnership with patient and provider end-users and follows best practices for patient-centered design.
After their review of the literature on human-centered design both generally [, ] and in relation to digital [ ] and VR interventions [ ], the committee identified three key principles that are fundamental for developing “desirable, feasible and viable” VR treatments [ ]. These principles promote empathy, team collaboration, and continuous user feedback ( ). The committee believes that the use of these principles allows development teams to better identify users’ needs, incorporate user feedback, and institute rapid cycle improvements that generate more relevant products at lower cost [ ]. The key principles for VR1 studies are outlined in .
|Design principles and strategies||Best practices|
|Inspiration through empathizing|
|Journey mapping and personas|
|Ideation through team collaboration|
|Sharing stories and notes|
|Prototyping through continuous user feedback|
|Continuously testing prototype|
The Design Process of Virtual Reality Treatments Should Promote Empathy
The committee believes that the more attuned a development team is to the specific perspective and needs of patients, the more likely they are to design meaningful VR treatments. Promoting empathy toward the design process involves carefully listening to and elucidating patients’ social environment, needs, fears, desires, habits, hopes, aspirations, and expectations. The committee recommends initiating the design process with an inspiration step, or exercise focused on culling patients’ voice and understanding their needs, struggles, and experiences.describes best practices for sparking inspiration within the framework of empathy. Different patient profiles and scenarios should be included in this first step. Many techniques can be used to develop empathy and inspiration of the design team. These include qualitative assessments, observations, spending time with users, and conducting interviews and user experiments. In addition, a patient journey map can be used to illustrate the interpretation of a story from a patient’s perspective. The working group also recommends seeking input from relevant nonpatient end-users, including health care providers who may prescribe the VR treatment or interact with patient users.
The Design Process of Virtual Reality Treatments Should Promote Team Collaboration
The committee believes that team collaboration is fundamental for collectively designing a VR treatment and synthesizing data collected during the inspiration step. Brainstorming helps generate ideas from the initial corpus of data and findings.describes best practices for ideation within the framework of team collaboration. The process of ideation allows team members to think expansively and divergently. As a range of ideas is generated, some ideas will be extreme or ambitious, whereas others will be achievable. Depending on the time and the available budget, the team decides what ideas should be prototyped further.
The Design Process of Virtual Reality Treatments Should Promote Continuous User Feedback
An effective VR treatment should be developed through continuous user feedback and iterative prototyping, thereby enabling the team to rapidly test their ideas during real-time assessment from end-users.describes best practices for VR treatment prototyping within the framework of user feedback. Prototypes should be refined with continuous testing by patient end-users, and failures are viewed as a way to learn and improve the prototype to better meet users’ needs. Hence, the number of defects will tend to be lower and less costly in the future. To help facilitate the learning process for patients, it is recommended, when feasible, that the research team use a “mirroring” program [ ] to allow the research staff to see what the patient is viewing through the VR headset and help them learn the user interface.
Briefly, the committee believes that the VR1 treatment design process should start with end-users. VR-CORE recommends specifying who the real users are and what they say, see, feel, and do. Hence, implementation of a patient-design approach is an important way to place users at the center of the VR design process. For researchers who are developing an open-source VR intervention that they would like to share with the academic community for collaborative V1 development process, the use of a software-development platform such as GitHub.com  and citation of the latest version of the program within the methods section of VR1 research papers are recommended. The committee also recommends use of the Integrate, Design, Assess, and Share checklist developed by Mummah and colleagues [ ] as a supplemental, structured guide for conducting a VR1 study.
Once the research team has developed a VR treatment in partnership with end-users, the resulting product should undergo initial assessment in the target patient population within a representative clinical setting, herein termed a VR2 trial. Modeled after the work of Mosadeghi and colleagues , the purpose of VR2 trials is to conduct early testing with a focus on acceptability, feasibility, tolerability, and initial clinical efficacy prior to initiating a more definitive VR3 clinical trial. Although developers may opt to bypass a VR2 trial in lieu of a VR3 trial, there is a risk of subjecting an incompletely tested intervention to a larger and costlier RCT, and best practices in digital intervention development suggest an intermediary stage between initial VR design and definitive testing [ ]. The following sections describe the features of a VR2 trial.
In contrast to a VR1 study, which is focused on collaborative content development in a design environment, the VR2 trial evaluates what happens when the VR treatment is placed in the hands of target patients within the intended clinical setting. For example, a VR treatment focused on management of inpatient pain should be tested in an inpatient environment. A VR treatment targeting outpatient stroke rehabilitation should be evaluated in locations where patients receive rehabilitation, such as in a physical therapy center or, if intended, at home. In short, a comprehensive VR2 trial evaluates the VR treatment in the natural setting(s) where the product is intended to be used.summarizes the best practices for VR2.
In the context of a VR2 trial, acceptability refers to a patient’s willingness to use the VR treatment. Previous research on therapeutic VR reveals a drop off in the relation between patient eligibility to receive VR and patient willingness to try VR . The disconnect emphasizes that many patients are uninterested in using novel health technologies such as VR, particularly when hospitalized or under duress. Among those who are eligible for a VR trial, some choose not to participate for a wide variety of reasons. Patients may express varying degrees of skepticism, fear, vulnerability, and concern regarding psychological consequences or simply not want to be bothered by the equipment [ ]. In a VR2 trial, investigators collect data regarding patient willingness to try the VR treatment, including reasons why they did or did not find the intervention to be acceptable for use. Researchers should collect and report acceptability data using techniques such as focus groups, cognitive interviews, or structured questionnaires.
In the context of a VR2 trial, feasibility is the degree to which the VR treatment can be successfully integrated within the flow of usual care. The committee noted that even the best designed VR treatments can face implementation challenges when applied on the front lines of health care delivery . It is wise for developers to understand potential barriers early and often, identify workarounds and solutions to these barriers, and only then consider testing their interventions in VR3 RCT trials. For example, patients and providers often seek information regarding the frequency and “dosing” of a VR treatment; these details could be manually collected in the context of a VR2 trial. Similarly, treatments deployed in a clinical environment may be unfamiliar to doctors, nurses, and other health care providers, giving researchers an opportunity to study the interaction among staff and proactively identify areas of confusion or misuse. The committee recommends including a table that enumerates patient, provider, technical, and operational barriers to use; identifies root causes; and offers solutions to enhance effectiveness in future clinical applications.
The VR2 trial offers an early opportunity to evaluate patient tolerability of the VR treatment, including both hardware and software components. Researchers should measure and report the prevalence of patient-reported physical (eg, vertigo, nausea, and “cybersickness”) and emotional (eg, fear and anxiety) adverse effects of the VR treatment, along with any discomfort or inconvenience related to the VR equipment (eg, ill-fitting headset, facial or nasal pain, inability to explore the three-dimensional environment fully due to limited mobility).
Cybersickness (or VR sickness) is a unique side effect of VR. There a several different terms used interchangeably within the literature, such as simulator sickness or “sim sickness,” although some believe they are different types of motion sickness . When the vestibular system and oculomotor system notice a discrepancy between reality and the virtual environment, one or more of following symptoms ensue: eyestrain, nausea, fatigue, headache, blurred vision, and postural instability [ ]. The specific mechanism of cybersickness is still unknown.
|Trial attribute||Best practice|
|Assessment of acceptability|
|Assessment of feasibility|
|Assessment of tolerability|
|Assessment of initial clinical efficacy|
aVR: virtual reality.
bPRO: patient-reported outcome.
Recommendations for developers already exist [, ]: appropriately accelerate within the program [ , ], anticipate changes in direction [ ], affect changes in the field of view [ ], establish realistic virtual avatar movements, reduce drops in the frame rate below 60 fps [ ], blur the display with movement [ ], and provide other solutions at the level of program design.
There are also several strategies for medical staff and researchers including habituation , assessment of the risk of side effects before the intervention [ ], use of oculomotor exercises before the intervention [ ], and diaphragmatic breathing during the intervention [ ]. One of the most useful strategies is to limit the total duration of each treatment session, particularly early in the process [ ].
The VR-CORE recommends assessing for side effects at every phase (VR1, VR2, and VR3). Regarding assessment scales, the Simulator Sickness Questionnaire is the most commonly used scale in the literature [, , , ].
Initial Clinical Efficacy
Although the VR2 trial is not designed to definitively test whether a VR treatment is efficacious or effective, it offers an early opportunity to measure efficacy within the context of a small clinical trial. There is no requirement in a VR2 trial to include a control group, although uncontrolled case series carry a higher risk of bias than controlled studies; even studies with nonrandomized concurrent controls, “wait list” controls, or retrospective controls may reduce the risk of bias as compared to an uncontrolled series.
Regardless of the inclusion of a control group, investigators should identify a clinically relevant and validated patient- reported outcome (PRO) to evaluate the evidence of efficacy. For example, a study evaluating pain might include a standard 11-point numeric rating scale  before and after exposure to the VR treatment. A study evaluating stroke rehabilitation might measure physical function with the National Institutes of Health Patient Reported Outcomes Measurement Information System [ ]. Selection of the most appropriate PRO is at the discretion of the research team, but should be carefully justified and capture the most salient features of patient-reported health that might improve with the VR treatment.
The most definitive clinical validation of a VR treatment is the VR3 trial, which is a prospective, adequately powered, methodologically rigorous RCT evaluating clinical outcomes and safety in target patients receiving the VR treatment as compared to a control condition. Although the therapeutic mechanism of action may be studied as a secondary goal in a VR3 trial (eg, through neuroimaging, blood biomarkers, and physiologic testing), the principal goal is to evaluate the treatment’s impact on a clinically meaningful patient outcome rather than surrogate markers.
Although the committee acknowledged understandable costs and resource barriers involved in conducting VR3 trials, there was broad agreement that RCTs are of equal scientific importance in therapeutic VR as any other form of treatment and should be prioritized whenever possible. Multicenter collaborations may facilitate VR3 trials by combining patients and resources through shared protocols. The features of a VR3 trials are described below and summarized in.
Standardization of Intervention and Patient Population
Having been developed in a VR1 study and initially tested in a VR2 trial, the study intervention should be clearly described in preparation for a VR3 trial. Researchers should provide details regarding the equipment used; visualizations employed (with representative screenshots or videos); and frequency, duration, and timing of use. Optimally, the intervention should be manualized, and at the very least, enough details should be provided to allow other investigators to repeat the trial, if desired. The Template for Intervention Description and Replication checklist provides a useful framework for describing study interventions  and should be applied to VR treatments. The target patient population should be clearly described, including explicit inclusion and exclusion criteria employed. Certain exclusion criteria may be standardized among VR trials, such as a history of significant motion sickness, active nausea, and vomiting or epilepsy.
Selection of Control Condition
The committee acknowledged that there is no perfect or standardized control condition for all VR treatment trials; the optimal control depends on the patient population, proposed mechanism of action of the intervention, and clinical setting, among other considerations. Selection of the control is at the discretion of the research team but should be justified and explained. The committee described a hierarchy of control conditions, ranging from “usual care” without any active intervention to passive visualizations on a two-dimensional screen and nonimmersive visualizations within a headset, immersive but passive experiences within a headset, and immersive and active experiences within a headset. Selection of the optimal control may be guided by considering the hypothesized target of engagement and the proposed mechanism of action.
Randomization should be described and ideally achieved using an appropriate computer program (eg, MS Excel Random Number Generator)  or random number tables without involvement of the investigators who enrolled the patients.
Blinding and Concealment of Allocation
The committee acknowledged that blinding and concealment can be challenging, but they identified techniques to incorporate these RCT principles within the constraints of VR research. For example, Spiegel and colleagues (2017) achieved concealment of allocation in an RCT comparing a library of VR content to a “health and wellness” television channel in hospitalized patients experiencing pain . At the time of consent, the researchers explained to patients that the study compared “two different audiovisual experiences designed to reduce pain,” but did not describe the details of the competing interventions. Patients randomized to the television intervention did not know that VR was the other condition and vice versa. This approach may reduce the “novelty effect” of receiving VR rather than a familiar experience like television. Equipoise may also be achieved by exposing patients in both arms to headsets, but varying the content viewed within the headset (eg, immersive vs nonimmersive, active vs passive). At a minimum, study analysts should be blinded to patient group allocation, allowing for unbiased evaluation of the data without the knowledge of the study group. Patients should be asked not to reveal details of the program they experienced to decrease the chance of unblinding the study analysts. The measurement of perceived group assignment at the end of the study can help assess the success of blinding within the study. This should be done at the discretion of the research team.
Like the VR2 trial, VR3 trials must prespecify a clinically relevant and validated PRO as the primary endpoint. The study must be appropriately powered to demonstrate a minimally clinically important difference (MCID)  in that endpoint between the VR treatment and control arms. The psychometrics of PRO measurement are beyond the scope of this document, but existing references may assist investigators in protocol development [ , ]. Secondary endpoints may include a variety of clinical, imaging, biometric, and physiologic surrogate markers, as deemed appropriate by the study team. Like VR2 trials, potential adverse events must be prospectively measured and reported.
VR3 studies should monitor patients for a sufficient period to determine whether the VR treatment meaningfully impacts clinically important outcomes. One-time, short-term evaluations may be insufficient to evaluate the true clinical value of an intervention. Follow-up over several days may be appropriate if the study only focuses on hospital stay, but measurement over weeks, or even months, may be necessary to assess the impact on long-term clinical benefits.
Presentation and Analysis of Results
VR-CORE recommends that the primary outcome be reported as the before and after difference in difference between study arms, with accompanying 95% CIs. For example, the change in the mean PRO score before and after the VR intervention should be compared against the change in the mean PRO score before and after the control intervention. In addition, the panel recommends predefining a binary response criterion, guided by the MCID of the primary endpoint. The proportion achieving the MCID should be reported and compared between groups, and the resulting number needed to treat should be calculated.
|Trial attribute||Best practices|
|Selecting control condition|
|Blinding and concealment of allocation|
|Presentation and analysis of results|
|Reporting the trial|
aVR: virtual reality.
bTIDIER: Template for Intervention Description and Replication.
cPRO: patient-reported outcome.
dMCID: minimally clinically important difference.
eCONSORT: Consolidated Standards for Reporting Trials
The primary analyses should use the intention-to-treat population, including all patients randomized regardless of follow-up or receipt of study interventions. However, per-protocol analysis may be appropriate in certain situation, such as if patients refuse the VR treatment after randomization; in this instance, reporting the rate of refusal would be important, but investigators might also seek to compare therapeutic responses only among those receiving the intervention.
Multivariable analysis may be useful in adjusting for prespecified confounding factors (especially if not equally distributed in the study groups) and exploring independent predictors of outcomes. To perform a multivariable analysis, it is optimal to have at least 10 (preferably, 20) observations for each independent variable included in the multivariable model.
VR3 trials must be registered in a publicly accessible registry (eg, such as ClinicalTrials.gov). All completed trials should be published, regardless of whether they are positive or negative. The Consolidated Standards for Reporting Trials (CONSORT) guidelines provide the framework for reporting RCTs  and should be followed in VR3 trials. VR3 trials must include a CONSORT diagram to demonstrate the flow of patients through each stage of the trial, including the number screened to the number randomized into each study group and the number analyzed.
To improve methodological quality in the therapeutic VR literature, the VR-CORE international working group presents a three-part framework for best practices in developing and testing VR treatments. This framework may be used to facilitate development of high-quality, effective, and safe VR treatments that meaningfully improve patient outcomes. Patients, providers, payers, and regulators should consider this framework when assessing the validity of VR treatments.
BS is supported by the Marc and Sheri Rapaport Fund for Digital Health Sciences and Precision Health. In addition to the authorship team, the following committee members actively participated in the online discussion forums and contributed to this paper: Tom Caruso, MD, MEd - Stanford University School of Medicine, Stanford, CA; Ali Fardinpour, PhD - Wise Realities, Perth, Australia; Diane Gromala, PhD - Simon Fraser University, Surrey, BC; Rafael Grossmann, MD - Eastern Maine Healthcare System, Bangor, ME; Kate Hardy, Clin PsychD - Stanford University School of Medicine, Stanford, CA; Constance Johnson, PhD - The University of Texas Health Science Center, Houston, TX; Ted Jones, PhD - Pain Consultants of East Tennessee, Knoxville, TN; Kate Laver, PhD - Flinders University, Adelaide, Australia; Sheila Parinas, BSN, RN - University of Illinois, Champaign, IL; Les Posen, FAPS, Clin Psych - Flightwise, Melbourne, Australia; Barbara Rothbaum, PhD, ABPP - Emory School of Medicine, Atlanta, GA; David Thomas, PhD - National Institute of Drug Abuse (NIDA), Bethesda, MD; Stefan Van Rooijen, MD PhD - Maastricht University, Maastricht, Netherlands; Herve Rosay, MD - Centre Leon Berard, Lyon, France; Earl Scott, MD - HCA Midwest Physician, Kansas City, MO; Andrea Stevenson Won, PhD - Cornell University, Ithaca, NY.
Conflicts of Interest
BS was the Principal Investigator of a 2016 Virtual Reality (VR) research grant (#CSR211835), administered by his academic institution, from AppliedVR (Los Angeles, California). He is currently the Principal Investigator of a VR research grant (#CSR212943), administered by his academic institution, from Traveler’s Insurance (New York City, NY) and Samsung Electronics (Suwon, South Korea). AR’s research relating to this paper has been funded by the National Institutes of Health (NIH), National Science Foundation, US Army Research Office, Telemedicine and Advanced Technology Research Center, US Army Medical Research Acquisition Activity, Department of Veterans Affairs, and Kesseler Foundation. All other authors (BB, CK, XL, SC, ID, and KB) have no conflicts of interest. Regarding the other Virtual Reality Clinical Outcomes Research Experts members, Dr Rothbaum owns equity in Virtual Better, Inc, which is developing products related to virtual reality research related to this paper. The terms of this arrangement have been approved by Emory University in accordance with its conflict of interest policies. Dr Johnson receives funding through the NIH to study virtual environments. Some of Dr van Rooijen’s VR research has been funded by Phillips, Inc (Amsterdam, Netherlands). All other members of the committee (Tom Caruso, Ali Fardinpour, Diane Gromala, Rafael Grossmann, Kate Hardy, Ted Jones, Kate Laver, Sheila Parinas, Les Posen, David Thomas, Herve Rosay, Earl Scott, and Andrea Stevenson Won) have no conflicts of interest.
Multimedia Appendix 1
Excerpted transcripts of Virtual Reality Committee of Outcomes Research Experts responses to selected discussion topics. Key themes and phraseology included in the manuscript are highlighted. Note that not all committee members responded to all questions.PDF File (Adobe PDF File), 76KB
- Dascal J, Reid M, IsHak WW, Spiegel B, Recacho J, Rosen B, et al. Virtual Reality and Medical Inpatients: A Systematic Review of Randomized, Controlled Trials. Innov Clin Neurosci 2017;14(1-2):14-21 [FREE Full text] [Medline]
- Mahrer NE, Gold JI. The use of virtual reality for pain control: a review. Curr Pain Headache Rep 2009 Apr;13(2):100-109. [Medline]
- Valmaggia LR, Latif L, Kempton MJ, Rus-Calafell M. Virtual reality in the psychological treatment for mental health problems: An systematic review of recent evidence. Psychiatry Res 2016 Feb 28;236:189-195. [CrossRef] [Medline]
- Wismeijer AAJ, Vingerhoets AJJM. The use of virtual reality and audiovisual eyeglass systems as adjunct analgesic techniques: a review of the literature. Ann Behav Med 2005 Dec;30(3):268-278. [CrossRef] [Medline]
- Lohse KR, Hilderman CGE, Cheung KL, Tatla S, Van der Loos HFM. Virtual reality therapy for adults post-stroke: a systematic review and meta-analysis exploring virtual environments and commercial games in therapy. PLoS One 2014;9(3):e93318 [FREE Full text] [CrossRef] [Medline]
- Botella C, Serrano B, Baños RM, Garcia-Palacios A. Virtual reality exposure-based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the adequacy of the treatment protocol, and its acceptability. Neuropsychiatr Dis Treat 2015;11:2533-2545 [FREE Full text] [CrossRef] [Medline]
- Gonçalves R, Pedrozo AL, Coutinho ESF, Figueira I, Ventura P. Efficacy of virtual reality exposure therapy in the treatment of PTSD: a systematic review. PLoS One 2012;7(12):e48469 [FREE Full text] [CrossRef] [Medline]
- de Carvalho MR, Dias TRDS, Duchesne M, Nardi AE, Appolinario JC. Virtual Reality as a Promising Strategy in the Assessment and Treatment of Bulimia Nervosa and Binge Eating Disorder: A Systematic Review. Behav Sci (Basel) 2017 Jul 09;7(3) [FREE Full text] [CrossRef] [Medline]
- Chirico A, Lucidi F, De Laurentiis M, Milanese C, Napoli A, Giordano A. Virtual Reality in Health System: Beyond Entertainment. A Mini-Review on the Efficacy of VR During Cancer Treatment. J Cell Physiol 2016 Feb;231(2):275-287. [CrossRef] [Medline]
- da Costa RMEM, de Carvalho LAV. The acceptance of virtual reality devices for cognitive rehabilitation: a report of positive results with schizophrenia. Comput Methods Programs Biomed 2004 Mar;73(3):173-182. [CrossRef] [Medline]
- Macedo M, Marques A, Queirós C. Virtual reality in assessment and treatment of schizophrenia: a systematic review. J Bras Psiquiatr 2015 Mar;64(1):70-81. [CrossRef]
- Luque-Moreno C, Ferragut-Garcías A, Rodríguez-Blanco C, Heredia-Rizo AM, Oliva-Pascual-Vaca J, Kiper P, et al. A Decade of Progress Using Virtual Reality for Poststroke Lower Extremity Rehabilitation: Systematic Review of the Intervention Methods. Biomed Res Int 2015;2015:342529 [FREE Full text] [CrossRef] [Medline]
- Malloy KM, Milling LS. The effectiveness of virtual reality distraction for pain reduction: a systematic review. Clin Psychol Rev 2010 Dec;30(8):1011-1018. [CrossRef] [Medline]
- Parsons TD, Rizzo AA. Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis. J Behav Ther Exp Psychiatry 2008 Sep;39(3):250-261. [CrossRef] [Medline]
- Powers MB, Emmelkamp PMG. Virtual reality exposure therapy for anxiety disorders: A meta-analysis. J Anxiety Disord 2008;22(3):561-569. [CrossRef] [Medline]
- Rodrigues-Baroni JM, Nascimento LR, Ada L, Teixeira-Salmela LF. Walking training associated with virtual reality-based training increases walking speed of individuals with chronic stroke: systematic review with meta-analysis. Braz J Phys Ther 2014;18(6):502-512 [FREE Full text] [CrossRef] [Medline]
- de Rooij IJM, van de Port IGL, Meijer JG. Effect of Virtual Reality Training on Balance and Gait Ability in Patients With Stroke: Systematic Review and Meta-Analysis. Phys Ther 2016 Dec;96(12):1905-1918. [CrossRef] [Medline]
- Triberti S, Repetto C, Riva G. Psychological factors influencing the effectiveness of virtual reality-based analgesia: a systematic review. Cyberpsychol Behav Soc Netw 2014 Jun;17(6):335-345. [CrossRef] [Medline]
- Rus-Calafell M, Garety P, Sason E, Craig TJK, Valmaggia LR. Virtual reality in the assessment and treatment of psychosis: a systematic review of its utility, acceptability and effectiveness. Psychol Med 2018 Dec;48(3):362-391. [CrossRef] [Medline]
- Girard B, Turcotte V, Bouchard S, Girard B. Crushing virtual cigarettes reduces tobacco addiction and treatment discontinuation. Cyberpsychol Behav 2009 Oct;12(5):477-483. [CrossRef] [Medline]
- Kuntze MF, Stoermer R, Mager R, Roessler A, Mueller-Spahn F, Bullinger AH. Immersive virtual environments in cue exposure. Cyberpsychol Behav 2001 Aug;4(4):497-501. [CrossRef] [Medline]
- Bordnick PS, Traylor AC, Carter BL, Graap KM. A Feasibility Study of Virtual Reality-Based Coping Skills Training for Nicotine Dependence. Res Soc Work Pract 2012 May;22(3):293-300 [FREE Full text] [CrossRef] [Medline]
- Carter BL, Bordnick P, Traylor A, Day SX, Paris M. Location and longing: the nicotine craving experience in virtual reality. Drug Alcohol Depend 2008 May 01;95(1-2):73-80 [FREE Full text] [CrossRef] [Medline]
- Thompson-Lake DGY, Cooper KN, Mahoney JJ, Bordnick PS, Salas R, Kosten TR, et al. Withdrawal Symptoms and Nicotine Dependence Severity Predict Virtual Reality Craving in Cigarette-Deprived Smokers. Nicotine Tob Res 2015 Jul;17(7):796-802 [FREE Full text] [CrossRef] [Medline]
- Bordnick PS, Traylor A, Copp HL, Graap KM, Carter B, Ferrer M, et al. Assessing reactivity to virtual reality alcohol based cues. Addict Behav 2008 Jun;33(6):743-756. [CrossRef] [Medline]
- Maples-Keller JL, Price M, Rauch S, Gerardi M, Rothbaum BO. Investigating Relationships Between PTSD Symptom Clusters Within Virtual Reality Exposure Therapy for OEF/OIF Veterans. Behav Ther 2017 Dec;48(2):147-155. [CrossRef] [Medline]
- Maples-Keller JL, Bunnell BE, Kim S, Rothbaum BO. The Use of Virtual Reality Technology in the Treatment of Anxiety and Other Psychiatric Disorders. Harv Rev Psychiatry 2017;25(3):103-113 [FREE Full text] [CrossRef] [Medline]
- Meyerbröker K, Emmelkamp PMG. Virtual reality exposure therapy in anxiety disorders: a systematic review of process-and-outcome studies. Depress Anxiety 2010 Oct;27(10):933-944. [CrossRef] [Medline]
- Emmelkamp PMG, Krijn M, Hulsbosch AM, de Vries S, Schuemie MJ, van der Mast CA. Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia. Behav Res Ther 2002 May;40(5):509-516. [Medline]
- Bouchard S, Dumoulin S, Robillard G, Guitard T, Klinger E, Forget H, et al. Virtual reality compared with exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial. Br J Psychiatry 2017 Dec;210(4):276-283. [CrossRef] [Medline]
- Morina N, Ijntema H, Meyerbröker K, Emmelkamp PM. Can virtual reality exposure therapy gains be generalized to real-life? A meta-analysis of studies applying behavioral assessments. Behav Res Ther 2015 Nov;74:18-24. [CrossRef] [Medline]
- Rizzo AA, Difede J, Rothbaum BO, Johnston S, McLay RN, Reger G, et al. VR PTSD exposure therapy results with active duty OIF/OEF combatants. Stud Health Technol Inform 2009;142:277-282. [Medline]
- Maples-Keller JL, Yasinski C, Manjin N, Rothbaum BO. Virtual Reality-Enhanced Extinction of Phobias and Post-Traumatic Stress. Neurotherapeutics 2017 Jul;14(3):554-563 [FREE Full text] [CrossRef] [Medline]
- Difede J, Cukor J, Jayasinghe N, Patt I, Jedel S, Spielman L, et al. Virtual reality exposure therapy for the treatment of posttraumatic stress disorder following September 11, 2001. J Clin Psychiatry 2007 Nov;68(11):1639-1647. [Medline]
- Freeman D. Studying and treating schizophrenia using virtual reality: a new paradigm. Schizophr Bull 2008 Jul;34(4):605-610 [FREE Full text] [CrossRef] [Medline]
- Kim K, Kim J, Kim J, Park D, Jang HJ, Ku J, et al. Characteristics of social perception assessed in schizophrenia using virtual reality. Cyberpsychol Behav 2007 Apr;10(2):215-219. [CrossRef] [Medline]
- Sorkin A, Weinshall D, Modai I, Peled A. Improving the accuracy of the diagnosis of schizophrenia by means of virtual reality. Am J Psychiatry 2006 Mar;163(3):512-520. [CrossRef] [Medline]
- Veling W, Pot-Kolder R, Counotte J, van Os J, van der Gaag M. Environmental Social Stress, Paranoia and Psychosis Liability: A Virtual Reality Study. Schizophr Bull 2016 Dec;42(6):1363-1371 [FREE Full text] [CrossRef] [Medline]
- Ferrer-Garcia M, Gutiérrez-Maldonado J, Riva G. Virtual Reality Based Treatments in Eating Disorders and Obesity: A Review. J Contemp Psychother 2013 Jun 25;43(4):207-221. [CrossRef]
- Cesa GL, Manzoni GM, Bacchetta M, Castelnuovo G, Conti S, Gaggioli A, et al. Virtual reality for enhancing the cognitive behavioral treatment of obesity with binge eating disorder: randomized controlled study with one-year follow-up. J Med Internet Res 2013 Jun 12;15(6):e113 [FREE Full text] [CrossRef] [Medline]
- Riva G. The key to unlocking the virtual body: virtual reality in the treatment of obesity and eating disorders. J Diabetes Sci Technol 2011 Mar 01;5(2):283-292 [FREE Full text] [CrossRef] [Medline]
- Gutiérrez-Maldonado J, Ferrer-García M, Caqueo-Urízar A, Moreno E. Body image in eating disorders: the influence of exposure to virtual-reality environments. Cyberpsychol Behav Soc Netw 2010 Oct;13(5):521-531. [CrossRef] [Medline]
- Ferrer-García M, Gutiérrez-Maldonado J. The use of virtual reality in the study, assessment, and treatment of body image in eating disorders and nonclinical samples: a review of the literature. Body Image 2012 Jan;9(1):1-11. [CrossRef] [Medline]
- Marco JH, Perpiñá C, Botella C. Effectiveness of cognitive behavioral therapy supported by virtual reality in the treatment of body image in eating disorders: one year follow-up. Psychiatry Res 2013 Oct 30;209(3):619-625. [CrossRef] [Medline]
- Gutiérrez-Maldonado J, Pla-Sanjuanelo J, Ferrer-García M. Cue-exposure software for the treatment of bulimia nervosa and binge eating disorder. Psicothema 2016 Nov;28(4):363-369. [CrossRef] [Medline]
- Subramanian SK, Lourenço CB, Chilingaryan G, Sveistrup H, Levin MF. Arm motor recovery using a virtual reality intervention in chronic stroke: randomized control trial. Neurorehabil Neural Repair 2013 Jan;27(1):13-23. [CrossRef] [Medline]
- Crosbie JH, Lennon S, McGoldrick MC, McNeill MDJ, McDonough SM. Virtual reality in the rehabilitation of the arm after hemiplegic stroke: a randomized controlled pilot study. Clin Rehabil 2012 Sep;26(9):798-806. [CrossRef] [Medline]
- Bergeron M, Lortie CL, Guitton MJ. Use of Virtual Reality Tools for Vestibular Disorders Rehabilitation: A Comprehensive Analysis. Adv Med 2015;2015:916735 [FREE Full text] [CrossRef] [Medline]
- You SH, Jang SH, Kim Y, Kwon Y, Barrow I, Hallett M. Cortical reorganization induced by virtual reality therapy in a child with hemiparetic cerebral palsy. Dev Med Child Neurol 2005 Sep;47(9):628-635. [Medline]
- Rothbaum BO, Hodges LF, Kooper R, Opdyke D, Williford JS, North M. Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia. Am J Psychiatry 1995 Apr;152(4):626-628. [CrossRef] [Medline]
- Hoffman HG, Richards TL, Bills AR, Van Oostrom T, Magula J, Seibel EJ, et al. Using FMRI to study the neural correlates of virtual reality analgesia. CNS Spectr 2006 Jan;11(1):45-51. [Medline]
- Hoffman HG, Richards TL, Coda B, Bills AR, Blough D, Richards AL, et al. Modulation of thermal pain-related brain activity with virtual reality: evidence from fMRI. Neuroreport 2004 Jun 07;15(8):1245-1248. [Medline]
- Li A, Montaño Z, Chen VJ, Gold JI. Virtual reality and pain management: current trends and future directions. Pain Manag 2011 Mar;1(2):147-157 [FREE Full text] [CrossRef] [Medline]
- McCaul KD, Malott JM. Distraction and coping with pain. Psychol Bull 1984 May;95(3):516-533. [Medline]
- U.S Food & Drug Administration. Step 3: Clinical Research URL: https://www.fda.gov/ForPatients/Approvals/Drugs/ucm405622.htm [accessed 2017-10-09] [WebCite Cache]
- Laine L, Spiegel B, Rostom A, Moayyedi P, Kuipers EJ, Bardou M, et al. Methodology for randomized trials of patients with nonvariceal upper gastrointestinal bleeding: recommendations from an international consensus conference. Am J Gastroenterol 2010 Mar;105(3):540-550. [CrossRef] [Medline]
- Grade Working Group. GRADE URL: http://www.gradeworkinggroup.org [accessed 2006-04-07] [WebCite Cache]
- Slack. URL: https://slack.com/ [accessed 2018-12-13] [WebCite Cache]
- Ericsson AKS, Simon H. Protocol Analysis: Verbal Reports as Data. Cambridge, Massachusetts: MIT Press; 1993.
- Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior. J Med Internet Res 2016 Dec 16;18(12):e317 [FREE Full text] [CrossRef] [Medline]
- Field Guide to Human-Centered Design. Palo Alto, California: IDEO.org/Design Kit; 2015.
- Brown T. Design thinking. Harv Bus Rev 2008 Jun;86(6):84-92, 141. [Medline]
- Jerald J. The VR Book: Human-centered Design For Virtual Reality. New York, NY: Morgan & Claypool Publishers; 2015.
- Carr S, Halliday A, King A, Liedtka J, Lockwood T. The Influence of Design Thinking in Businessome Preliminary Observations. Design Management Review 2010;21:58-63.
- Oculus. 2017. Introducing Chromecast for Gear VR: Step into VR and Stream to Your Flatscreen Internet URL: https://www.oculus.com/blog/introducing-chromecast-for-gear-vr-step-into-vr-and-stream-to-your-flatscreen/ [accessed 2018-07-09] [WebCite Cache]
- Github. URL: https://github.com/ [accessed 2017-07-05] [WebCite Cache]
- Mosadeghi S, Reid MW, Martinez B, Rosen BT, Spiegel BMR. Feasibility of an Immersive Virtual Reality Intervention for Hospitalized Patients: An Observational Cohort Study. JMIR Ment Health 2016 Jun 27;3(2):e28 [FREE Full text] [CrossRef] [Medline]
- Stanney KM, Kennedy RS, Drexler JM. Cybersickness is not simulator sickness. : SAGE Publications; 1997 Presented at: In Proceedings of the Human Factors and Ergonomics Society annual meeting; September 22-26, 1997; Albuquerque, New Mexico URL: https://doi.org/10.1177/107118139704100292
- Bruck S, Watters PA. Estimating cybersickness of simulated motion using the simulator sickness questionnaire (SSQ): A controlled study. 2009 Presented at: International Conference on Computer Graphics, Imaging and Visualization; Aug 11-14, 2009; Tianjin, China p. 486-488. [CrossRef]
- Porcino TM, Clua E, Trevisan D, Vasconcelos CN, Valente L. Minimizing cyber sickness in head mounted display systems: design guidelines and applications. 2017 Apr Presented at: IEEE 5th International Conference; April 2-4, 2017; Perth, Western Australia p. 1-6. [CrossRef]
- Yao R, Heath T, Davies A, Forsyth T, Mitchell N, Hoberman P. Oculus. 2014. Oculus VR Best Practices Guide URL: http://brianschrank.com/vrgames/resources/OculusBestPractices.pdf [accessed 2018-10-22] [WebCite Cache]
- Mazloumi Gavgani A, Hodgson DM, Nalivaiko E. Effects of visual flow direction on signs and symptoms of cybersickness. PLoS One 2017;12(8):e0182790 [FREE Full text] [CrossRef] [Medline]
- Lin J, Abi-Rached H, Lahav M. Virtual guiding avatar: An effective procedure to reduce simulator sickness in virtual environments. In: Proceedings of the SIGCHI conference on Human factors in computing systems. 2004 Presented at: ACM; April 24-29, 2004; Vienna, Austria p. 719-726. [CrossRef]
- Budhiraja P, Miller MR, Modi AK, Forsyth D. ARXIV. 2017 Oct 06. Rotation Blurring: Use of Artificial Blurring to Reduce Cybersickness in Virtual Reality First Person Shooters URL: https://arxiv.org/pdf/1710.02599.pdf [accessed 2018-12-13] [WebCite Cache]
- Gavgani AM, Nesbitt KV, Blackmore KL, Nalivaiko E. Profiling subjective symptoms and autonomic changes associated with cybersickness. Auton Neurosci 2017 Mar;203:41-50. [CrossRef] [Medline]
- Tyrrell R, Sarig-Bahat H, Williams K, Williams G, Treleaven J. Simulator sickness in patients with neck pain and vestibular pathology during virtual reality tasks. Virtual Reality 2017 Sep 22;22(3):211-219 [FREE Full text] [CrossRef]
- Park WD, Kim YS, Jang SW, Kim GA, Kim YH, Son W. A study on cyber sickness reduction by oculo-motor exercise performed immediately prior to viewing virtual reality (VR) content on head mounted display (HMD). JVE Journals 2017(14):260-264 [FREE Full text] [CrossRef]
- Russell MEB, Hoffman B, Stromberg S, Carlson CR. Use of controlled diaphragmatic breathing for the management of motion sickness in a virtual reality environment. Appl Psychophysiol Biofeedback 2014 Dec;39(3-4):269-277. [CrossRef] [Medline]
- Dworkin RH, Turk DC, Farrar JT, Haythornthwaite JA, Jensen MP, Katz NP, IMMPACT. Core outcome measures for chronic pain clinical trials: IMMPACT recommendations. Pain 2005 Jan;113(1-2):9-19. [CrossRef] [Medline]
- Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, PROMIS Cooperative Group. The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care 2007 May;45(5 Suppl 1):S3-S11 [FREE Full text] [CrossRef] [Medline]
- Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014 Mar 07;348:g1687. [CrossRef] [Medline]
- Saghaei M. An overview of randomization and minimization programs for randomized clinical trials. J Med Signals Sens 2011 Jan;1(1):55-61 [FREE Full text] [Medline]
- Tashjian VC, Mosadeghi S, Howard AR, Lopez M, Dupuy T, Reid M, et al. Virtual Reality for Management of Pain in Hospitalized Patients: Results of a Controlled Trial. JMIR Ment Health 2017 Mar 29;4(1):e9 [FREE Full text] [CrossRef] [Medline]
- Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Control Clin Trials 1989 Dec;10(4):407-415. [Medline]
- Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care 2003 May;41(5):582-592. [CrossRef] [Medline]
- Curran PJ, Bauer DJ. The disaggregation of within-person and between-person effects in longitudinal models of change. Annu Rev Psychol 2011;62:583-619 [FREE Full text] [CrossRef] [Medline]
- Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996 Aug 28;276(8):637-639. [Medline]
|CONSORT: Consolidated Standards for Reporting Trials|
|HMD: head-mounted display|
|MCID: minimally clinically important difference|
|PRO: patient-reported outcome|
|RCT: randomized controlled trial|
|TIDIER: Template for Intervention Description and Replication|
|VR: virtual reality|
|VR-CORE: Virtual Reality Clinical Outcomes Research Experts|
Edited by W Greenleaf; submitted 19.08.18; peer-reviewed by C Fowler, J Torous; comments to author 01.10.18; revised version received 23.10.18; accepted 23.10.18; published 31.01.19
©Brandon Birckhead, Carine Khalil, Xiaoyu Liu, Samuel Conovitz, Albert Rizzo, Itai Danovitch, Kim Bullock, Brennan Spiegel. Originally published in JMIR Mental Health (http://mental.jmir.org), 31.01.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.