Background

JMH

JMIR Ment Health

JMIR Mental Health

2368-7959

JMIR Publications

Toronto, Canada

v6i1e11973

30702436

10.2196/11973

Viewpoint

Recommendations for Methodology of Virtual Reality Clinical Trials in Health Care by an International Working Group: Iterative Study

Greenleaf

Walter

Fowler

Christopher

Torous

John

Birckhead

Brandon

MD 1

http://orcid.org/0000-0002-5822-7318

Khalil

Carine

PhD 1 2

http://orcid.org/0000-0001-5071-0500

Liu

Xiaoyu

MPH 1 3

http://orcid.org/0000-0003-4628-980X

Conovitz

Samuel

MPH 1 3

http://orcid.org/0000-0002-5192-046X

Rizzo

Albert

PhD 4

http://orcid.org/0000-0003-2647-7731

Danovitch

Itai

MD, MBA 5

http://orcid.org/0000-0001-9580-6450

Bullock

Kim

MD 6

http://orcid.org/0000-0003-1214-1356

Spiegel

Brennan

MD, MSHS 1

Division of Health Services Research Department of Medicine Cedars-Sinai Health System

Pacific Theaters Building, Suite 800

116 North Robertson Boulevard

Los Angeles, CA, 90048

United States 1 310 423 6784 Brennan.Spiegel@cshs.org

http://orcid.org/0000-0002-4608-6896

1 Division of Health Services Research Department of Medicine Cedars-Sinai Health System

Los Angeles, CA

United States 2 Laboratoire Interdisciplinaire de Recherche Appliquée en Économie de la Santé Paris Descartes University

Paris

France 3 Department of Health Policy and Management Fielding School of Public Health University of California, Los Angeles

Los Angeles, CA

United States 4 Institute for Creative Technologies University of Southern California

Los Angeles, CA

United States 5 Department of Psychiatry Cedars-Sinai Health System

Los Angeles, CA

United States 6 Department of Psychiatry Stanford University School of Medicine

Palo Alto, CA

United States

Corresponding Author: Brennan Spiegel Brennan.Spiegel@cshs.org

01 2019

31 01 2019

6 1

e11973

19 8 2018 1 10 2018 23 10 2018 23 10 2018

©Brandon Birckhead, Carine Khalil, Xiaoyu Liu, Samuel Conovitz, Albert Rizzo, Itai Danovitch, Kim Bullock, Brennan Spiegel. Originally published in JMIR Mental Health (http://mental.jmir.org), 31.01.2019.

2019

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.

Background

Therapeutic virtual reality (VR) has emerged as an efficacious treatment modality for a wide range of health conditions. However, despite encouraging outcomes from early stage research, a consensus for the best way to develop and evaluate VR treatments within a scientific framework is needed.

Objective

We aimed to develop a methodological framework with input from an international working group in order to guide the design, implementation, analysis, interpretation, and communication of trials that develop and test VR treatments.

Methods

A group of 21 international experts was recruited based on their contributions to the VR literature. The resulting Virtual Reality Clinical Outcomes Research Experts held iterative meetings to seek consensus on best practices for the development and testing of VR treatments.

Results

The interactions were transcribed, and key themes were identified to develop a scientific framework in order to support best practices in methodology of clinical VR trials. Using the Food and Drug Administration Phase I-III pharmacotherapy model as guidance, a framework emerged to support three phases of VR clinical study designs—VR1, VR2, and VR3. VR1 studies focus on content development by working with patients and providers through the principles of human-centered design. VR2 trials conduct early testing with a focus on feasibility, acceptability, tolerability, and initial clinical efficacy. VR3 trials are randomized, controlled studies that evaluate efficacy against a control condition. Best practice recommendations for each trial were provided.

Conclusions

Patients, providers, payers, and regulators should consider this best practice framework when assessing the validity of VR treatments.

clinical trials consensus virtual reality

Introduction

Therapeutic virtual reality (VR) is an innovative treatment modality to manage a broad range of health conditions and is gaining considerable attention [1-19]. Users of VR wear a head-mounted display (HMD) with a close-proximity screen that creates a sense of being transported into life-like, three-dimensional worlds. VR has been used to assess and treat a wide variety of medical, surgical, psychiatric, and neurocognitive conditions including pain [1,2,4,9,13,18], addiction [20-25], anxiety disorders [3,6,7,14-15,26-34], schizophrenia [10,11,19,35-38], eating disorders [1,8,39-45], stroke rehabilitation [5,12,16-17,45-47], vestibular disorders [48], and movement disorders [49]. One of the first published uses of HMD-based therapy was the treatment of acrophobia in 1995 [50]. There have also been functional magnetic resonance imaging studies demonstrating the effect of VR on the brain during receipt of a painful stimuli [51,52]. VR is thought to work through a combination of distraction, extinction learning, cognitive-behavioral principles, mindful meditation, stress reduction, gate-control theory, and the spotlight theory of attention [53,54]. Importantly, VR has become increasingly portable, immersive, and vivid, which has enabled the technology to be used in a broad range of inpatient and outpatient applications.

As the use of therapeutic VR expands, it is essential that guidelines are established to ensure scientific rigor in the development and evaluation of VR applications, similar to established standards for pharmacotherapies [30,55]. VR developers would benefit from systematic guidance on best practices for designing and conducting VR clinical trials. To fulfil this unmet need, we garnered input from an international working group, called the Virtual Reality Clinical Outcomes Research Experts (VR-CORE) committee. This paper presents the resulting best practice framework informed by expert input, along with specific recommendations on ways to conduct high-quality VR treatment trials. Although the focus of this paper is VR, the framework also applies to other emerging “XR” technologies, including augmented reality and mixed reality, as the methodologic considerations for clinical trials are largely similar across XR platforms.

Methods Identifying Virtual Reality Clinical Outcomes Research Experts

We performed a systematic review of randomized controlled trials (RCTs) using therapeutic VR to help identify eligible VR-CORE committee members through review of author lists. To cover the largest breadth of studies, the literature search focused on existing meta-analyses of therapeutic VR RCTs identified through search of PubMed, Google Scholar, and the Cochrane Database of Systematic Reviews using a combination of keywords: (“virtual reality” OR “VR”) AND (“review [pt]” OR “systematic review [pt]” OR “meta-anal*” OR “metaanaly*”). Based on our literature search, and supplemented by recommendations from established experts, we developed a multidisciplinary group for the VR-CORE, including experts in fields relevant to developing and testing VR treatments such as user-centered design principles, software design, epidemiology, statistics, and clinical trial methodology. The committee was formulated to balance expertise across clinical disciplines (medicine, pediatrics, surgery, psychology, psychiatry, neuroscience, anesthesia, nursing, and rehabilitation) and reflect multinational perspectives.

Collecting Input From the Virtual Reality Clinical Outcomes Research Experts

To obtain systematic feedback from the committee, a series of electronic meetings were held to collect and synthesize structured input. An iterative approach was modeled after similar processes were employed by our previous working groups in other fields of health care [56,57]. Using an online meeting platform that allows users to view and react to each other’s comments [58], committee members initially responded to open-ended “think aloud” prompts [59] (eg, “When you think about the current state of the clinical VR research, what comes to your mind?”), followed by increasingly specific probes prepared by the moderators (eg, “What should be the role of human centered design principles in developing VR treatments?”). The full set of questions and responses is listed in Multimedia Appendix 1. The active members of the VR-CORE at the time of this discussion are listed in the Acknowledgments section. Emergent themes and proposed methodologic best practices were culled from the online dialogue, and the resulting recommendations were distributed to the members for synthesis and iterative rephrasing.

Results Emergent Themes from Virtual Reality Clinical Outcomes Research Experts Meetings

Multimedia Appendix 1 provides excerpted transcripts of the VR-CORE responses to discussion topics. Key themes drawn from the online dialogue are summarized in the following sections.

Perceptions Regarding the Current State of Clinical Virtual Reality Research

Committee members described the current state of clinical VR research as the “Wild West” with a “lack of clear guidelines and standards.” The state of current VR research was described as “heterogeneous,” often focused “more on the tech rather than the theories behind it.” Committee members expressed concern that much of the current research is “merely descriptive” in nature, often insufficiently powered, focused on small case reports and retrospective analyses, and often does not employ experimental designs.

Perceptions About Ways to Improve Virtual Reality Literature

The committee believed it is vital to “include the patients’ voice early and often in the development of VR treatments” and that developers must “carefully, systematically, and meticulously seek the patients’ feedback” through participatory research and design thinking that involves multidisciplinary collaboration. The committee acknowledged the importance of including the voice of providers as well. The committee also called for better definitions and standardization of therapeutic VR study designs.

Most Important Considerations for Designing and Standardizing Clinical Virtual Reality Trials

The committee described various stages for developing and validating VR treatments, beginning with content development in partnership with end-users, progressing through initial clinical testing and safety evaluation, and ending with properly powered RCTs. The committee outlined a wide range of considerations for each stage (Multimedia Appendix 1), including the importance of standardizing control groups, selecting clinically relevant outcome measures, reporting which equipment was used in the trial, accounting for dropouts and disqualified participants, and allowing for pragmatic features of each study design.

Clinical Trial Framework of the Virtual Reality Clinical Outcomes Research Experts The Framework

Although there are fundamental best practices in study design that apply to all biomedical intervention trials, the committee identified VR-specific attributes that are unique considerations for VR trials. Using the Food and Drug Administration Phase I-III pharmacotherapy model as guidance [55] and combining the results of literature synthesis with VR-CORE input, a framework emerged to support three phases of VR clinical study designs, named VR1, VR2, and VR3.

VR1 studies focus on content development by working with patient and provider end-users through principles of human-centered design. VR2 trials conduct early testing with a focus on feasibility, acceptability, tolerability, and initial clinical efficacy. VR3 trials are RCTs that compare clinically important outcomes between intervention groups and a control condition. Each study should undergo ethical review before initiation. Figure 1 summarizes each phase of the VR-CORE model. Best practice recommendations for each trial design are described below.

VR1 Studies

The committee strongly believes that therapeutic VR applications should be designed with direct input from patient and provider end-users. Lack of patient involvement, poor requirement definitions, and nonadaptation to user feedback are some of the common factors that explain failures of digital interventions [60]. Incorporating patients into the design process enables developers to increase the relevance and effectiveness of VR treatments. The committee stresses that VR treatments should be created with acknowledgment of patients’ knowledge, attitudes, beliefs, preferences, and expectations of therapeutic VR. VR-CORE refers to a VR1 study as one that results in the development of VR treatment in partnership with patient and provider end-users and follows best practices for patient-centered design.

After their review of the literature on human-centered design both generally [61,62] and in relation to digital [60] and VR interventions [63], the committee identified three key principles that are fundamental for developing “desirable, feasible and viable” VR treatments [61]. These principles promote empathy, team collaboration, and continuous user feedback (Table 1). The committee believes that the use of these principles allows development teams to better identify users’ needs, incorporate user feedback, and institute rapid cycle improvements that generate more relevant products at lower cost [64]. The key principles for VR1 studies are outlined in Table 1.

Figure 1

Summary of VR1, VR2, VR3. VR: virtual reality.

Table 1

Summary of design principles, strategies, and recommended best practices for VR1 studies.

Design principles and strategies		Best practices
Inspiration through empathizing
	Recruitment	Determine the population of interest (who do we need to hear from?). Think about a variety of factors (age, gender, ethnicity, health conditions, and social position).
	Observation	Learn about patients and their behavior by observing them in a clinically relevant context. Observe what patients do in a specific context and what they see and say.
	Patient interviews	Perform individual cognitive interviews and focus groups with patients to learn about their relevant needs, struggles, experiences, fears, aspirations, and expectations. Document a diverse set of opinions from a variety of patient profiles across ages (eg, above vs below “digital divide”), comorbidities, and experience and comfort with technology (eg, technophiles vs technophobes).
	Expert interviews	Perform cognitive interviews and focus groups with relevant experts representing different points of view such as treating providers and other staff members.
	Journey mapping and personas	Define the patient user and describe the sequence of events in which the patient will experience the virtual reality treatment within the context of their illness experience.
Ideation through team collaboration
	Sharing stories and notes	Collect stories, pictures, impressions, and notes about patients’ experiences and behavior. Share information among team members to generate many ideas through techniques such as storyboarding, storytelling, and mind mapping.
	Generating ideas	Encourage team members to generate ambitious ideas without being judged. The committee believes that idea generation should be distinguished from idea evaluation. After generating ideas, the team evaluates each idea and culls out the most feasible and appropriate idea for prototyping within technical and budgetary constraints.
Prototyping through continuous user feedback
	Building prototype	Convert ideas into tangible figures through drawings or mock-ups and obtain initial user feedback prior to advanced prototyping. Iteratively improve designs with user feedback.
	Continuously testing prototype	Test quickly and iterate on the design of the prototype by collecting both positive and negative user feedback. Document all stages of user feedback in the resulting VR1 study paper.

The Design Process of Virtual Reality Treatments Should Promote Empathy

The committee believes that the more attuned a development team is to the specific perspective and needs of patients, the more likely they are to design meaningful VR treatments. Promoting empathy toward the design process involves carefully listening to and elucidating patients’ social environment, needs, fears, desires, habits, hopes, aspirations, and expectations. The committee recommends initiating the design process with an inspiration step, or exercise focused on culling patients’ voice and understanding their needs, struggles, and experiences. Table 1 describes best practices for sparking inspiration within the framework of empathy. Different patient profiles and scenarios should be included in this first step. Many techniques can be used to develop empathy and inspiration of the design team. These include qualitative assessments, observations, spending time with users, and conducting interviews and user experiments. In addition, a patient journey map can be used to illustrate the interpretation of a story from a patient’s perspective. The working group also recommends seeking input from relevant nonpatient end-users, including health care providers who may prescribe the VR treatment or interact with patient users.

The Design Process of Virtual Reality Treatments Should Promote Team Collaboration

The committee believes that team collaboration is fundamental for collectively designing a VR treatment and synthesizing data collected during the inspiration step. Brainstorming helps generate ideas from the initial corpus of data and findings. Table 1 describes best practices for ideation within the framework of team collaboration. The process of ideation allows team members to think expansively and divergently. As a range of ideas is generated, some ideas will be extreme or ambitious, whereas others will be achievable. Depending on the time and the available budget, the team decides what ideas should be prototyped further.

The Design Process of Virtual Reality Treatments Should Promote Continuous User Feedback

An effective VR treatment should be developed through continuous user feedback and iterative prototyping, thereby enabling the team to rapidly test their ideas during real-time assessment from end-users. Table 1 describes best practices for VR treatment prototyping within the framework of user feedback. Prototypes should be refined with continuous testing by patient end-users, and failures are viewed as a way to learn and improve the prototype to better meet users’ needs. Hence, the number of defects will tend to be lower and less costly in the future. To help facilitate the learning process for patients, it is recommended, when feasible, that the research team use a “mirroring” program [65] to allow the research staff to see what the patient is viewing through the VR headset and help them learn the user interface.

Briefly, the committee believes that the VR1 treatment design process should start with end-users. VR-CORE recommends specifying who the real users are and what they say, see, feel, and do. Hence, implementation of a patient-design approach is an important way to place users at the center of the VR design process. For researchers who are developing an open-source VR intervention that they would like to share with the academic community for collaborative V1 development process, the use of a software-development platform such as GitHub.com [66] and citation of the latest version of the program within the methods section of VR1 research papers are recommended. The committee also recommends use of the Integrate, Design, Assess, and Share checklist developed by Mummah and colleagues [60] as a supplemental, structured guide for conducting a VR1 study.

VR2 Trials

Once the research team has developed a VR treatment in partnership with end-users, the resulting product should undergo initial assessment in the target patient population within a representative clinical setting, herein termed a VR2 trial. Modeled after the work of Mosadeghi and colleagues [67], the purpose of VR2 trials is to conduct early testing with a focus on acceptability, feasibility, tolerability, and initial clinical efficacy prior to initiating a more definitive VR3 clinical trial. Although developers may opt to bypass a VR2 trial in lieu of a VR3 trial, there is a risk of subjecting an incompletely tested intervention to a larger and costlier RCT, and best practices in digital intervention development suggest an intermediary stage between initial VR design and definitive testing [60]. The following sections describe the features of a VR2 trial.

Clinical Setting

In contrast to a VR1 study, which is focused on collaborative content development in a design environment, the VR2 trial evaluates what happens when the VR treatment is placed in the hands of target patients within the intended clinical setting. For example, a VR treatment focused on management of inpatient pain should be tested in an inpatient environment. A VR treatment targeting outpatient stroke rehabilitation should be evaluated in locations where patients receive rehabilitation, such as in a physical therapy center or, if intended, at home. In short, a comprehensive VR2 trial evaluates the VR treatment in the natural setting(s) where the product is intended to be used. Table 2 summarizes the best practices for VR2.

Acceptability

In the context of a VR2 trial, acceptability refers to a patient’s willingness to use the VR treatment. Previous research on therapeutic VR reveals a drop off in the relation between patient eligibility to receive VR and patient willingness to try VR [67]. The disconnect emphasizes that many patients are uninterested in using novel health technologies such as VR, particularly when hospitalized or under duress. Among those who are eligible for a VR trial, some choose not to participate for a wide variety of reasons. Patients may express varying degrees of skepticism, fear, vulnerability, and concern regarding psychological consequences or simply not want to be bothered by the equipment [67]. In a VR2 trial, investigators collect data regarding patient willingness to try the VR treatment, including reasons why they did or did not find the intervention to be acceptable for use. Researchers should collect and report acceptability data using techniques such as focus groups, cognitive interviews, or structured questionnaires.

Feasibility

In the context of a VR2 trial, feasibility is the degree to which the VR treatment can be successfully integrated within the flow of usual care. The committee noted that even the best designed VR treatments can face implementation challenges when applied on the front lines of health care delivery [67]. It is wise for developers to understand potential barriers early and often, identify workarounds and solutions to these barriers, and only then consider testing their interventions in VR3 RCT trials. For example, patients and providers often seek information regarding the frequency and “dosing” of a VR treatment; these details could be manually collected in the context of a VR2 trial. Similarly, treatments deployed in a clinical environment may be unfamiliar to doctors, nurses, and other health care providers, giving researchers an opportunity to study the interaction among staff and proactively identify areas of confusion or misuse. The committee recommends including a table that enumerates patient, provider, technical, and operational barriers to use; identifies root causes; and offers solutions to enhance effectiveness in future clinical applications.

Tolerability

The VR2 trial offers an early opportunity to evaluate patient tolerability of the VR treatment, including both hardware and software components. Researchers should measure and report the prevalence of patient-reported physical (eg, vertigo, nausea, and “cybersickness”) and emotional (eg, fear and anxiety) adverse effects of the VR treatment, along with any discomfort or inconvenience related to the VR equipment (eg, ill-fitting headset, facial or nasal pain, inability to explore the three-dimensional environment fully due to limited mobility).

Cybersickness (or VR sickness) is a unique side effect of VR. There a several different terms used interchangeably within the literature, such as simulator sickness or “sim sickness,” although some believe they are different types of motion sickness [68]. When the vestibular system and oculomotor system notice a discrepancy between reality and the virtual environment, one or more of following symptoms ensue: eyestrain, nausea, fatigue, headache, blurred vision, and postural instability [69]. The specific mechanism of cybersickness is still unknown.

Table 2

Summary of best practice recommendations for VR2 trials.

Trial attribute	Best practice
Patient population	Study a representative population for whom the VR^a treatment is intended. Recruit a large enough sample to represent the breadth and depth of target patients and provide statistically stable estimates in descriptive analytics.
Clinical setting	Select a clinical setting that represents the intended environment for the VR treatment to be used (eg, inpatient vs outpatient, clinic vs home based)
Assessment of acceptability	Collect data regarding patient willingness to try the VR treatment, including reasons why they did, or did not, find the intervention to be acceptable for use. Researchers should collect and report acceptability data using techniques such as focus groups, cognitive interviews, or structured questionnaires.
Assessment of feasibility	Conduct patient and provider interviews to identify potential barriers and facilitators to using the VR treatment in the intended clinical environment. Collect information regarding the optimal frequency and “dosing” of a VR treatment; consider manualizing these details, where possible. Study interactions among staff and proactively identify areas of confusion or misuse. Consider including a table that enumerates patient, provider, technical, and operational barriers to use; identifies root causes; and offers solutions to enhance effectiveness in future clinical applications.
Assessment of tolerability	Measure and report the prevalence of patient-reported physical and emotional adverse effects of the VR treatment, along with any discomfort or inconvenience related to the VR equipment.
Assessment of initial clinical efficacy	Identify and justify selection of a clinically relevant and validated PRO^b to evaluate the evidence of efficacy. Measure the PRO before and after receipt of the VR treatment; consider comparing results against nonrandomized concurrent or retrospective control groups, where available.

^aVR: virtual reality.

^bPRO: patient-reported outcome.

Recommendations for developers already exist [70,71]: appropriately accelerate within the program [71,72], anticipate changes in direction [73], affect changes in the field of view [73], establish realistic virtual avatar movements, reduce drops in the frame rate below 60 fps [71], blur the display with movement [74], and provide other solutions at the level of program design.

There are also several strategies for medical staff and researchers including habituation [75], assessment of the risk of side effects before the intervention [76], use of oculomotor exercises before the intervention [77], and diaphragmatic breathing during the intervention [78]. One of the most useful strategies is to limit the total duration of each treatment session, particularly early in the process [70].

The VR-CORE recommends assessing for side effects at every phase (VR1, VR2, and VR3). Regarding assessment scales, the Simulator Sickness Questionnaire is the most commonly used scale in the literature [70,72,75,76].

Initial Clinical Efficacy

Although the VR2 trial is not designed to definitively test whether a VR treatment is efficacious or effective, it offers an early opportunity to measure efficacy within the context of a small clinical trial. There is no requirement in a VR2 trial to include a control group, although uncontrolled case series carry a higher risk of bias than controlled studies; even studies with nonrandomized concurrent controls, “wait list” controls, or retrospective controls may reduce the risk of bias as compared to an uncontrolled series.

Regardless of the inclusion of a control group, investigators should identify a clinically relevant and validated patient- reported outcome (PRO) to evaluate the evidence of efficacy. For example, a study evaluating pain might include a standard 11-point numeric rating scale [79] before and after exposure to the VR treatment. A study evaluating stroke rehabilitation might measure physical function with the National Institutes of Health Patient Reported Outcomes Measurement Information System [80]. Selection of the most appropriate PRO is at the discretion of the research team, but should be carefully justified and capture the most salient features of patient-reported health that might improve with the VR treatment.

VR3 Trials

The most definitive clinical validation of a VR treatment is the VR3 trial, which is a prospective, adequately powered, methodologically rigorous RCT evaluating clinical outcomes and safety in target patients receiving the VR treatment as compared to a control condition. Although the therapeutic mechanism of action may be studied as a secondary goal in a VR3 trial (eg, through neuroimaging, blood biomarkers, and physiologic testing), the principal goal is to evaluate the treatment’s impact on a clinically meaningful patient outcome rather than surrogate markers.

Although the committee acknowledged understandable costs and resource barriers involved in conducting VR3 trials, there was broad agreement that RCTs are of equal scientific importance in therapeutic VR as any other form of treatment and should be prioritized whenever possible. Multicenter collaborations may facilitate VR3 trials by combining patients and resources through shared protocols. The features of a VR3 trials are described below and summarized in Table 3.

Standardization of Intervention and Patient Population

Having been developed in a VR1 study and initially tested in a VR2 trial, the study intervention should be clearly described in preparation for a VR3 trial. Researchers should provide details regarding the equipment used; visualizations employed (with representative screenshots or videos); and frequency, duration, and timing of use. Optimally, the intervention should be manualized, and at the very least, enough details should be provided to allow other investigators to repeat the trial, if desired. The Template for Intervention Description and Replication checklist provides a useful framework for describing study interventions [81] and should be applied to VR treatments. The target patient population should be clearly described, including explicit inclusion and exclusion criteria employed. Certain exclusion criteria may be standardized among VR trials, such as a history of significant motion sickness, active nausea, and vomiting or epilepsy.

Selection of Control Condition

The committee acknowledged that there is no perfect or standardized control condition for all VR treatment trials; the optimal control depends on the patient population, proposed mechanism of action of the intervention, and clinical setting, among other considerations. Selection of the control is at the discretion of the research team but should be justified and explained. The committee described a hierarchy of control conditions, ranging from “usual care” without any active intervention to passive visualizations on a two-dimensional screen and nonimmersive visualizations within a headset, immersive but passive experiences within a headset, and immersive and active experiences within a headset. Selection of the optimal control may be guided by considering the hypothesized target of engagement and the proposed mechanism of action.

Randomization

Randomization should be described and ideally achieved using an appropriate computer program (eg, MS Excel Random Number Generator) [81] or random number tables without involvement of the investigators who enrolled the patients.

Blinding and Concealment of Allocation

The committee acknowledged that blinding and concealment can be challenging, but they identified techniques to incorporate these RCT principles within the constraints of VR research. For example, Spiegel and colleagues (2017) achieved concealment of allocation in an RCT comparing a library of VR content to a “health and wellness” television channel in hospitalized patients experiencing pain [83]. At the time of consent, the researchers explained to patients that the study compared “two different audiovisual experiences designed to reduce pain,” but did not describe the details of the competing interventions. Patients randomized to the television intervention did not know that VR was the other condition and vice versa. This approach may reduce the “novelty effect” of receiving VR rather than a familiar experience like television. Equipoise may also be achieved by exposing patients in both arms to headsets, but varying the content viewed within the headset (eg, immersive vs nonimmersive, active vs passive). At a minimum, study analysts should be blinded to patient group allocation, allowing for unbiased evaluation of the data without the knowledge of the study group. Patients should be asked not to reveal details of the program they experienced to decrease the chance of unblinding the study analysts. The measurement of perceived group assignment at the end of the study can help assess the success of blinding within the study. This should be done at the discretion of the research team.

Endpoints

Like the VR2 trial, VR3 trials must prespecify a clinically relevant and validated PRO as the primary endpoint. The study must be appropriately powered to demonstrate a minimally clinically important difference (MCID) [84] in that endpoint between the VR treatment and control arms. The psychometrics of PRO measurement are beyond the scope of this document, but existing references may assist investigators in protocol development [84,85]. Secondary endpoints may include a variety of clinical, imaging, biometric, and physiologic surrogate markers, as deemed appropriate by the study team. Like VR2 trials, potential adverse events must be prospectively measured and reported.

Study Duration

VR3 studies should monitor patients for a sufficient period to determine whether the VR treatment meaningfully impacts clinically important outcomes. One-time, short-term evaluations may be insufficient to evaluate the true clinical value of an intervention. Follow-up over several days may be appropriate if the study only focuses on hospital stay, but measurement over weeks, or even months, may be necessary to assess the impact on long-term clinical benefits.

Presentation and Analysis of Results

VR-CORE recommends that the primary outcome be reported as the before and after difference in difference between study arms, with accompanying 95% CIs. For example, the change in the mean PRO score before and after the VR intervention should be compared against the change in the mean PRO score before and after the control intervention. In addition, the panel recommends predefining a binary response criterion, guided by the MCID of the primary endpoint. The proportion achieving the MCID should be reported and compared between groups, and the resulting number needed to treat should be calculated.

Table 3

Summary of best practice recommendations for VR3 Trials.

Trial attribute	Best practices
Patient population	Study a representative population for whom the VR^a treatment is intended. The target patient population should be clearly described, including explicit inclusion and exclusion criteria employed.
Clinical setting	Select a clinical setting that represents the intended environment for the VR treatment to be used (eg, inpatient vs outpatient, clinic vs home based).
Standardizing intervention	Provide details regarding the equipment used; visualizations employed; and frequency, duration, and timing of use for VR treatment. Consider following the TIDIER^b checklist [81] as a useful framework for describing features of the VR treatment.
Selecting control condition	Select and justify the control condition(s) by considering the hypothesized target of engagement and the proposed mechanism of action.
Randomization	Randomization should be achieved using an appropriate computer program (eg, MS Excel Random Number Generator) [82] or random number tables without involvement of the investigators who enrolled the patients.
Blinding and concealment of allocation	Describe efforts to conceal allocation of the study intervention to the participants. Describe efforts to blind patient, providers, and analysts, wherever possible. Measure perceived group assignment to assess success of blinding.
Endpoints	Prespecify a clinically relevant and validated PRO^c as the primary endpoint. The psychometric properties of available PRO measures may need to be modified in the context of immersive therapy and then revalidated as needed. Trials must be appropriately powered to demonstrate an MCID^d [83] in the primary endpoint between the VR treatment and control arms. Secondary endpoints may include a variety of clinical, imaging, biometric, and physiologic surrogate markers, as deemed appropriate by the study team. Potential adverse events must be prospectively measured and reported.
Study duration	Select and justify the follow-up period that is sufficient to determine whether the VR treatment meaningfully impacts clinically important outcomes.
Presentation and analysis of results	Report the before and after difference in difference in the primary outcome measure between study arms, with accompanying 95% CIs. Predefine a binary response criterion, guided by the MCID of the primary endpoint. The proportion achieving the MCID should be reported and compared between groups, and the resulting number needed to treat should be calculated. Use intention-to-treat analysis for primary outcome assessment. Per-protocol analysis may be reported if prespecified, as relevant. To perform a multivariable analysis, it is optimal to have at least 10 (preferably, 20) observations for each independent variable included in the multivariable model.
Reporting the trial	Trial must be registered on a publicly accessible registry (eg, clinicaltrials.gov). All completed trials should be published, whether positive or negative. The CONSORT^e guidelines provide the framework for reporting RCTs [86] and should be followed in VR3 trials. Include a CONSORT diagram demonstrating the flow of patients through each stage of the trial, including the number screened to the number randomized into each study group and the number analyzed.

^aVR: virtual reality.

^bTIDIER: Template for Intervention Description and Replication.

^cPRO: patient-reported outcome.

^dMCID: minimally clinically important difference.

^eCONSORT: Consolidated Standards for Reporting Trials

The primary analyses should use the intention-to-treat population, including all patients randomized regardless of follow-up or receipt of study interventions. However, per-protocol analysis may be appropriate in certain situation, such as if patients refuse the VR treatment after randomization; in this instance, reporting the rate of refusal would be important, but investigators might also seek to compare therapeutic responses only among those receiving the intervention.

Multivariable analysis may be useful in adjusting for prespecified confounding factors (especially if not equally distributed in the study groups) and exploring independent predictors of outcomes. To perform a multivariable analysis, it is optimal to have at least 10 (preferably, 20) observations for each independent variable included in the multivariable model.

Trial Reporting

VR3 trials must be registered in a publicly accessible registry (eg, such as ClinicalTrials.gov). All completed trials should be published, regardless of whether they are positive or negative. The Consolidated Standards for Reporting Trials (CONSORT) guidelines provide the framework for reporting RCTs [86] and should be followed in VR3 trials. VR3 trials must include a CONSORT diagram to demonstrate the flow of patients through each stage of the trial, including the number screened to the number randomized into each study group and the number analyzed.

Conclusions

To improve methodological quality in the therapeutic VR literature, the VR-CORE international working group presents a three-part framework for best practices in developing and testing VR treatments. This framework may be used to facilitate development of high-quality, effective, and safe VR treatments that meaningfully improve patient outcomes. Patients, providers, payers, and regulators should consider this framework when assessing the validity of VR treatments.

Multimedia Appendix 1

Excerpted transcripts of Virtual Reality Committee of Outcomes Research Experts responses to selected discussion topics. Key themes and phraseology included in the manuscript are highlighted. Note that not all committee members responded to all questions.

Abbreviations

CONSORT

Consolidated Standards for Reporting Trials

HMD

head-mounted display

MCID

minimally clinically important difference

PRO

patient-reported outcome

RCT

randomized controlled trial

TIDIER

Template for Intervention Description and Replication

virtual reality

VR-CORE

Virtual Reality Clinical Outcomes Research Experts

BS is supported by the Marc and Sheri Rapaport Fund for Digital Health Sciences and Precision Health. In addition to the authorship team, the following committee members actively participated in the online discussion forums and contributed to this paper: Tom Caruso, MD, MEd - Stanford University School of Medicine, Stanford, CA; Ali Fardinpour, PhD - Wise Realities, Perth, Australia; Diane Gromala, PhD - Simon Fraser University, Surrey, BC; Rafael Grossmann, MD - Eastern Maine Healthcare System, Bangor, ME; Kate Hardy, Clin PsychD - Stanford University School of Medicine, Stanford, CA; Constance Johnson, PhD - The University of Texas Health Science Center, Houston, TX; Ted Jones, PhD - Pain Consultants of East Tennessee, Knoxville, TN; Kate Laver, PhD - Flinders University, Adelaide, Australia; Sheila Parinas, BSN, RN - University of Illinois, Champaign, IL; Les Posen, FAPS, Clin Psych - Flightwise, Melbourne, Australia; Barbara Rothbaum, PhD, ABPP - Emory School of Medicine, Atlanta, GA; David Thomas, PhD - National Institute of Drug Abuse (NIDA), Bethesda, MD; Stefan Van Rooijen, MD PhD - Maastricht University, Maastricht, Netherlands; Herve Rosay, MD - Centre Leon Berard, Lyon, France; Earl Scott, MD - HCA Midwest Physician, Kansas City, MO; Andrea Stevenson Won, PhD - Cornell University, Ithaca, NY.

BS was the Principal Investigator of a 2016 Virtual Reality (VR) research grant (#CSR211835), administered by his academic institution, from AppliedVR (Los Angeles, California). He is currently the Principal Investigator of a VR research grant (#CSR212943), administered by his academic institution, from Traveler’s Insurance (New York City, NY) and Samsung Electronics (Suwon, South Korea). AR’s research relating to this paper has been funded by the National Institutes of Health (NIH), National Science Foundation, US Army Research Office, Telemedicine and Advanced Technology Research Center, US Army Medical Research Acquisition Activity, Department of Veterans Affairs, and Kesseler Foundation. All other authors (BB, CK, XL, SC, ID, and KB) have no conflicts of interest. Regarding the other Virtual Reality Clinical Outcomes Research Experts members, Dr Rothbaum owns equity in Virtual Better, Inc, which is developing products related to virtual reality research related to this paper. The terms of this arrangement have been approved by Emory University in accordance with its conflict of interest policies. Dr Johnson receives funding through the NIH to study virtual environments. Some of Dr van Rooijen’s VR research has been funded by Phillips, Inc (Amsterdam, Netherlands). All other members of the committee (Tom Caruso, Ali Fardinpour, Diane Gromala, Rafael Grossmann, Kate Hardy, Ted Jones, Kate Laver, Sheila Parinas, Les Posen, David Thomas, Herve Rosay, Earl Scott, and Andrea Stevenson Won) have no conflicts of interest.

Dascal

Reid

IsHak

Spiegel

Recacho

Rosen

Danovitch

Virtual Reality and Medical Inpatients: A Systematic Review of Randomized, Controlled Trials

Innov Clin Neurosci 2017 14 1-2 14 21

28386517

PMC5373791

Mahrer

Gold

The use of virtual reality for pain control: a review

Curr Pain Headache Rep 2009 04 13 2 100 9

19272275

Valmaggia

Latif

Kempton

Rus-Calafell

Virtual reality in the psychological treatment for mental health problems: An systematic review of recent evidence

Psychiatry Res 2016 02 28 236 189 95

10.1016/j.psychres.2016.01.015

26795129

S0165-1781(16)30025-7

Wismeijer

AAJ

Vingerhoets

AJJM

The use of virtual reality and audiovisual eyeglass systems as adjunct analgesic techniques: a review of the literature

Ann Behav Med 2005 12 30 3 268 78

10.1207/s15324796abm3003_11

16336078

Lohse

Hilderman

CGE

Cheung

Tatla

Van der Loos

HFM

Virtual reality therapy for adults post-stroke: a systematic review and meta-analysis exploring virtual environments and commercial games in therapy

PLoS One 2014 9 3 e93318

10.1371/journal.pone.0093318

24681826

PONE-D-14-02313

PMC3969329

Botella

Serrano

Baños

Garcia-Palacios

Virtual reality exposure-based therapy for the treatment of post-traumatic stress disorder: a review of its efficacy, the adequacy of the treatment protocol, and its acceptability

Neuropsychiatr Dis Treat 2015 11 2533 45

10.2147/NDT.S89542

26491332

ndt-11-2533

PMC4599639

Gonçalves

Pedrozo

Coutinho

ESF

Figueira

Ventura

Efficacy of virtual reality exposure therapy in the treatment of PTSD: a systematic review

PLoS One 2012 7 12 e48469

10.1371/journal.pone.0048469

23300515

PONE-D-12-18828

PMC3531396

de Carvalho

Dias

TRDS

Duchesne

Nardi

Appolinario

Virtual Reality as a Promising Strategy in the Assessment and Treatment of Bulimia Nervosa and Binge Eating Disorder: A Systematic Review

Behav Sci (Basel) 2017 07 09 7 3

10.3390/bs7030043

28698483

bs7030043

PMC5618051

Chirico

Lucidi

De Laurentiis

Milanese

Napoli

Giordano

Virtual Reality in Health System: Beyond Entertainment. A Mini-Review on the Efficacy of VR During Cancer Treatment

J Cell Physiol 2016 02 231 2 275 87

10.1002/jcp.25117

26238976

da Costa

RMEM

de Carvalho

LAV

The acceptance of virtual reality devices for cognitive rehabilitation: a report of positive results with schizophrenia

Comput Methods Programs Biomed 2004 03 73 3 173 82

10.1016/S0169-2607(03)00066-X

14980398

S016926070300066X

Macedo

Marques

Queirós

Virtual reality in assessment and treatment of schizophrenia: a systematic review

J Bras Psiquiatr 2015 03 64 1 70 81

10.1590/0047-2085000000059

Luque-Moreno

Ferragut-Garcías

Rodríguez-Blanco

Heredia-Rizo

Oliva-Pascual-Vaca

Kiper

Oliva-Pascual-Vaca

A Decade of Progress Using Virtual Reality for Poststroke Lower Extremity Rehabilitation: Systematic Review of the Intervention Methods

Biomed Res Int 2015 2015 342529

10.1155/2015/342529

26539480

PMC4619784

Malloy

Milling

The effectiveness of virtual reality distraction for pain reduction: a systematic review

Clin Psychol Rev 2010 12 30 8 1011 8

10.1016/j.cpr.2010.07.001

20691523

S0272-7358(10)00109-1

Parsons

Rizzo

Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis

J Behav Ther Exp Psychiatry 2008 09 39 3 250 61

10.1016/j.jbtep.2007.07.007

17720136

S0005-7916(07)00045-6

Powers

Emmelkamp

PMG

Virtual reality exposure therapy for anxiety disorders: A meta-analysis

J Anxiety Disord 2008 22 3 561 9

10.1016/j.janxdis.2007.04.006

17544252

S0887-6185(07)00103-X

Rodrigues-Baroni

Nascimento

Ada

Teixeira-Salmela

Walking training associated with virtual reality-based training increases walking speed of individuals with chronic stroke: systematic review with meta-analysis

Braz J Phys Ther 2014 18 6 502 12

10.1590/bjpt-rbf.2014.0062

25590442

S1413-35552014000600502

PMC4311594

de Rooij

IJM

van de Port

IGL

Meijer

Effect of Virtual Reality Training on Balance and Gait Ability in Patients With Stroke: Systematic Review and Meta-Analysis

Phys Ther 2016 12 96 12 1905 1918

10.2522/ptj.20160054

27174255

ptj.20160054

Triberti

Repetto

Riva

Psychological factors influencing the effectiveness of virtual reality-based analgesia: a systematic review

Cyberpsychol Behav Soc Netw 2014 06 17 6 335 45

10.1089/cyber.2014.0054

24892195

Rus-Calafell

Garety

Sason

Craig

TJK

Valmaggia

Virtual reality in the assessment and treatment of psychosis: a systematic review of its utility, acceptability and effectiveness

Psychol Med 2018 12 48 3 362 391

10.1017/S0033291717001945

28735593

S0033291717001945

Girard

Turcotte

Bouchard

Girard

Crushing virtual cigarettes reduces tobacco addiction and treatment discontinuation

Cyberpsychol Behav 2009 10 12 5 477 83

10.1089/cpb.2009.0118

19817561

Kuntze

Stoermer

Mager

Roessler

Mueller-Spahn

Bullinger

Immersive virtual environments in cue exposure

Cyberpsychol Behav 2001 08 4 4 497 501

10.1089/109493101750527051

11708729

Bordnick

Traylor

Carter

Graap

A Feasibility Study of Virtual Reality-Based Coping Skills Training for Nicotine Dependence

Res Soc Work Pract 2012 05 22 3 293 300

10.1177/1049731511426880

25484549

PMC4254682

Carter

Bordnick

Traylor

Day

Paris

Location and longing: the nicotine craving experience in virtual reality

Drug Alcohol Depend 2008 05 01 95 1-2 73 80

10.1016/j.drugalcdep.2007.12.010

18243586

S0376-8716(07)00510-8

PMC2346538

Thompson-Lake

DGY

Cooper

Mahoney

Bordnick

Salas

Kosten

Dani

De La Garza

Withdrawal Symptoms and Nicotine Dependence Severity Predict Virtual Reality Craving in Cigarette-Deprived Smokers

Nicotine Tob Res 2015 07 17 7 796 802

10.1093/ntr/ntu245

25475087

ntu245

PMC4542676

Bordnick

Traylor

Copp

Graap

Carter

Ferrer

Walton

Assessing reactivity to virtual reality alcohol based cues

Addict Behav 2008 06 33 6 743 56

10.1016/j.addbeh.2007.12.010

18282663

S0306-4603(07)00348-6

Maples-Keller

Price

Rauch

Gerardi

Rothbaum

Investigating Relationships Between PTSD Symptom Clusters Within Virtual Reality Exposure Therapy for OEF/OIF Veterans

Behav Ther 2017 12 48 2 147 155

10.1016/j.beth.2016.02.011

28270326

S0005-7894(16)30001-6

Maples-Keller

Bunnell

Kim

Rothbaum

The Use of Virtual Reality Technology in the Treatment of Anxiety and Other Psychiatric Disorders

Harv Rev Psychiatry 2017 25 3 103 113

10.1097/HRP.0000000000000138

28475502

00023727-201705000-00003

PMC5421394

Meyerbröker

Emmelkamp

PMG

Virtual reality exposure therapy in anxiety disorders: a systematic review of process-and-outcome studies

Depress Anxiety 2010 10 27 10 933 44

10.1002/da.20734

20734361

Emmelkamp

PMG

Krijn

Hulsbosch

de Vries

Schuemie

van der Mast

Virtual reality treatment versus exposure in vivo: a comparative evaluation in acrophobia

Behav Res Ther 2002 05 40 5 509 16

12038644

S0005-7967(01)00023-7

Bouchard

Dumoulin

Robillard

Guitard

Klinger

Forget

Loranger

Roucaut

Virtual reality compared with exposure in the treatment of social anxiety disorder: a three-arm randomised controlled trial

Br J Psychiatry 2017 12 210 4 276 283

10.1192/bjp.bp.116.184234

27979818

S0007125000281129

Morina

Ijntema

Meyerbröker

Emmelkamp

Can virtual reality exposure therapy gains be generalized to real-life? A meta-analysis of studies applying behavioral assessments

Behav Res Ther 2015 11 74 18 24

10.1016/j.brat.2015.08.010

26355646

S0005-7967(15)30033-4

Rizzo

Difede

Rothbaum

Johnston

McLay

Reger

Gahm

Parsons

Graap

Pair

VR PTSD exposure therapy results with active duty OIF/OEF combatants

Stud Health Technol Inform 2009 142 277 82

19377167

Maples-Keller

Yasinski

Manjin

Rothbaum

Virtual Reality-Enhanced Extinction of Phobias and Post-Traumatic Stress

Neurotherapeutics 2017 07 14 3 554 563

10.1007/s13311-017-0534-y

28512692

10.1007/s13311-017-0534-y

PMC5509629

Difede

Cukor

Jayasinghe

Patt

Jedel

Spielman

Giosan

Hoffman

Virtual reality exposure therapy for the treatment of posttraumatic stress disorder following September 11, 2001

J Clin Psychiatry 2007 11 68 11 1639 47

18052556

Freeman

Studying and treating schizophrenia using virtual reality: a new paradigm

Schizophr Bull 2008 07 34 4 605 10

10.1093/schbul/sbn020

18375568

sbn020

PMC2486455

Kim

Park

Jang

Kim

Characteristics of social perception assessed in schizophrenia using virtual reality

Cyberpsychol Behav 2007 04 10 2 215 9

10.1089/cpb.2006.9966

17474838

Sorkin

Weinshall

Modai

Peled

Improving the accuracy of the diagnosis of schizophrenia by means of virtual reality

Am J Psychiatry 2006 03 163 3 512 20

10.1176/appi.ajp.163.3.512

16513875

163/3/512

Veling

Pot-Kolder

Counotte

van Os

van der Gaag

Environmental Social Stress, Paranoia and Psychosis Liability: A Virtual Reality Study

Schizophr Bull 2016 12 42 6 1363 1371

10.1093/schbul/sbw031

27038469

sbw031

PMC5049523

Ferrer-Garcia

Gutiérrez-Maldonado

Riva

Virtual Reality Based Treatments in Eating Disorders and Obesity: A Review

J Contemp Psychother 2013 6 25 43 4 207 221

10.1007/s10879-013-9240-1

Cesa

Manzoni

Bacchetta

Castelnuovo

Conti

Gaggioli

Mantovani

Molinari

Cárdenas-López

Riva

Virtual reality for enhancing the cognitive behavioral treatment of obesity with binge eating disorder: randomized controlled study with one-year follow-up

J Med Internet Res 2013 06 12 15 6 e113

10.2196/jmir.2441

23759286

v15i6e113

PMC3713949

Riva

The key to unlocking the virtual body: virtual reality in the treatment of obesity and eating disorders

J Diabetes Sci Technol 2011 03 01 5 2 283 92

10.1177/193229681100500213

21527095

PMC3125918

Gutiérrez-Maldonado

Ferrer-García

Caqueo-Urízar

Moreno

Body image in eating disorders: the influence of exposure to virtual-reality environments

Cyberpsychol Behav Soc Netw 2010 10 13 5 521 31

10.1089/cyber.2009.0301

20950176

Ferrer-García

Gutiérrez-Maldonado

The use of virtual reality in the study, assessment, and treatment of body image in eating disorders and nonclinical samples: a review of the literature

Body Image 2012 01 9 1 1 11

10.1016/j.bodyim.2011.10.001

22119329

S1740-1445(11)00140-9

Marco

Perpiñá

Botella

Effectiveness of cognitive behavioral therapy supported by virtual reality in the treatment of body image in eating disorders: one year follow-up

Psychiatry Res 2013 10 30 209 3 619 25

10.1016/j.psychres.2013.02.023

23499231

S0165-1781(13)00095-4

Gutiérrez-Maldonado

Pla-Sanjuanelo

Ferrer-García

Cue-exposure software for the treatment of bulimia nervosa and binge eating disorder

Psicothema 2016 11 28 4 363 369

10.7334/psicothema2014.274

27776602

4336

Subramanian

Lourenço

Chilingaryan

Sveistrup

Levin

Arm motor recovery using a virtual reality intervention in chronic stroke: randomized control trial

Neurorehabil Neural Repair 2013 01 27 1 13 23

10.1177/1545968312449695

22785001

1545968312449695

Crosbie

Lennon

McGoldrick

McNeill

MDJ

McDonough

Virtual reality in the rehabilitation of the arm after hemiplegic stroke: a randomized controlled pilot study

Clin Rehabil 2012 09 26 9 798 806

10.1177/0269215511434575

22275463

0269215511434575

Bergeron

Lortie

Guitton

Use of Virtual Reality Tools for Vestibular Disorders Rehabilitation: A Comprehensive Analysis

Adv Med 2015 2015 916735

10.1155/2015/916735

26556560

PMC4590967

You

Jang

Kim

Kwon

Barrow

Hallett

Cortical reorganization induced by virtual reality therapy in a child with hemiparetic cerebral palsy

Dev Med Child Neurol 2005 09 47 9 628 35

16138671

Rothbaum

Hodges

Kooper

Opdyke

Williford

North

Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia

Am J Psychiatry 1995 04 152 4 626 8

10.1176/ajp.152.4.626

7694917

Hoffman

Richards

Bills

Van Oostrom

Magula

Seibel

Sharar

Using FMRI to study the neural correlates of virtual reality analgesia

CNS Spectr 2006 01 11 1 45 51

16400255

Hoffman

Richards

Coda

Bills

Blough

Richards

Sharar

Modulation of thermal pain-related brain activity with virtual reality: evidence from fMRI

Neuroreport 2004 06 07 15 8 1245 8

15167542

00001756-200406070-00005

Montaño

Chen

Gold

Virtual reality and pain management: current trends and future directions

Pain Manag 2011 03 1 2 147 157

10.2217/pmt.10.15

21779307

PMC3138477

McCaul

Malott

Distraction and coping with pain

Psychol Bull 1984 05 95 3 516 33

6399756

U.S Food & Drug Administration 2017-10-09 Step 3: Clinical Research https://www.fda.gov/ForPatients/Approvals/Drugs/ucm405622.htm

Laine

Spiegel

Rostom

Moayyedi

Kuipers

Bardou

Sung

Barkun

Methodology for randomized trials of patients with nonvariceal upper gastrointestinal bleeding: recommendations from an international consensus conference

Am J Gastroenterol 2010 03 105 3 540 50

10.1038/ajg.2009.702

20029415

ajg2009702

Grade Working Group 2006-04-07 GRADE http://www.gradeworkinggroup.org

Slack 2018-12-13

https://slack.com/

Ericsson

AKS

Simon

Protocol Analysis: Verbal Reports as Data 1993

Cambridge, Massachusetts

MIT Press

Mummah

Robinson

King

Gardner

Sutton

IDEAS (Integrate, Design, Assess, and Share): A Framework and Toolkit of Strategies for the Development of More Effective Digital Interventions to Change Health Behavior

J Med Internet Res 2016 12 16 18 12 e317

10.2196/jmir.5927

27986647

v18i12e317

PMC5203679

Field Guide to Human-Centered Design 2015

Palo Alto, California

IDEO.org/Design Kit

Brown

Design thinking

Harv Bus Rev 2008 06 86 6 84 92, 141

18605031

Jerald

The VR Book: Human-centered Design For Virtual Reality 2015

New York, NY

Morgan & Claypool Publishers

Carr

Halliday

King

Liedtka

Lockwood

The Influence of Design Thinking in Businessome Preliminary Observations

Design Management Review 2010 21 58 63

Oculus 2017

2018-07-09

Introducing Chromecast for Gear VR: Step into VR and Stream to Your Flatscreen Internet https://www.oculus.com/blog/introducing-chromecast-for-gear-vr-step-into-vr-and-stream-to-your-flatscreen/

Github 2017-07-05

https://github.com/

Mosadeghi

Reid

Martinez

Rosen

Spiegel

BMR

Feasibility of an Immersive Virtual Reality Intervention for Hospitalized Patients: An Observational Cohort Study

JMIR Ment Health 2016 06 27 3 2 e28

10.2196/mental.5801

27349654

v3i2e28

PMC4940605

Stanney

Kennedy

Drexler

Cybersickness is not simulator sickness

1997

In Proceedings of the Human Factors and Ergonomics Society annual meeting

September 22-26, 1997

Albuquerque, New Mexico

SAGE Publications

Bruck

Watters

Estimating cybersickness of simulated motion using the simulator sickness questionnaire (SSQ): A controlled study

2009

International Conference on Computer Graphics, Imaging and Visualization

Aug 11-14, 2009

Tianjin, China

486 488

10.1109/CGIV.2009.83

Porcino

Clua

Trevisan

Vasconcelos

Valente

Minimizing cyber sickness in head mounted display systems: design guidelines and applications

2017 04

IEEE 5th International Conference

April 2-4, 2017

Perth, Western Australia

1 6

10.1109/SeGAH.2017.7939283

Yao

Heath

Davies

Forsyth

Mitchell

Hoberman

Oculus 2014

2018-10-22

Oculus VR Best Practices Guide http://brianschrank.com/vrgames/resources/OculusBestPractices.pdf

Mazloumi Gavgani

Hodgson

Nalivaiko

Effects of visual flow direction on signs and symptoms of cybersickness

PLoS One 2017 12 8 e0182790

10.1371/journal.pone.0182790

28777827

PONE-D-17-21572

PMC5544223

Lin

Abi-Rached

Lahav

Virtual guiding avatar: An effective procedure to reduce simulator sickness in virtual environments

Proceedings of the SIGCHI conference on Human factors in computing systems 2004

ACM

April 24-29, 2004

Vienna, Austria

719 726

10.1145/985692.985783

Budhiraja

Miller

Modi

Forsyth

ARXIV 2017 10 06

2018-12-13

Rotation Blurring: Use of Artificial Blurring to Reduce Cybersickness in Virtual Reality First Person Shooters https://arxiv.org/pdf/1710.02599.pdf

Gavgani

Nesbitt

Blackmore

Nalivaiko

Profiling subjective symptoms and autonomic changes associated with cybersickness

Auton Neurosci 2017 03 203 41 50

10.1016/j.autneu.2016.12.004

28010995

S1566-0702(16)30109-6

Tyrrell

Sarig-Bahat

Williams

Treleaven

Simulator sickness in patients with neck pain and vestibular pathology during virtual reality tasks

Virtual Reality 2017 9 22 22 3 211 219

10.1007/s10055-017-0324-1

Park

Kim

Jang

Kim

Son

A study on cyber sickness reduction by oculo-motor exercise performed immediately prior to viewing virtual reality (VR) content on head mounted display (HMD)

JVE Journals 2017 14 260 264

10.21595/vp.2017.19170

Russell

MEB

Hoffman

Stromberg

Carlson

Use of controlled diaphragmatic breathing for the management of motion sickness in a virtual reality environment

Appl Psychophysiol Biofeedback 2014 12 39 3-4 269 77

10.1007/s10484-014-9265-6

25280524

Dworkin

Turk

Farrar

Haythornthwaite

Jensen

Katz

Kerns

Stucki

Allen

Bellamy

Carr

Chandler

Cowan

Dionne

Galer

Hertz

Jadad

Kramer

Manning

Martin

McCormick

McDermott

McGrath

Quessy

Rappaport

Robbins

Robinson

Rothman

Royal

Simon

Stauffer

Stein

Tollett

Wernicke

Witter

IMMPACT

Core outcome measures for chronic pain clinical trials: IMMPACT recommendations

Pain 2005 01 113 1-2 9 19

10.1016/j.pain.2004.09.012

15621359

S0304-3959(04)00440-3

Cella

Yount

Rothrock

Gershon

Cook

Reeve

Ader

Fries

Bruce

Rose

PROMIS Cooperative Group

The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years

Med Care 2007 05 45 5 Suppl 1 S3 S11

10.1097/01.mlr.0000258615.42478.55

17443116

00005650-200705001-00002

PMC2829758

Hoffmann

Glasziou

Boutron

Milne

Perera

Moher

Altman

Barbour

Macdonald

Johnston

Lamb

Dixon-Woods

McCulloch

Wyatt

Chan

Michie

Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide

BMJ 2014 03 07 348 g1687

10.1136/bmj.g1687

24609605

Saghaei

Mahmoud

An overview of randomization and minimization programs for randomized clinical trials

J Med Signals Sens 2011 01 1 1 55 61

22606659

PMC3317766

Tashjian

Mosadeghi

Howard

Lopez

Dupuy

Reid

Martinez

Ahmed

Dailey

Robbins

Rosen

Fuller

Danovitch

IsHak

Spiegel

Virtual Reality for Management of Pain in Hospitalized Patients: Results of a Controlled Trial

JMIR Ment Health 2017 03 29 4 1 e9

10.2196/mental.7387

28356241

v4i1e9

PMC5390112

Jaeschke

Singer

Guyatt

Measurement of health status. Ascertaining the minimal clinically important difference

Control Clin Trials 1989 12 10 4 407 15

2691207

0197-2456(89)90005-6

Norman

Sloan

Wyrwich

Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation

Med Care 2003 05 41 5 582 92

10.1097/01.MLR.0000062554.74615.4C

12719681

Curran

Bauer

The disaggregation of within-person and between-person effects in longitudinal models of change

Annu Rev Psychol 2011 62 583 619

10.1146/annurev.psych.093008.100356

19575624

PMC3059070

Begg

Cho

Eastwood

Horton

Moher

Olkin

Pitkin

Rennie

Schulz

Simel

Stroup

Improving the quality of reporting of randomized controlled trials. The CONSORT statement

JAMA 1996 08 28 276 8 637 9

8773637