Background: The success of electronic medical records (EMRs) is dependent on implementation features, such as usability and fit with clinical processes. The use of EMRs in mental health settings brings additional and specific challenges owing to the personal, detailed, narrative, and exploratory nature of the assessment, diagnosis, and treatment in this field. Understanding the determinants of successful EMR implementation is imperative to guide the future design, implementation, and investment of EMRs in the mental health field.
Objective: We intended to explore evidence on effective EMR implementation for mental health settings and provide recommendations to support the design, adoption, usability, and outcomes.
Methods: The scoping review combined two search strategies that focused on clinician-facing EMRs, one for primary studies in mental health settings and one for reviews of peer-reviewed literature in any health setting. Three databases (Medline, EMBASE, and PsycINFO) were searched from January 2010 to June 2020 using keywords to describe EMRs, settings, and impacts. The Proctor framework for implementation outcomes was used to guide data extraction and synthesis. Constructs in this framework include adoption, acceptability, appropriateness, feasibility, fidelity, cost, penetration, and sustainability. Quality assessment was conducted using a modified Hawker appraisal tool and the Joanna Briggs Institute Critical Appraisal Checklist for Systematic Reviews and Research Syntheses.
Results: This review included 23 studies, namely 12 primary studies in mental health settings and 11 reviews. Overall, the results suggested that adoption of EMRs was impacted by financial, technical, and organizational factors, as well as clinician perceptions of appropriateness and acceptability. EMRs were perceived as acceptable and appropriate by clinicians if the system did not interrupt workflow and improved documentation completeness and accuracy. Clinicians were more likely to value EMRs if they supported quality of care, were fit for purpose, did not interfere with the clinician-patient relationship, and were operated with readily available technical support. Evidence on the feasibility of the implemented EMRs was mixed; the primary studies and reviews found mixed impacts on documentation quality and time; one primary study found downward trends in adverse events, whereas a review found improvements in care quality. Five papers provided information on implementation outcomes such as cost and fidelity, and none reported on the penetration and sustainability of EMRs.
Conclusions: The body of evidence relating to EMR implementation in mental health settings is limited. Implementation of EMRs could benefit from methods used in general health settings such as co-designing the software and tailoring EMRs to clinical needs and workflows to improve usability and acceptance. Studies in mental health and general health settings rarely focused on long-term implementation outcomes such as penetration and sustainability. Future evaluations of EMRs in all settings should consider long-term impacts to address current knowledge gaps.
Information and information transfer are critical to the delivery of health care services, including in mental health settings . Modern health care increasingly relies on new information technology (IT) systems to store, retrieve, and transfer information to support decision-making for care and administrative processes [ ]. Among the health-related IT systems currently in use, electronic medical records (EMRs) are the most widely implemented across many settings [ ]. In their simplest form, EMRs are digital versions of case histories containing patient health–related information, but they can also support artificial intelligence capabilities, clinical decision-support systems, natural language processing, and so on [ ]. EMRs have the potential to improve adherence to clinical guidelines across all settings [ ], thereby reducing resource wastage, increasing care quality, and reducing patient harm. Examples include improved prescribing practices and medication safety through integrated electronic ordering systems [ ] and reductions in inappropriate laboratory testing because of integrated decision-support tools [ ]. Ultimately, EMRs are expected to contribute to creating safer and more effective health systems [ ].
Although several studies identifying the potential of EMRs have been published, evidence on their benefit to organizational, clinical, and patient outcomes after implementation continues to be mixed, with success appearing to be largely dependent on the design and fit with the local health care settings and workflows. For example, implementation of the same EMR system in two different university hospitals revealed that the time spent on documentation increased in one site but decreased in the other . Furthermore, high-profile, unintended consequences because of EMR implementation by-products have been reported in recent times. A notable example includes the implementation of a £200 million EMR system in a major UK teaching hospital, leading to reduced performance and demoralized staff [ ]. Poor usability of EMRs can impact quality of care and patient safety, as poor fit and design may cause fatigue, delayed case note entry, and adjacency errors [ ]. As research on the implementation of EMRs continues to emerge, there is a strong need to understand the processes, systems, contexts, and human factors that influence successful implementation [ ].
Although the adoption of EMRs has grown significantly in recent years, research that is specific to mental health settings or mental health clinicians has been minimal. Documentation in mental health settings brings unique challenges for the implementation of EMRs. Effective mental health documentation requires the recording of individualized, detailed, and narrative information, which is not easily reduced to checklists . Care is often long term and multidisciplinary, requiring staff of different disciplines to record and retrieve information over long periods or in different settings (eg, hospital and community). Hence, the implementation of EMRs in mental health settings may have specific negative impacts, either real or perceived, on patient-centered care, the ability to develop the patient-clinician rapport, and on clinician time. Understanding the available evidence on implementation determinants and outcomes of EMRs in mental health settings, as well as the implementation features that contribute to its success or failure, could aid the future investment, design, and implementation of EMRs in this field.
The aim of this scoping review of the peer-reviewed literature was to provide a synthesis of implementation studies relevant to EMRs in mental health settings and inform EMR mental health policy recommendations in New South Wales, Australia. To provide in-depth recommendations, the review also considered broader evidence from general health settings to reflect on EMR implementation lessons. The specific objectives of this scoping review were to (1) identify published studies pertaining to the implementation of EMRs in mental health settings and literature reviews in general health settings, (2) synthesize the specific implementation determinants and outcomes examined in these studies according to the Proctor framework for implementation outcomes , and (3) provide local policy recommendations for future design and implementation of EMRs in mental health settings based on the findings.
Our scoping review followed a predetermined (but unregistered) protocol that was developed in accordance with the PRISMA-ScR (Preferred Reporting Items of Systematic Review and Meta-Analyses Extension for Scoping Reviews) [, ] and followed methods used in published peer-reviewed scoping reviews [ ]. An exploratory search of 1 database over a 2-year period, conducted in consultation with a medical librarian and mental health experts on our team (GS and CT), confirmed that the studies on EMRs implemented in mental health settings were limited. Therefore, in our scoping review, we also conducted a review of reviews to capture implementation literature across EMRs in all health settings and not just mental health, given that the broad issues around the usability of EMRs in general health settings are potentially relevant. The results of both search strategies were analyzed; for synthesis, we used a combination of results from primary studies and review papers.
In our scoping review, we defined mental health professionals as psychiatrists, psychologists, nurses, and any other health professional involved in treating people with mental health disorders in health service settings, including allied health professionals. These settings could be mental health clinics, or general inpatient or outpatient clinics but needed to be in high-income countries. High-income countries were classified as category 1 countries by the Organisation for Economic Co-operation and Development (OECD) . This criterion was used to maintain relevance to the local policy setting context. Implementation determinants were defined as barriers and enablers that may prevent or facilitate improvements in practice [ ], as reported in the included studies. The Proctor framework provides a systematic taxonomy of implementation outcomes (ie, acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability), distinguishing these from service and patient outcomes [ ].
Our scoping review combined two systematic searches; the first captured published studies reporting primary data on the use and implementation of clinician-facing EMRs specifically in mental health settings (henceforth termed “primary studies”). The second search captured published reviews on the use of clinician-facing EMRs as implemented in all health settings irrespective of their relationship to mental health (henceforth termed “reviews”). The searches were conducted in three academic databases (MEDLINE via the PubMed Interface, EMBASE, and PsycINFO) and used the terms outlined in. Additionally, we manually searched the reference lists of the included studies (primary studies and reviews) for other relevant publications. All searches were limited to studies and reviews published between January 2010 and June 2020. The search strategies were devised by the review team with the assistance of an experienced medical librarian.
|Construct||Search terms for primary studies||Search terms for reviews|
|EMRa-related terms||“Electronic Health Records”[MeSH] OR Medical Records Systems, Computerized [MeSH] OR ((health record*b OR medical record* OR healthcare record* OR health care record* OR clinical record*) AND (digital OR electronic OR computerized OR computerized OR ambulatory)) [Title/Abstract]||“Electronic Health Records”[MeSH] OR Medical Records Systems, Computerized [MeSH] OR ((health record* OR medical record* OR healthcare record* OR health care record* OR clinical record*) AND (digital OR electronic OR computerized OR computerized OR ambulatory)) [Title/Abstract]|
|Health professional–related terms||“Psychiatry”[MeSH] OR “Psychiatric Nursing”[MeSH] OR (mental health OR psychiatric nurs* OR psychiatry OR psychiatrist OR psychology OR psychologist) [Title/Abstract]||“Psychiatry”[MeSH] OR “Psychiatric Nursing”[MeSH] OR “Physicians”[MeSH] OR “Nurses”[MeSH] OR “Health Personnel”[MeSH] OR (Physician OR nurse OR doctor OR psychiatrist OR psychologist OR health professional OR health personnel OR psychiatric nursing) [Title/Abstract]|
|Impact-related terms||(uptake OR adoption OR usability OR utility OR utilization OR utilization OR evaluate OR evaluation OR implementation OR acceptance OR acceptability) [Title/Abstract]||(uptake OR adoption OR usability OR utility OR utilization OR utilization OR evaluate OR evaluation OR implementation OR acceptance OR acceptability) [Title/Abstract]|
|Additional limiters||Published in English AND published between January 2010 and June 2020||(Systematic review or meta-analysis) AND published in English AND published between January 2010 and June 2020|
aEMR: electronic medical record.
bAsterisk indicates truncation.
Inclusion and Exclusion Criteria
In both searches, articles were included if they met the following inclusion criteria: investigated implemented clinician-facing EMRs; conducted in high-income countries (countries classified as category 1 by the OECD ); assessed and reported implementation outcomes and contextual determinants of implementation (ie, barriers and facilitators); and published between January 1, 2010, and June 30, 2020. The population and study type inclusion criteria differed between the searches. The review of primary studies included studies related to mental health clinicians, whereas the review of reviews included literature reviews of studies about any health professionals in any health setting.
In both searches, articles were excluded if the implemented EMRs were exclusively patient-facing ones, they did not report on implementation processes or outcomes, or they were not published in English. The complete list of the inclusion and exclusion criteria is available in.
Screening, Data Extraction, and Synthesis Procedures
Reference details, including abstracts, were downloaded into the reference management software EndNote X8 (Clarivate) ; duplicates were removed and the deduplicated list was exported to Rayyan QCRI [ ], a systematic reviews web app, for title and abstract screening. Five investigators (HLT, LT, LAE, AG, and IM) independently conducted the two-phase screening process: (1) title and abstract screening and (2) full-text screening. Two investigators (HLT and IM) cross-checked 50% of the records to ensure that article screening was consistent in accordance with accepted practices [ ]. Interrater reliability (Cohen kappa coefficient) in this cross-checking indicated strong agreement (>0.8) [ ]. A custom data extraction workbook in Excel (Microsoft Corporation) was developed and tested. Data were systematically extracted by six investigators (HLT, LL, LT, AG, LAE, and IM). Four investigators (LT, LAE, YZ, and IM) examined the data for consistency and cross-checked the extracted data against original articles.
Key information extracted included the study publication details (authors, date of publication, country of study, and number of studies in reviews), health settings, study methods (quantitative, qualitative, and mixed methods), design features of EMRs, and implementation barriers, enablers, and outcomes. To ensure consistency in our review, the Proctor framework of implementation outcomes presented inwas used as the guiding structure, with definitions tailored to suit the EMR implementation context [ ].
|Adoption||Uptake of the EMRa from the professionals, organizations, and settings|
|Acceptability||Clinician satisfaction with various aspects of the innovation (eg, content, complexity, comfort, delivery, and credibility)|
|Appropriateness||Perceived fit, relevance, compatibility, suitability, usefulness, and practicability defined by clinicians|
|Feasibility||Actual fit or usefulness, suitability for everyday use, and practicability assessed at the level of the health service provider/organization/setting|
|Fidelity||Program delivered as intended, adherence by clinicians, integrity, and quality of program delivery|
|Cost||Financial impact of technology implementation on the health provider or organization|
|Penetration||Spread or reach of the technology assessed at the organization or setting level|
|Sustainability||Maintenance or integration of a technology within a health service|
aEMR: electronic medical record.
Assessment of Evidence Quality
Primary studies were appraised for quality using a modified Hawker appraisal tool and scoring system [, ]. This tool was selected as it is designed to review evidence from a variety of methods [ ]. The Critical Appraisal Checklist for Systematic Reviews and Research Syntheses developed by the Joanna Briggs Institute was used to appraise the systematic review studies [ ]. Two investigators (HLT and IM) appraised 10% of the articles independently to ensure consistency. Quality assessment results were reported to reflect the quality of the studies and reviews included in our scoping review. We did not exclude studies based on quality assessment.
Data Analysis and Synthesis
The extracted data were analyzed for common features and summarized into tables. Implementation outcomes were grouped by outcomes (eg, satisfaction), and barriers and enablers were grouped by themes (eg, technical factors). Barriers, enablers, and implementation outcomes were also categorized using the Proctor framework (), while recognizing some degree of overlap among constructs as suggested by other publications [ , ]. Assignments to the constructs were based on the definitions applied by the review team ( ) rather than the assignment made by the authors of the included articles owing to inconsistencies in the manner of defining, measuring, and reporting implementation outcomes [ ]. Three investigators (YZ, LAE, and IM) reviewed the assignment of all the results, and any discrepancies were discussed among the three authors until a consensus was reached. Summary statistics (frequencies and proportions) were calculated for the final assignment.
Search Results and Study Selection
The search for primary studies yielded 1546 results relevant to mental health professionals or settings (Medline: 606; EMBASE: 620; PsycINFO: 320). Manually searching the article reference lists yielded 2 more papers. Among these, 271 duplicates were removed; after title/abstract screening, 1209 papers were excluded because they did not meet the inclusion and exclusion criteria. Furthermore, 68 studies underwent full-text review, and another 56 papers were excluded. We included 12 primary studies for data extraction and synthesis, as shown in.
The search for reviews yielded 484 results (Medline: 175; EMBASE: 297; PsycINFO: 12). We identified 4 additional papers by manually searching article reference lists. Then, 73 duplicates were removed, and after title and abstract screening, 377 papers were excluded. Another 27 were excluded after full-text review, and 11 were included for data extraction and synthesis (). A total of 23 studies were included for data extraction and synthesis from the 2 searches.
Half of the primary studies (6/12, 50%) were from the United States of America. The remaining primary studies were from Canada (2/12, 16.7%), the United Kingdom (2/12, 16.7%), France (1/12, 8.3%), and Sweden (1/12, 8.3%), as shown in. Most primary studies were conducted using quantitative methodologies (5/12, 42%), and fewer studies were conducted using qualitative (3/12, 25%) or mixed (4/12, 33%) methodologies. Each of the review studies included publications from several countries; however, in each review, at least more than 50% of the countries were OECD nations, as observed in . Among these 11 review studies, 1 (9%) focused on mental health settings [ ], and the remaining 10 (91%) involved general health settings.
|Boyer et al ||France||Psychiatric hospital||Hospital EMR including coded data, unstructured text, and scanned paper documents||115 health|
|Bruns et al ||United States||Mental health facilities||EMR with standardization of information, assessments, and diagnosis; facilitated a coordinated care plan, team communication, and routine reporting||34 wraparound care coordinators|
|Erlingsdóttir et al ||Sweden||Psychiatry services||Patient-accessible EMR||871 mental health|
professionals preimplementation; 699 postimplementation
|Golberstein et al ||United States||Primary care clinics||EMR prompting specific mental health questions and enabling e-consult ordering with psychiatry||Primary care providers (457 in the first wave; 499 in the second) from 45 clinics|
|Jetelina et al ||United States||Primary care||EMR with referral pathways, screening tools (point-and-click tools, drop-down menus, auto calculators, and auto population of some fields), and tracking and documentation of clinical and social information and goal setting||6 community care clinics with a mix of primary care, and psychology and social work|
|Madden et al ||United States||Medical practice||Not specified||Health insurance plan members with depression (5140), bipolar disorder (462), and a control group (43,582)|
|Martin et al ||Canada||Psychiatric hospital||Not specified||24 nurses|
|Reyes-Portillo et al ||United States||Child and youth psychiatry clinic||Alert in existing EMR that triggered a safety plan when a suicidal ideation, a plan, or an attempt was recorded||40 mental health clinicians|
|Riahi et al ||Canada||Mental health facility||EMR containing closed-loop medication administration, assessment and screening tools, care plan, details of restraint and seclusion, clinical practice guidelines, and infection control details||1300 facility staff|
|Ser et al ||United Kingdom||Mental health hospitals||Interoperable EMR||33 hospital staff|
|Skelton et al ||United Kingdom||Older adult psychiatric inpatient ward||Out-of-hours handover built into existing EMR||10 doctors|
|Stanhope et al ||United States||Community mental health clinics||Delivering person-centered care in the context of different EMRs||31 clinical supervisors and 52 direct care staff|
aEMR: electronic medical record.
|Study||Country (number of studies)||Setting||EMRa implemented||Included studies, n|
|Baumann et al ||United States (12), Australia (5), Germany (5), United Kingdom (1), Canada (1), Austria (1), Denmark (1), Greece (1), and France (1)||Academic, private, and community hospitals||Not specified||28|
|Boonstra et al ||United States (17), Canada (2), Norway (1), and Ireland (1)||General health settings||Not specified||22|
|Castillo et al ||United States (52), Canada (4), Australia (3), Germany (2), International group (1), Denmark (1), France (1), Sweden (1), Hong Kong (1), United Kingdom (1), and Norway (1)||General health settings||Not specified||68|
|Delardes et al ||United Kingdom (4), United States (3), Ireland (1), and Taiwan (1)||General health settings||Not specified||9|
|Gephart et al ||United States (4) and Sweden (1)||General health settings||Not specified||5|
|Goldstein et al ||United States (8), Austria (1), Brazil (1), Canada (1), and Switzerland (1)||General health settings||Not specified||12|
|Goldzweig et al ||United States (20), France (1), Canada (1), and Austria (1)||Academic medical centers||Classification of radiology ordering of EMR interventions into four categories: (1) display of information, (2) patients’ clinical information linked with recommendations, (3) soft stop if order contradicts recommendations, and (4) hard-stop software preventing inappropriate ordering||23|
|Lau et al ||United States (11), United Kingdom (10), the Netherlands (5), Canada (4), Australia (4), Norway (2), and New Zealand (2)||General health settings||Not specified||43|
|Meißner and Schnepp ||United States (4) and Australia (3)||Residential aged care facilities||Not specified||7|
|Nguyen et al ||United States (62), Denmark (5), England (5), Norway (4), Canada (3), Sweden (1), Australia (2), the Netherlands (2), Ireland (2), Israel (2), Austria (1), Cyprus (1), France (1), Serbia (1), Sweden (1), Japan (1), Korea (1), Kuwait (1), Cameroon (1), and Uganda (1)||General health settings||Not specified||98|
|Strudwick and Eyasu ||Germany (1), England (2), France (1), Finland (1), United States (1), and Sweden (1)||Mental health/ psychiatric clinic settings||Not specified||7|
aEMR: electronic medical record.
The primary studies scored highly on the modified Hawker appraisal tool , with an average score of 30.3 (SD 3.81) out of a possible 36. The reviews scored an average of 7.6 (SD 1.45) out of a possible score of 11 on the Joanna Briggs Institute Critical Appraisal Checklist for Systematic Reviews and Research Syntheses [ ] (see Table S1 of for details).
Features of Implemented EMRs
The features of the EMRs were described in 8 of the 12 (66.7%) primary studies in mental health settings. Features ranged from the simple electronic storage of personal and health information documentation , e-ordering of consultations [ ], and capability to enter free-text notes [ ] to features that aimed to improve care quality including embedded assessment tools [ , , ], and care coordination plans [ , , ]. Specific examples included implementing automated alerts to develop safety plans for children and youth with suicidal ideations [ ] and embedding an e-consultation pathway prompt linking primary health providers with a psychiatrist [ ]. Overall, the description of EMR features was limited among the included studies. Four studies did not report on specific features; instead, they simply described the EMRs as storage of clinical notes and test results to improve the accuracy and completeness of clinical information [ , , , ]. Only 1 of the 11 (9.1%) reviews provided a comprehensive description of EMRs among the included studies [ ] ( ).
Implementation Outcomes and Determinants
Adoption was reported in 10 of the 23 included studies (43.5%), namely 4 of the 12 primary studies (33.3%) [, , , ] and 6 of the 11 reviews (54.5%) [ , , , , , ], as shown in . Factors influencing the adoption of EMRs fell into three categories: organizational, technical, and financial.
First, in the primary studies, high adoption rates were attributed to organizational support and prioritization , strong leadership and buy-in, greater capacity and willingness to change, engagement of staff, and formal training [ ]. In contrast, poor leadership and buy-in, high staff turnover, and poor capacity or unwillingness to change, resulted in lower adoption rates [ ]. Two primary studies suggested that adoption was high without reflecting on the reasons for this [ , ]. Similarly, the reviews reported that organizational structure, readiness for change, participation of leaders and end users in planning and implementation, and support for end users impacted adoption [ , , ]. Specifically, adoption was facilitated by training [ ], larger facility sizes [ , ], clinical champions/leaders [ , ], and the removal of all paper-based notes [ ]. A lack of clear implementation plans was identified as an important limitation to adoption [ ].
In 6 of the 11 (54.5%) reviews, adoption was reported to be limited by the technological functions and design of EMRs such as perceived limited functionality [, , , ], interoperability [ , , , ], lack of technical support, limited clinician technical skills (real or perceived) [ , , , , ], insufficient hardware [ , , ], and system failures (software or hardware breakdowns, errors, and need for frequent rebooting) [ ]. None of the primary studies reflected on technological factors influencing adoption.
One review concluded that the start-up financial cost of EMRs were the second most common barrier to adoption, after technical issues . Three other reviews also cited high start-up costs as a barrier to adoption [ , , ]. None of the primary studies reported on cost factors influencing adoption.
Among the included studies, 2 out of the 12 primary studies (16.7%) [, ] and 8 out of the 11 reviews (72.7%) [ , , , - , , ] reported on clinician acceptance of (or satisfaction with) the implemented EMRs or aspects of EMRs ( ). Reviews reported that positive clinician attitudes were necessary for successful adoption of EMRs [ , , ]. However, clinician satisfaction with EMRs varied in primary studies and reviews.
One primary study found that mental health clinicians were neutral about the addition of an alert for a mental health safety plan in EMRs , whereas another found there were fewer complaints regarding the quality of clinical handover following the introduction of out-of-hours electronic handover systems in the EMRs [ ].
In reviews, poor clinician satisfaction was associated with perceptions that the new software was complex, it took time to learn and use, and that time could be used for patient care [, , , ]. There were also concerns regarding patient data privacy [ , , , , ]. Rigidly designed EMRs and exclusion of end users from design processes [ , , ], poor trust in the quality of EMR vendors [ ], the perception that the software demanded excessive detail [ ], and previous negative experiences or negative beliefs about the usefulness of EMRs were additional important barriers to acceptability [ , ]. Likewise, one review identified that high satisfaction was associated with the perceived reliability and usability of EMRs, and adequate support for end users [ ].
The perceived appropriateness of EMRs was reported in 7 of the 11 reviews (63.6%) and 8 of the 12 primary studies, as shown in. Clinicians often assessed EMRs as appropriate or inappropriate based on their perceived impact on the clinical workflow and productivity, quality of clinical documentation, quality of care, and patient-clinician relationships.
In the primary studies, mental health clinicians perceived EMRs as appropriate when access to documentation improved , the time needed to send reminders to patients decreased [ ], administration time decreased [ ], and time was saved on documenting follow-up appointments owing to prefilled data [ ]. On the other hand, mental health clinicians believed that EMRs lacked appropriateness when workflows were blocked or slowed [ ], and when clinicians needed to take additional time to design workarounds for EMRs that did not meet their needs [ , ]. Similarly, in the reviews, EMRs reportedly lacked appropriateness when documentation time increased [ ], at least temporarily [ ], whereas other reviews found EMRs to be appropriate when access to documentation improved [ ], and when there was minimal impact on documentation time [ ]. One review further identified that although EMRs saved documentation time, the standard forms were not always appropriate for documenting assessments, treatments, and goals for patients receiving mental health care [ ].
In the reviews, perceptions of improved documentation quality in terms of legibility, accuracy, completeness, and consistency were associated with clinicians’ views that EMRs were appropriate [, , ]. However, some clinicians in mental health settings [ , ] and in general health settings [ , ], believed that EMRs lacked appropriateness owing to the requirement of excessive or redundant information or when access to patient notes became a “watered-down” version of free-text clinical notes that lacked detail [ ].
Across the primary studies and reviews, EMRs were perceived as appropriate if they were also perceived to be effective in terms of improving patient care  through supported decision-making based on availability of up-to-date information [ , , , ], better team communication, and averted potential medication errors [ , ].
One review suggested that EMRs improved patient-clinician interactions owing to the accessibility of information to clinicians . Other reviews found EMRs that impacted decision-making processes and workflows left clinicians feeling devalued in their clinical role and were hence considered inappropriate [ , ]. No primary studies discussed the acceptability and impact of EMRs on the patient-clinician relationship.
Feasibility of the EMRs or EMR components was investigated in 5 out of the 12 primary studies (41.7%) [- , ] and 7 out of the 11 reviews (63.6%) [ , , , , , , ] ( ). Across all studies, the feasibility of implementation and use of EMRs by clinicians was assessed through proxies such as documentation outcomes (time taken and completeness), frequency of adverse events, quality of care, and face-to-face clinical time. These outcome measures differ from those used under adoption, acceptability, and appropriateness, as these include quantifiable impacts of the implemented EMRs or the actual fit for purpose rather than perceptions or opinions.
In the reviews, the measured impacts included improved documentation time, as mentioned in one review involving mental health settings , whereas others found no difference [ , ]. Documentation time was not quantified in any of the primary studies.
Completeness of documentation varied, with one review reporting increased completeness , whereas another one found no impact [ ]. In the primary studies, an alert system increased the number of completed mental health safety plans [ ] and reduced the amount of missing data [ ]. However, another primary study found that events (eg, emergency department and hospital visits or mental health diagnoses and related procedures) for mental health patients were less likely to be recorded in EMRs compared with other types of patients [ ].
The impact on patient outcomes was rarely reported. A primary study found that an electronic handover system was associated with a downward trend in adverse events, but this was not statistically significant . A review found that EMRs had no impact or had a small impact on adverse events such as hospital readmission [ ].
Impacts of EMRs on care quality in mental health settings were not reported. However, reviews reported that EMRs reduced the time from orders to procedures , decreased medication errors [ ], and improved appropriate ordering of radiographic tests, although they increased the number of missed tests [ ].
None of the primary studies assessed the cost-effectiveness of EMRs. However, 4 of the 11 reviews identified that cost was a barrier to adoption [, , ], and interoperability of EMRs could improve long-term costs [ ]; there was no evidence that costs decreased owing to improved administrative effectiveness [ ]. In addition, the ongoing costs of maintaining and upgrading EMRs were reported to be high and the return on investment uncertain [ ].
Fidelity, Penetration, and Sustainability
These domains were seldom addressed across all the 23 included studies. One primary study reported improved patient-centered care, which was one of the intended impacts (fidelity) of that specific EMR system , and a review reported that the rate of EMR usage across clinical settings was exceedingly slow [ ]. No study addressed the sustainability of the implemented EMRs.
In mental health settings, the adoption of EMRs is seemingly impacted by technical and organizational factors, as well as by clinician perceptions of appropriateness and acceptability. Clinicians perceived EMRs as acceptable and appropriate if they improved documentation completeness without interrupting workflow. Clinicians tend to value EMRs that support quality of care, are fit for purpose, have readily available technical support, and do not interfere with the clinician-patient relationship. Overall, the body of evidence specific to mental health was small. The implementation determinants and outcomes identified in general health settings aligned with and expanded on the mental health–specific findings. For example, the cost of implementation was identified as an additional barrier to adoption, apart from the technical and organizational factors identified in the mental health literature. However, evidence from general health settings did not consider the unique challenges of implementing EMRs in mental health settings. We have drawn on the evidence from general and mental health settings to make three recommendations for future implementation of EMRs in mental health settings.
Firstly, EMR implementation requires embedded long-term evaluation. In this review, we identified that approximately half of the studies focused on the early‐to-middle stage implementation outcomes (ie, adoption, acceptability, appropriateness, and feasibility) , whereas later‐stage implementation outcomes (ie, penetration and sustainability) [ ] and implementation costs were rarely evaluated. Fidelity was assessed in only one primary study in a mental health setting. This is in contrast with the implementation research outside of research on EMRs, where implementation fidelity has more often been assessed compared with other outcomes [ ]. This may be related to the nature of EMR technologies, which can be tailored and used flexibly to suit particular practices or service needs [ ]. Sustainability was also not reported in any of the included studies, a finding that is consistent with implementation research outside of the EMR field where the assessment of program sustainability has been identified as a neglected area [ , , ]. Limited research on the cost, fidelity, penetration, and sustainability of EMRs suggests limited evaluation and impact assessment, and the lack of long-term goal setting, particularly at the organizational level. We recommend that future implementation of EMRs in mental health settings must include continuous and embedded evaluation to explore long-term outcomes and impacts for health professionals and patients while identifying the determinants of cost, fidelity, penetration, and sustainability. Findings from thorough evaluations are needed to inform the future design, policies, and uptake of EMRs in mental health and other health settings.
Secondly, implementation of EMRs needs to adopt co-design principles and a human factors approach, including clinician participation in formative and summative usability testing prior to and during implementation [, ]. The successful uptake of EMRs is influenced by clinicians’ perceptions of appropriateness and acceptability. In mental health settings, this was negatively impacted when EMRs misaligned with established workflows. It was also affected by organizational factors such as high staff turnover, low staff buy-in, and low capacity or willingness to change shown by clinicians. Evidence from general health settings suggests that these determinants can be modified by specific facilitating features such as staff training, clinical champions, buy-in from clinicians and leaders, IT support, and, above all, good fit for purpose with minimal disruption to clinical workflows. However, EMRs are commonly designed by IT professionals; although well intentioned, the software is often insufficiently flexible to meet the needs of clinicians at the frontlines of care [ ]. Good fit with clinical workflows and local clinical contexts can be achieved through user-centered design processes and collaboration between clinicians and IT professionals [ , ]. Outside of this review and in general health settings, authors have recommended routine use of co-design principles and frameworks, formative evaluations in consultation with clinicians, and frameworks to assess the fit of off-the-shelf EMRs [ ]. In the mental health–specific literature covered in this review, co-designing was not analyzed. In future, to enhance the fit of EMRs to the unique and sensitive clinical work undertaken in mental health, we recommend that the development and implementation of EMRs include co-design and formative evaluations to achieve an optimal fit to support usability for clinicians and patient centeredness.
Lastly, the implementation of EMRs needs to be guided by theories and frameworks to successfully navigate behavior change, and interactions between people and technology. In an environment where sensitive issues are addressed and building rapport and trust with patients is especially important, simply “injecting technology” is unlikely to yield better care, experience for health professionals, or successful implementation. For example, in this review, organizational factors such as leadership and culture were the common determinants of EMR implementation [, , ]. It is also likely that external factors (eg, health system structure, funding, and governance) impact EMR implementation as seen in other areas of mental health [ ]; however, this was not addressed in the included studies. Successful implementation of EMRs requires structured methodology and careful planning, as changes in a social environment often require new skills and can have unpredictable impacts [ ]. We recommend that the development, planning, implementation, and evaluation of EMRs could be improved by applying appropriately structured guiding theories and frameworks (eg, behavior change theory [ ] or the normalization process theory [ ]).
Strengths and Limitations
Despite a rigorous search strategy, it is possible that some potentially relevant studies were missed owing to a wide range of terms used to describe EMRs (eg, health information systems and electronic health records). Nevertheless, our search strategy identified over 2000 potential publications across the two search strategies, reflecting its high level of comprehensiveness.
Further, the inconsistent use of implementation outcome terminologies across the literature and some degree of overlap among constructs as suggested by other publications [, ] made it challenging at times to classify outcomes into the Proctor categories. Although this may have resulted in the misclassification of some findings, they were applied as closely as possible to the Proctor definitions. A robust process where classifications were reviewed by three of the authors (YZ, LAE, and IM) and any discrepancies were discussed until a consensus was reached is a methodological strength supporting our synthesis.
Lastly, although we assessed study quality using validated tools, owing to the limited evidence available, it was not feasible to exclude studies or distinguish findings based on quality. Quality assessment results are described in Table S2 of.
The body of evidence about the implementation of EMRs in mental health settings is currently limited. Key enablers of the adoption of EMRs by clinicians in all health settings included clinician buy-in, staff training, IT support, and appropriate fit with the clinical context and workflows. Specific issues identified in mental health settings included limited suitability of the drop-down or checklist options and their impact on clinical workflows and patient-clinician interactions. Future implementation of EMRs could be facilitated through co-design with clinician end users, embedding routine implementation process evaluations, and including routine feedback from clinicians to facilitate adjustments and ensure usability and the best fit with the clinical context and person-centered care. Additionally, it is imperative that future implementations include embedded evaluations to assess long-term impacts on organizations, clinicians, and patients in mental health settings to inform future design, implementation, policy, and funding decisions. Lastly, the implementation of EMRs needs to recognize and address the interplay between the social factors and technical aspects of EMRs as a sociotechnical system to support successful uptake. Future research should consider the application of guiding social theories, implementation frameworks, and consistent use of terminology.
This publication is the result of an Evidence Check rapid review “Effectiveness of mental health electronic medical records” brokered by the Sax Institute and funded by the New South Wales Ministry of Health in November 2019. This research was also supported by the National Health and Medical Research Council (NHMRC) Partnership Centre for Health System Sustainability (PCHSS, grant 9100002) administered by the Australian Institute of Health Innovation, Macquarie University. Along with the NHMRC, the funding partners in the PCHSS are The Bupa Health Foundation; New South Wales Ministry of Health; Department of Health, Western Australia; and The University of Notre Dame, Australia.
This study was designed by YZ, LAE, CT, RCW, and GS. The search strategy was executed by HLT and IM. Data extraction and screening were conducted by YZ, LAE, HLT, LL, LT, and IM. Quality was assessed by HLT, LT, and IM. The extracted data were assessed for consistency by YZ, HLT, LAE, and IM. The first draft of the results section was written by IM, LAE, and HLT. The final draft was completed by YZ, LAE and IM with inputs from LL, RCW, LT, IM, CT, and GS.
Conflicts of Interest
Inclusion and exclusion criteria.DOCX File , 17 KB
Quality appraisal results of primary studies and reviews.DOCX File , 22 KB
- National Academy of Engineering (US), Institute of Medicine (US) Committee on Engineering and the Health Care System. Information and communications systems: the backbone of the health care delivery system. In: Building a Better Delivery System: A New Engineering/Health Care Partnership. Washington DC: National Academies Press; 2005.
- Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC: National Academy Press; 2001.
- Lakbala P, Dindarloo K. Physicians' perception and attitude toward electronic medical record. SpringerPlus 2014;3:63 [FREE Full text] [CrossRef] [Medline]
- Defining an electronic medical record. State of Victoria Department of Health. 2012. URL: http://www.health.vic.gov.au/divisions/fcs/cio.htm [accessed 2020-12-10]
- Campanella P, Lovato E, Marone C, Fallacara L, Mancuso A, Ricciardi W, et al. The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. Eur J Public Health 2016 Feb;26(1):60-64. [CrossRef] [Medline]
- Imai C, Hardie R, Franco GS, Sezgin G, Tepper K, McLeod A, et al. Harnessing the potential of electronic general practice pathology data in Australia: an examination of the quality use of pathology for type 2 diabetes patients. Int J Med Inform 2020 Sep;141:104189 [FREE Full text] [CrossRef] [Medline]
- Delvaux N, Piessens V, Burghgraeve TD, Mamouris P, Vaes B, Stichele RV, et al. Clinical decision support improves the appropriateness of laboratory test ordering in primary care without increasing diagnostic error: the ELMO cluster randomized trial. Implement Sci 2020 Nov;15(1):100 [FREE Full text] [CrossRef] [Medline]
- Joukes E, Abu-Hanna A, Cornet R, de Keizer N. Time spent on dedicated patient care and documentation tasks before and after the introduction of a structured and standardized electronic health record. Appl Clin Inform 2018 Dec;9(1):46-53. [CrossRef] [Medline]
- Shah S. Addenbrooke's hospital £200m IT system proves an epic fail. Computing. 2015. URL: https://www.computing.co.uk/news/2427100/addenbrookes-hospital-gbp200m-it-system-proves-an-epic-fail [accessed 2020-12-10]
- Bowman S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 2013;10:1c [FREE Full text] [Medline]
- Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci 2016;11(1):146 [FREE Full text] [CrossRef] [Medline]
- Triplett P. Psychiatry and the meaningful use of electronic health records. Perspect Biol Med 2013;56(3):407-421. [CrossRef] [Medline]
- Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 2018 Oct;169(7):467-473 [FREE Full text] [CrossRef] [Medline]
- Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth 2020 Oct;18(10):2119-2126. [CrossRef] [Medline]
- Kenyon LK, Krajenka VM, Lach K, VanBeek H, Williams B, Bower Russa M. Motivate-to-move: development of an intervention promoting parental adherence to early power mobility programs. Disabil Rehabil Assist Technol 2020 Nov:1-10. [CrossRef] [Medline]
- Country classification 2019 - as of 14 August 2019. Organisation for Economic Co-operation and Development. 2019. URL: https://www.oecd.org/trade/topics/export-credits/documents/oecd-export-credits-prevailing-list-of-countries-repayment-terms-and-aid-eligibility.pdf [accessed 2020-11-30]
- Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci 2015 Apr;10:53 [FREE Full text] [CrossRef] [Medline]
- Clarivate Analytics. Endnote X8. Philadelphia: Clarivate Analytics; 2016.
- Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Syst Rev 2016 Dec;5(1):210 [FREE Full text] [CrossRef] [Medline]
- Neely LC, Ganz JB, Davis JL, Boles MB, Hong ER, Ninci J, et al. Generalization and maintenance of functional living skills for individuals with autism spectrum disorder: a review and meta-analysis. Rev J Autism Dev Disord 2015 Nov;3(1):37-47. [CrossRef]
- McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22(3):276-282 [FREE Full text] [Medline]
- Hawker S, Payne S, Kerr C, Hardey M, Powell J. Appraising the evidence: reviewing disparate data systematically. Qual Health Res 2002 Nov;12(9):1284-1299. [CrossRef] [Medline]
- Lorenc T, Petticrew M, Whitehead M, Neary D, Clayton S, Wright K. Crime, fear of crime and mental health: synthesis of theory and systematic reviews of interventions and qualitative evidence. Public Health Reserarch. Southampton, UK: National Institute for Health Research; 2014. URL: https://www.journalslibrary.nihr.ac.uk/phr/phr02020/#/abstract [accessed 2021-08-27]
- Checklist for systematic reviews and research syntheses. Joanna Briggs Insitute. 2017. URL: https://joannabriggs.org/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Systematic_Reviews2017_0.pdf [accessed 2020-06-20]
- Long JC, Winata T, Debono D, Phan-Thien K, Zhu C, Taylor N. Process evaluation of a behaviour change approach to improving clinical practice for detecting hereditary cancer. BMC Health Serv Res 2019 Mar;19(1):180 [FREE Full text] [CrossRef] [Medline]
- Strudwick G, Eyasu T. Electronic health record use by nurses in mental health settings: a literature review. Arch Psychiatr Nurs 2015 Aug;29(4):238-241. [CrossRef] [Medline]
- Boyer L, Samuelian J, Fieschi M, Lancon C. Implementing electronic medical records in a psychiatric hospital: a qualitative study. Int J Psychiatry Clin Pract 2010 Sep;14(3):223-227. [CrossRef] [Medline]
- Bruns EJ, Hook AN, Parker EM, Esposito I, Sather A, Parigoris RM, et al. Impact of a web-based electronic health record on behavioral health service delivery for children and adolescents: randomized controlled trial. J Med Internet Res 2018 Jun;20(6):e10197 [FREE Full text] [CrossRef] [Medline]
- Erlingsdóttir G, Petersson L, Jonnergård K. A theoretical twist on the transparency of open notes: qualitative analysis of health care professionals' free-text answers. J Med Internet Res 2019 Sep;21(9):e14347 [FREE Full text] [CrossRef] [Medline]
- Golberstein E, Kolvenbach S, Carruthers H, Druss B, Goering P. Effects of electronic psychiatric consultations on primary care provider perceptions of mental health care: survey results from a randomized evaluation. Healthc (Amst) 2018 Mar;6(1):17-22. [CrossRef] [Medline]
- Jetelina KK, Woodson TT, Gunn R, Muller B, Clark KD, DeVoe JE, et al. Evaluation of an electronic health record (EHR) tool for integrated behavioral health in primary care. J Am Board Fam Med 2018;31(5):712-723 [FREE Full text] [CrossRef] [Medline]
- Madden JM, Lakoma MD, Rusinak D, Lu CY, Soumerai SB. Missing clinical and behavioral health data in a large electronic health record (EHR) system. J Am Med Inform Assoc 2016 Nov;23(6):1143-1149 [FREE Full text] [CrossRef] [Medline]
- Martin K, Ham E, Hilton NZ. Documentation of psychotropic pro re nata medication administration: an evaluation of electronic health records compared with paper charts and verbal reports. J Clin Nurs 2018 Aug;27(15-16):3171-3178. [CrossRef] [Medline]
- Reyes-Portillo JA, Chin EM, Toso-Salman J, Blake Turner J, Vawdrey D, Mufson L. Using electronic health record alerts to increase safety planning with youth at-risk for suicide: a non-randomized trial. Child Youth Care Forum 2018 Jan;47(3):391-402. [CrossRef]
- Riahi S, Fischler I, Stuckey MI, Klassen PE, Chen J. The value of electronic medical record implementation in mental health care: a case study. JMIR Med Inform 2017 Jan;5(1):e1 [FREE Full text] [CrossRef] [Medline]
- Ser G, Robertson A, Sheikh A. A qualitative exploration of workarounds related to the implementation of national electronic health records in early adopter mental health hospitals. PLoS One 2014;9(1):e77669 [FREE Full text] [CrossRef] [Medline]
- Skelton L, Rogers J, Kalafatis C. Development and implementation of electronic medical handovers across psychiatric hospitals: quality improvement initiative. BMJ Open Qual 2019;8(4):e000630 [FREE Full text] [CrossRef] [Medline]
- Stanhope V, Matthews EB. Delivering person-centered care with an electronic health record. BMC Med Inform Decis Mak 2019 Aug;19(1):168 [FREE Full text] [CrossRef] [Medline]
- Baumann LA, Baker J, Elshaug AG. The impact of electronic health record systems on clinical documentation times: a systematic review. Health Policy 2018 Aug;122(8):827-836. [CrossRef] [Medline]
- Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res 2010 Aug;10:231 [FREE Full text] [CrossRef] [Medline]
- Castillo VH, Martínez-García AI, Pulido JRG. A knowledge-based taxonomy of critical factors for adopting electronic health record systems by physicians: a systematic literature review. BMC Med Inform Decis Mak 2010 Oct;10:60 [FREE Full text] [CrossRef] [Medline]
- Delardes B, McLeod L, Chakraborty S, Bowles K. What is the effect of electronic clinical handovers on patient outcomes? a systematic review. Health Informatics J 2020 Dec;26(4):2422-2434. [CrossRef] [Medline]
- Gephart S, Carrington JM, Finley B. A systematic review of nurses' experiences with unintended consequences when using the electronic health record. Nurs Adm Q 2015;39(4):345-356. [CrossRef] [Medline]
- Goldstein DH, Phelan R, Wilson R, Ross-White A, VanDenKerkhof EG, Penning JP, et al. Brief review: adoption of electronic medical records to enhance acute pain management. Can J Anaesth 2014 Feb;61(2):164-179. [CrossRef] [Medline]
- Goldzweig CL, Orshansky G, Paige NM, Miake-Lye IM, Beroes JM, Ewing BA, et al. Electronic health record-based interventions for improving appropriate diagnostic imaging: a systematic review and meta-analysis. Ann Intern Med 2015 Apr;162(8):557-565 [FREE Full text] [CrossRef] [Medline]
- Lau F, Price M, Boyd J, Partridge C, Bell H, Raworth R. Impact of electronic medical record on physician practice in office settings: a systematic review. BMC Med Inform Decis Mak 2012 Feb;12:10 [FREE Full text] [CrossRef] [Medline]
- Meißner A, Schnepp W. Staff experiences within the implementation of computer-based nursing records in residential aged care facilities: a systematic review and synthesis of qualitative research. BMC Med Inform Decis Mak 2014 Jun;14:54 [FREE Full text] [CrossRef] [Medline]
- Nguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform 2014 Nov;83(11):779-796. [CrossRef] [Medline]
- Cohen DJ, Crabtree BF, Etz RS, Balasubramanian BA, Donahue KE, Leviton LC, et al. Fidelity versus flexibility: translating evidence-based research into practice. Am J Prev Med 2008 Nov;35(5 Suppl):S381-S389. [CrossRef] [Medline]
- Ellis LA, Augustsson H, Grødahl AI, Pomare C, Churruca K, Long JC, et al. Implementation of e-mental health for depression and anxiety: a critical scoping review. J Community Psychol 2020 Apr;48(3):904-920. [CrossRef] [Medline]
- Shelton RC, Cooper BR, Stirman SW. The sustainability of evidence-based interventions and practices in public health and health care. Annu Rev Public Health 2018 Apr;39:55-76. [CrossRef] [Medline]
- Carayon P, editor. Handbook of human factors and ergonomics in health care patient safety. Mahwah, NJ: Lawrence Erlbaum Associates; 2007.
- Wiklund M, Kendler J, Strochlic A. Usability Testing of Medical Devices. Boca Raton: CRC Press; 2016.
- Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform 2016 May;Suppl 1:S48-S61 [FREE Full text] [CrossRef] [Medline]
- Bardach SH, Real K, Bardach DR. Perspectives of healthcare practitioners: an exploration of interprofessional communication using electronic medical records. J Interprof Care 2017 May;31(3):300-306 [FREE Full text] [CrossRef] [Medline]
- Elias B, Barginere M, Berry PA, Selleck CS. Implementation of an electronic health records system within an interprofessional model of care. J Interprof Care 2015;29(6):551-554. [CrossRef] [Medline]
- Ratwani RM, Zachary Hettinger A, Kosydar A, Fairbanks RJ, Hodgkins ML. A framework for evaluating electronic health record vendor user-centered design and usability testing processes. J Am Med Inform Assoc 2017 Apr;24(e1):e35-e39 [FREE Full text] [CrossRef] [Medline]
- Bruns EJ, Parker EM, Hensley S, Pullmann MD, Benjamin PH, Lyon AR, et al. The role of the outer setting in implementation: associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare. Implement Sci 2019 Nov;14(1):96 [FREE Full text] [CrossRef] [Medline]
- Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools. theory in implementing evidence-based findings into health care practice. J Gen Intern Med 2006 Feb;21 Suppl 2:S43-S49 [FREE Full text] [CrossRef] [Medline]
- Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011 Apr;6:42 [FREE Full text] [CrossRef] [Medline]
- Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med 2010 Oct;8:63 [FREE Full text] [CrossRef] [Medline]
|EMR: electronic medical record|
|IT: information technology|
|OECD: Organisation for Economic Co-operation and Development|
Edited by J Torous, G Eysenbach; submitted 20.05.21; peer-reviewed by B Druss, A Newton; comments to author 10.06.21; revised version received 15.07.21; accepted 22.07.21; published 07.09.21Copyright
©Yvonne Zurynski, Louise A Ellis, Huong Ly Tong, Liliana Laranjo, Robyn Clay-Williams, Luke Testa, Isabelle Meulenbroeks, Charmaine Turton, Grant Sara. Originally published in JMIR Mental Health (https://mental.jmir.org), 07.09.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on https://mental.jmir.org/, as well as this copyright and license information must be included.