This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Mental Health, is properly cited. The complete bibliographic information, a link to the original publication on http://mental.jmir.org/, as well as this copyright and license information must be included.
Mild traumatic brain injury (mTBI) occurs at a higher frequency among military personnel than among civilians. A common symptom of mTBIs is cognitive dysfunction. Health care professionals use neuropsychological assessments as part of a multidisciplinary and best practice approach for mTBI management. Such assessments support clinical diagnosis, symptom management, rehabilitation, and return-to-duty planning. Military health care organizations currently use computerized neurocognitive assessment tools (NCATs). NCATs and more traditional neuropsychological assessments present unique challenges in both clinical and military settings. Many research gaps remain regarding psychometric properties, usability, acceptance, feasibility, effectiveness, sensitivity, and utility of both types of assessments in military environments.
The aims of this study were to explore evidence regarding the use of NCATs among military personnel who have sustained mTBIs; evaluate the psychometric properties of the most commonly tested NCATs for this population; and synthesize the data to explore the range and extent of NCATs among this population, clinical recommendations for use, and knowledge gaps requiring future research.
Studies were identified using MEDLINE, Embase, American Psychological Association PsycINFO, CINAHL Plus with Full Text, Psych Article, Scopus, and Military & Government Collection. Data were analyzed using descriptive analysis, thematic analysis, and the Randolph Criteria. Narrative synthesis and the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews) guided the reporting of findings. The psychometric properties of NCATs were evaluated with specific criteria and summarized.
Of the 104 papers, 33 met the inclusion criteria for this scoping review. Thematic analysis and NCAT psychometrics were reported and summarized.
When considering the psychometric properties of the most commonly used NCATs in military populations, these assessments have yet to demonstrate adequate validity, reliability, sensitivity, and clinical utility among military personnel with mTBIs. Additional research is needed to further validate NCATs within military populations, especially for those living outside of the United States and individuals experiencing other conditions known to adversely affect cognitive processing. Knowledge gaps remain, warranting further study of psychometric properties and the utility of baseline and normative testing for NCATs.
Mild traumatic brain injuries (mTBIs), also known as concussions, are generally defined as a temporary change in brain functioning caused by an insult to the head, with a period of posttraumatic amnesia lasting less than a day [
Premature return to duty after sustaining mTBIs is inherently associated with heightened risk. This includes increased chances of sustaining a subsequent concussion before neurological recovery. This has the potential to amplify the risk for impaired performance, making mission failure more likely, and endangering the safety of self and others [
Traditional neuropsychological assessments are generally composed of measures with large normative databases and demonstrate evidence of adequate psychometric properties [
Although neuropsychological assessments have been used in psychology for over 100 years, there remain many questions, logistical issues, and psychometric challenges around their use, especially in the military context. Traditional neuropsychological testing can be time intensive for both the health care professional and the patient, expensive for patients, and less feasible to administer in combat settings [
In the last 20 years, alternatives to traditional neuropsychological assessments have emerged in the form of computerized neurocognitive assessment tools (NCATs) [
NCATs developed in recent years are promising for use within military populations, especially with younger demographics. Currently, NCATs are used by military health care providers to assess the effects of mTBIs in both deployed and nondeployed settings [
NCATs may have multiple benefits such as faster administration time, automated scoring and statistical analysis, easier reporting, and ease of deidentification of patients for research purposes [
Although NCATs are currently used in military health care practices, a better understanding (or more information) about their feasibility, effectiveness and psychometric properties is needed. Owing to the relatively recent digital evolution, NCATs generally have not undergone the same degree of rigorous evidence-based psychometric evaluation as in traditional neuropsychological testing. Consequently, validity, reliability, specificity, and overall effectiveness may not be as well established for NCATs [
Diagnosing mTBIs on an individual basis has, to date, not been possible using a single traditional or computerized assessment. This diagnostic challenge can be attributed, in part, to large variations in baseline neurophysiological function and the presence of transient interferences such as learning effects, fatigue, anxiety, and unrelated states of mental alertness or illnesses [
There have been a number of literature reviews published in the past 20 years, focusing on the usage of NCATs to assess sport-related mTBIs [
It is essential that military personnel be considered a unique subset of the adult mTBI population for many reasons. First, military personnel exhibit higher rates of conditions such as posttraumatic stress disorder (PTSD), depression, anxiety, sleep disorders, chronic pain, substance abuse disorders, and mTBIs, which can cause and adversely affect the severity, longevity, and dysfunctionality of symptoms including associated cognitive dysfunction [
There is a need for the improved detection of neurocognitive deficits in the military setting to assist with the diagnosis of mTBIs, rehabilitation planning, tracking recovery, and making return-to-duty decisions while maintaining the productivity and safety of the military population and the civilians they may interact with at home and on deployment. An up-to-date scoping literature review of the current evidence related to NCAT usage among military members who sustained mTBIs is warranted because of (1) the lack of specificity to military populations among previous literature reviews regarding NCATs and mTBIs, (2) the rapid development of NCATs, and (3) the frequency of clinical usage among military health care. This scoping review aims to fill this knowledge gap.
The purpose of this scoping review is to (1) explore the existing evidence regarding the use of NCATs among military personnel who have sustained mTBIs, (2) evaluate the psychometric properties of the most commonly tested NCATs for this population, and (3) synthesize the data to explore the range and extent of NCATs among this population, clinical recommendations for use, and knowledge gaps requiring future research. This scoping review aims to answer the following research questions: (1) To what extent and which NCATs are being used within the military mTBI context? (2) What evidence exists regarding the validity, reliability, feasibility, technology acceptance, usability, and security of NCATs in the military and mTBI context? (3) What are the themes, clinical recommendations, and considerations in the evidence-based literature regarding the use of NCATs for military personnel who have sustained mTBIs? (4) What are the knowledge gaps and future directions of research that need to be addressed regarding the usage of NCATs for military personnel who have sustained mTBIs?
A scoping review is a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence, and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge [
This study employed the following overarching steps: (1) formulation of the research questions based on Population, Intervention, Comparison, and Outcome guidelines; (2) identification of relevant studies; (3) selection of studies; (4) charting of data; and (5) collation, analysis, summarization, and reporting of results [
Relevant studies were systematically identified. A description of the information sources, search strategy, inclusion and exclusion criteria, and selection process is provided in the following sections.
A search strategy was developed based on specific inclusion and exclusion criteria and included the following databases: MEDLINE (Ovid MEDLINE ALL), Embase (Ovid interface), the American Psychological Association (APA) PsycINFO (Ovid interface), CINAHL Plus with Full Text (EBSCOhost interface), Psych Article (EBSCOhost interface), Scopus, and Military & Government Collection (EBSCOhost interface). The search consisted of an extensive list of keywords and subject headings covering 3 concepts: (1) NCATs, (2) military personnel, and (3) mTBIs. The 3 concepts were then combined with the Boolean AND. Studies were limited to peer-reviewed and gray literature papers in English. The initial search for papers took place on April 15 and April 21, 2020, within the aforementioned databases. The full search strategy is available in
Papers selected for inclusion in this study focused on military personnel who had a primary diagnosis of mTBIs. Targeted papers specifically addressed the usability, feasibility, reliability, validity, sensitivity, and efficacy of one or more NCATs among military personnel who have sustained mTBIs. Studies were excluded if the NCAT was used to measure the outcome of an intervention such as cognitive rehabilitation therapy, hyperbaric oxygen, or psychotherapeutic interventions. If the published work included healthy participants or participants with comorbid conditions, such as other mental health disorders, disrupted sleep, chronic pain, or substance use disorder, it was included if the additional conditions were secondary to the mTBI diagnosis and not the primary focus of the specific research study. Cognitive assessment practices that incorporated virtual reality were permitted for inclusion.
The papers included in the data set were quantitative, qualitative, mixed methods, and meta-analyses, regardless of positive, negative, or neutral findings. Papers were excluded from the review if they did not meet the inclusion criteria. Studies that exclusively addressed civilians or veterans were also excluded.
The study selection phases followed a variation of the procedures used by Miguel Cruz et al [
The research team extracted data from the final selected papers according to the following domains: population (medical condition, age, specific military conflict, condition, race or ethnicity, sample size [N], and mean age [SD] in years), study features, clinical assessment, assessment of technology usability, technology outcome measures, technology, duration, and data analysis strategies. The researchers met regularly and reconciled the differences through discussion. In case of any disagreement, one of the researchers acted as a third rater.
All data were analyzed and validated by at least two team members involved in the analysis. The research team met regularly to discuss data extraction, analysis, and synthesis, which were iterative and, in some cases, concurrent. Any discrepancies in the analysis of quantitative or qualitative data were resolved through discussion. This nonlinear process served to improve the rigor and internal validity of the review.
A narrative synthesis was conducted to organize, describe, and interpret the results of the analysis [
The search strategy yielded 372 articles (PRISMA diagram,
A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for Scoping Reviews chart of the scoping review study identification, selection, exclusion, and inclusion.
All included studies (n=33) were quantitative, and most studies were published in the United States and used the US military personnel as participants (
A summary of the outcomes of the 33 included studies is presented in
The most commonly used NCATs among the 33 studies were the versions of the Automated Neuropsychological Assessment Metric (ANAM; 22/46, 46%), Defense Automated Neurobehavioral Assessment (DANA; 7/46, 15%), and Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT; 5/46, 10%). A variety of secondary measures were collected across the studies, including other neuropsychological assessments and screens related to mTBIs (
Thematic analysis and narrative synthesis revealed a number of topics related to the facilitators and barriers of NCAT usage among military populations. The 3 main themes that emerged through the studies included (1) comparing
A number of challenges around the approaches and comparisons used to establish the psychometric properties of the current NCATs were discussed across the included studies. Multiple studies noted that the comparisons made in research when assessing NCATs have important implications on the results and conclusions garnered from the current literature. These can have the potential to adversely affect reliability, validity, sensitivity, detection of reliable change, and overall clinical utility.
As a gold standard NCAT does not exist, comparisons between NCATs and traditional neuropsychological assessments are often used to determine how well tests relate to similar cognitive measures (convergent validity) and differ from dissimilar cognitive tests (discriminant validity) [
Similarly, comparing different NCATs among each other can also be problematic when trying to establish validity. Although these assessments may aim to measure similar cognitive domains or constructs; they may measure or calculate scores for a particular construct differently [
The included studies compared the participants’ results with their baseline data, normative data, or both. When synthesizing findings across the studies included in this review, questions were raised regarding whether baseline or normative data comparisons were the most effective for establishing a change in performance among military personnel who had sustained mTBIs. Two papers specifically discussed this issue at length [
Finally, even the comparison of those who have sustained mTBIs in healthy control groups can affect the results of studies related to NCATs. If some members of the mTBI group were asymptomatic, clinically meaningful differences between controls and those with symptoms could have been washed out, leading to limited effect sizes [
Four studies specifically addressed test-retest reliability for various NCATs among military populations. Dretsch et al [
Other types of reliability, such as internal consistency, were challenging to establish or judged to be not of adequate quality. Differences in the characteristics of test batteries, the design of test-retest studies, and insufficiently explained and nonstandardized methods of analysis makes it challenging to determine the reliability of NCATs [
One study specifically addressed the sensitivity of the DANA to detect small changes in neurofunctioning related to subconcussive blast pressure. LaValle et al [
Multiple studies included in the scoping literature review commented on issues with validity among NCATs both within military and nonmilitary populations who have sustained mTBIs. Studies discussed criterion, convergent, discriminant, and performance validity. Factors that can affect the construct validity of NCATs and traditional neuropsychological assessments include mental fatigue, physical environment, participant effort, practice effects, and the Monte Carlo effect, among others [
Some of the included papers discussed the threats to validity that arise with the variability of normative data sets in which NCAT scores are compared. For example, some NCATs, such as the ANAM, have normative data specific to the military population, whereas others, such as the ImPACT, are compiled from the general population. Some of the studies included in this scoping review discussed the problems that can occur when comparing military and civilian populations [
Recent evidence suggests that PCS is not specific to mTBIs and that symptoms following deployment are better accounted for by mental health diagnoses, such as PTSD, than by the history of mTBIs [
Many of the reviewed studies discussed the question of ecological validity among NCATs. Both NCATs and traditional neuropsychological assessments are typically administered in controlled clinical or research settings to obtain the best possible performance of the patient [
Four studies specifically addressed convergent, discriminant, and performance validity for various NCATs, including the Virtual Reality Stroop Test (VRST) [
Only one study specifically addressed reliable change estimates for the ANAM4-TBI-MIL [
Very few of the included studies addressed the clinical utility of the NCATs. Three studies addressed the usage of NCATs in varying environments, concluding that the DANA and the ANAM demonstrated comparable results and validity when comparing results from a controlled clinical setting with battlefield and deployment settings [
The purpose of this scoping literature review was to systematically explore the evidence regarding the use of NCATs among military personnel who have sustained mTBIs, evaluate the psychometric properties of the most commonly used NCATs for this population, and synthesize data around clinical recommendations for use, knowledge gaps, and future research directions. In total, 33 studies were included in this literature review, covering a range of constructs and topics related to NCATs. Three NCATs—the ANAM, ImPACT, and DANA—were the most commonly analyzed within the 33 studies.
This study was specific to military personnel who had sustained mTBIs. Many published articles have addressed the psychometric properties of NCATs, such as reliability, validity, sensitivity, and clinical utility, when used to assess cognition following mTBI in the general civilian population. Although this evidence was reviewed at length and used to lay the foundation of this paper, it did not meet the inclusion criteria to be included among the final 33 selected articles.
The 5 criteria proposed by Randolph et al [
For the first criterion of test-retest reliability, preliminary evidence demonstrates good test-retest reliability for the ImPACT and DANA among healthy military personnel and those with mTBIs and good-to-excellent test-retest reliability for the ANAM [
The second criterion was the sensitivity of the tests in the clinical issue of interest. This was addressed by one study in relation to the DANA, which showed favorable results among a group of healthy male military members (n=202) [
Validity is the most important aspect of test construction and must be considered when evaluating the clinical utility of a clinical assessment [
The fourth criterion includes establishing reliable change scores and scoring algorithms for classifying impairment. Of the 33 studies, 1 study addressed reliable change scores for the ANAM; however, a reliable change index was established in 2018 for the ANAM using norms from the general population [
The fifth criterion, clinical utility, reveals the most significant knowledge gaps pertinent to patients, health care professionals, and health care organizations. Although the psychometric properties of any clinical outcome measure or assessment are important to establish among the population and condition in question, the discussion of feasibility, accessibility, acceptability, usability, appropriateness, specificity, and other factors is also equally important [
Several additional issues were observed regarding the collective studies included in this study. First, the classification or diagnosis of mTBIs varied across all studies. Some studies relied on self-report to categorize participants into either an mTBI group or a healthy group. This practice is problematic for the following reasons. It is known that mTBIs and other injuries are widely underreported among military personnel. Participants may underreport mTBIs, whether intentionally or because they lack the health literacy to determine whether they have experienced a possible mTBI. It is also possible that some participants did not remember the event or pushed through it in a combat situation without recognizing it as an mTBI. Some studies either classified mTBIs as sustaining loss of consciousness or used symptom reporting to determine the incidence of mTBIs. Other studies used outcome measures with a set threshold to determine if participants would be in the mTBI or healthy group. Many of these outcome measures, which largely depend on self-reporting of somatic symptoms, do not have clear cut-offs to suspect mTBIs and do not have diagnostic utility. However, further studies reviewed medical records and relied on the diagnosis of mTBIs issued by a health care professional. The variability in methods and inclusion criteria for the mTBI group could affect validity and potentially facilitate the inclusion of those with mTBIs in the healthy group, which increases the chances of type 2 error.
The second concern highlighted by this review is the variation in whether or how the included studies screened participants for secondary conditions, such as depression, PTSD, fatigue and pain. These secondary conditions are known to adversely affect cognitive performance and could act as confounding variables [
The baseline-referenced comparison approach has minimal supportive evidence from clinical trials but is the standard approach used in sports mTBI management and is favored by the US military, particularly with the ANAM [
When comparing results from a military cohort on the ANAM with both normative and baseline data, no statistical differences between the baseline-referenced approach and the norm-referenced approach for determining decrements in ANAM performance following mTBIs were observed [
In their 2017 paper, Coffman et al [
Apart from the aforementioned occurrences of certain comorbidities within the military population, normative data based on a general adult civilian population tend to include wider age ranges from 18 to 85 years. The military population is much younger, often within the age range 18-60 years. Within this study, the average age of the participants included was 27.31 (SD 4.10) years, much younger than the normative age included in the general population norms. Studies addressing norm-based and baseline comparisons within military populations demonstrated variable results and raised more questions on best practices for clinical interpretation of cognitive performance scores on NCATs. This requires future consideration and research with military populations.
A number of key recommendations were isolated from studies that are relevant for health care professionals. Most prominently, NCATs should be used cautiously and only as one source of information from among many other types of clinical tools and observations. It is not advisable that NCATs be used as a definitive or standalone diagnostic tool [
The findings of this scoping literature review have led to the formulation of the following recommendations for future research. First, it is apparent that more research is needed to better establish the psychometric properties of NCATs among military and civilian populations from a global perspective. Studies conducted in countries or military organizations outside of the United States are needed to assess constructs related to clinical utility within their specific contexts and populations. Research on the usage of NCATs within different deployment environments would also be beneficial. Furthermore, longitudinal studies that address temporal stability or test-retest reliability over time with different NCATs would be an asset. Studies that address the psychometric properties and clinical utility of NCATs with other conditions known to adversely affect cognitive functioning among military populations, such as depression, PTSD, sleep deprivation, chronic pain, and others, would be particularly beneficial. This would allow clinicians to better assess cognitive performance allowing them to make more informed clinical decisions. These decisions have the potential to influence the function, productivity, and safety of military members, their units, and those they interact with through their high-stake occupations. This would also assist clinicians in designing rehabilitation plans that target specific domains of cognition, leveraging cognitive strengths, and targeting areas of reduced performance.
Studies with a larger number of military personnel with mTBIs, or other conditions that affect cognition, would be an asset, especially for clarifying recovery trajectories and possibly return-to-duty decisions [
Further research is also needed to better determine if using NCATs for baseline testing is indicated or if normative-based comparisons are valid for use in a clinical setting. Furthermore, the field would benefit from the establishment of standardized NCAT norms for military populations that represent not only healthy individuals but also those with mTBIs and other conditions that affect cognition.
Finally, studies that further address clinical utility, including the feasibility, accessibility, acceptability, usability, appropriateness, specificity, and other pragmatic factors, are needed to contextualize the use of NCATs and assist health care professionals with clinical decision making around which NCAT to use in practice, what rehabilitation is indicated, and how NCATs may guide the return-to-duty decisions. Evidence-based literature and guidelines on best practices that discuss facilitators, barriers, and recommendations for NCATs and digital health technologies would support health care professionals working with military personnel experiencing cognitive dysfunction.
There are a number of notable strengths of this scoping review. This study was conducted following a planned a priori procedure, with attention to ensuring quality control and minimizing bias. The detailed search strategy was extensive, including 7 databases. The inclusion and exclusion criteria were determined before study onset and adhered to throughout. Appropriate calibration and pilot testing, use of at least two independent reviewers for all stages of the process, and group discussion of conflicts improved the quality of this scoping review.
Several limitations of this scoping review also warrant discussion. First, although the review process was calculated and rigorous, it is possible that relevant studies related to military personnel with mTBIs and NCATs were overlooked. Second, it is noted that other studies specific to civilian populations exist that were not included in this scoping review, which may include important information. Third, with the rapid rate of research and publishing on this topic, it is plausible that additional research has been published before the release of this scoping review. Finally, the limits of aggregate data and specific nuanced details may have become generalized during the synthesis process.
Cognitive functioning is imperative to the day-to-day activities of military personnel in their work, self-care, and leisure activities. Military members must be able to make decisions in precarious and ambiguous situations where risk to self and others is high and must possess an adequate level of cognitive functioning to communicate, use weapons and technological devices, and perform other military duties without error. Assessing cognitive functioning is part of a multidisciplinary best practice protocol for the management and treatment of mTBIs [
The results of this study indicated that the published literature regarding NCAT usage among military personnel who have sustained mTBIs is quite heterogeneous in study design, construct being measured, and outcome goals. On the basis of the 5 Randolph Criteria [
Detailed search strategy.
Summary of included studies.
Detailed descriptive analysis of studies included in the scoping review.
Automated Neuropsychological Assessment Metric
CNS-Vital Sign
Axon Sports’ CogState Sport
Defense Automated Neurobehavioral Assessment
intraclass correlation coefficient
Immediate Post-Concussion Assessment and Cognitive Testing
mechanism of injury
mild traumatic brain injury
Computerized Neurocognitive Assessment Tools
Operation Enduring Freedom
postconcussion symptoms
Population, Intervention, Comparison, and Outcome
Preferred Reporting Items for Systematic Reviews and Meta-analyses extension for Scoping Reviews
prolonged reaction time
posttraumatic stress disorder
simple reaction time
Virtual Reality Stroop Test
None declared.