All procedures were approved by the Teachers College, Columbia University Institutional Review Board (protocol number: 21-104). Participants provided informed consent prior to participation and provided additional verbal consent immediately before the interview, at which time they were informed that their interview transcripts could be analyzed using machine-learning methods.

Consent materials specified that to maintain participants’ confidentiality and anonymity, they would be assigned a code number, that the association of the code number with their name would be kept in a password-protected file and stored on a secured server, that the code number would be used to record participants’ responses to all tasks and questionnaires and that their name would never be recorded on any task or questionnaire data, that only the principal investigator and members of the research team would view participants’ written, electronic, or digital materials, and that these materials would be stored on a password-protected computer. Consent materials also informed participants that the investigators may wish to contact them in the future regarding this study or a possible future study and asked them to initial a statement on the consent form indicating whether they were willing or not willing to be contacted for these purposes in the future. Participants were paid US $60 for completion of the questionnaires and interview.

Participants were 106 US military veterans recruited via online social media platforms (eg, Reddit) with permission from veteran-specific forum moderators and from prior related studies in which participants had consented to be recontracted for future research as part of a larger veteran project. Eligibility criteria included being 18 to 70 years of age, residing in the United States, and having served in the military after September 11, 2001. Veteran status was verified prior to participation through official documentation (DD214, military ID card, or VA ID card). Demographic characteristics are presented in Table 1.

Following consent, participants completed a semistructured interview assessing military-to-civilian transition experiences using the LOTTS interview protocol [11]. All interviews were conducted remotely via Zoom by graduate-level research assistants who received standardized training in the LOTTS protocol and were required to meet competency criteria established prior to conducting interviews independently.

Interviews were semistructured and included standardized prompts with follow-up questions used to clarify and elaborate on participants’ responses as needed. Total interview duration ranged from approximately 20 to 40 minutes, depending on the number and type of experiences reported (eg, losses and deployment experiences). The transition narrative segment used for the present analyses consisted of an open-ended prompt in which participants were invited to describe their transition to civilian life for up to 7 minutes. All interviews were transcribed and deidentified prior to analysis. Participants also completed standardized self-report clinical measures.

A multiphase approach was used (see Figure 1). In phase 1, the senior author (GAB) convened repeated meetings with 10 experts who had prior experience working with veteran populations. These iterative meetings served as the training and calibration procedure: the team collaboratively developed the operational definition of veteran transition stress and the 5-point rating scale with descriptive anchors at each level, ensuring a shared understanding of the construct and rating criteria before independent coding began. The 10 experts then independently rated 20 randomly selected transcripts for transition stress based on the consensus criteria (see Materials in Multimedia Appendices 1 and 2). Raters were provided only with deidentified transition narrative transcripts and the rating criteria; they did not have access to participants’ clinical measure scores, demographic information, or other raters’ scores during the coding process. Individual rater disagreements were not adjudicated through consensus discussion. Instead, the average of all 10 expert ratings was used as the composite human score for each transcript, preserving natural rater variability essential for a meaningful assessment of interrater reliability. These composite scores were then compared to ratings from an LLM for transition stress modeled after the same consensus criteria.

All AI ratings were generated using Google’s Gemini 2.5 Flash model, accessed via the Google Gemini application programming interface (API). The model version was locked for the duration of the study to ensure consistency across all rating sessions. The following API parameters were used: temperature=0 (greedy decoding, to minimize stochastic variation across runs), and max output tokens were set to accommodate both a numerical rating and a written explanation for each transcript. No system-level instructions were used beyond the prompt itself. The prompt structure for the transition stress rating task consisted of 3 components. First, a role assignment instructed the model to adopt the perspective of a veteran transition stress rater (“You are a veteran transition stress rater”). Second, the complete operational definition of transition stress was provided verbatim within the prompt, specifying that transition stress refers to the psychological and emotional strain experienced during periods of significant change for veterans, including domains such as employment, family reacclimation, relationships, finances, and schedule management. Third, the full 5-point rating scale with anchored descriptors (1=minimal or no distress; 2=minimal to moderate; 3=moderate; 4=moderate to high; and 5=high distress) was included. The model was then instructed to rate the level of transition stress in each attached transcript according to the definition and scale and to provide explanations for each rating.

Five prompting conditions were tested to evaluate the effect of in-context learning on rating performance: 0-shot (definition and scale only, no examples), 1-shot (1 expert-rated example transcript), 2-shot (2 examples), 3-shot (3 examples), and 5-shot (5 examples). In the few-shot conditions, example transcripts were selected to span the full range of the rating scale and were accompanied by the expert consensus rating (mean of 10 raters) and 3 illustrative expert explanations that articulated the rationale for the assigned rating. Specifically, the 1-shot condition included a high-stress example (expert mean 4.9/5, SD 0.32); the 2-shot condition added a low-stress example (expert mean 1.0/5, SD 0); the 3-shot condition added a moderate-stress example (expert mean 3.8/5, SD 0.42); and the 5-shot condition further added a low-moderate example (expert mean 2.0/5, SD 0.82) and a moderate example (expert mean 3.0/5, SD 0.67). Example transcripts were excluded from the rating set for their respective conditions to prevent circularity (eg, in the 1-shot condition, 19 of 20 calibration transcripts were rated; in the 5-shot condition, 15 were rated). Transcripts were provided to the model as attached Excel files containing the raw narrative text from the LOTTS transition interview. The complete prompts for all 5 conditions are provided in Multimedia Appendix 3.

The 0-shot condition was selected as the primary analysis because it achieved the highest agreement with human experts (intraclass correlation coefficient [ICC](A,k)=0.954, r=0.955) while requiring no training examples, maximizing scalability and eliminating any risk of overfitting to the calibration sample. The 0-shot prompt was subsequently applied to all 106 transcripts in the full sample for the construct validity and thematic analyses reported in Phases 2 and 3.

Ten human experts rated 20 randomly selected transcripts. The average-measures ICC for absolute agreement was excellent (ICC(A, 10)=0.978, 95% CI 0.960-0.990; F_{(19, 160)}=49.77, P<.001). The single-measures ICC was also strong (ICC(A, 1)=.817, 95% CI 0.708-0.908), indicating that even individual raters achieved good reliability. Krippendorff α for ordinal data was α=0.799, approaching the conventional threshold of 0.80. Pairwise correlations among the 10 raters ranged from 0.61 to 0.97 (mean 0.83, SD 0.10). Individual rater means ranged from 2.25 to 2.95 (on the 1‐5 scale), indicating minimal systematic rater bias. The zero-shot AI rater (Gemini 2.5 Flash) achieved excellent agreement with the human expert mean: ICC(A, k) 0.954, 95% CI 0.823-0.984; r=0.955, 95% CI 0.887-0.982; P<.001; mean absolute error=0.49; root mean square error=0.61. Performance was comparable to or slightly lower in few-shot conditions (see Table 3), confirming that additional training examples did not improve upon 0-shot performance. To evaluate whether exemplar-based calibration would improve scale interpretation, we additionally tested 1-, 2-, 3-, and 5-shot prompting conditions. In these conditions, the model was provided with human expert consensus ratings and qualitative rationales for representative transcripts spanning the rating scale. The 0-shot condition matched or exceeded the reliability of all few-shot conditions, suggesting that the construct definition and scale descriptors were sufficient for the model to achieve consistent interpretation aligned with human expert judgment.

As hypothesized, AI-rated transition stress showed significant positive correlations with clinical measures after FDR correction: PTSD symptoms (r=0.40, p_FDR<.001), depression (r=0.34, p_FDR=0.001), loneliness (r=0.42, p_FDR=0.011), and reintegration difficulties (r=0.38, p_FDR <0.001). These associations were robust to demographic controls; partial correlations controlling for age and gender were virtually unchanged (partial r range: 0.21–0.43; see Table 4). The AI-rated transition stress also correlated significantly with trait anxiety (r=0.25, p_FDR=0.042). AI-rated transition stress also showed a strong positive correlation with participants’ self-ratings of experiencing a “difficult/demanding” transition (r=0.70, P<.001). Regressing both the AI-rated transition stress and questionnaire-based reintegration difficulties (M2CQ) onto participant self-ratings explained more than half the variance (R²=0.51, F_{(2, 98)}=50.43, P<.001), with the AI-rated score emerging as the more robust predictor (β=.72, P<.001, sr²=.60) compared to the M2CQ (β=.01, P=.06, sr²=.13).

To rule out the possibility that AI ratings were driven by surface-level text properties rather than psychological content, we correlated transition stress scores with 5 linguistic features of the transcripts (Table 5). Linguistic complexity indicators—type-token ratio (r=.05, P=.59), mean word length (r=0.14, P=.16), and mean words per sentence (r=−0.11, P=.29)—were not significantly associated with AI ratings, confirming that the model was not simply responding to vocabulary sophistication or syntactic complexity. Narrative length (word count: r=0.32; sentence count: r=0.24) showed modest positive correlations with AI scores, which is expected and substantively interpretable: veterans experiencing greater distress had more to report about their transition difficulties. Crucially, this association is content-driven—more distress generates more narrative content—rather than an artifact of the AI rewarding verbosity per se.

AI transition stress score distribution across the 106 transcripts was as follows: score 1: 14 (13.2%); score 2: 23 (21.7%); score 3: 26 (24.5%); score 4: 17 (16.0%); score 5: 26 (24.5%). The primary grouping yielded 37 low (scores 1‐2) and 43 high (scores 4‐5), with 26 participants (24.5%) excluded at score 3. Sensitivity analyses confirmed that high versus low group differences on all 4 primary clinical measures were robust across all 6 threshold definitions tested, with Cohen d values consistently in the medium-to-large range (d=0.57‐2.01 across schemes; see Table 6). The most extreme contrast (score 1 vs 5) yielded the largest effects (d=0.87‐2.01), while broader groupings (eg, median split) produced attenuated but still significant effects (d=0.74‐0.81), confirming that the primary threshold captures meaningful group differences without being dependent on a specific cutoff.

A qualitative thematic analysis comparing low and high transition stress groups, as defined by the AI’s scores, along transition stress themes revealed markedly distinct narrative patterns. The narrative arc of veterans in the low transition stress group was redemptive and coherent and followed a logical cause-and-effect structure focused on starting a new chapter in the present and future. These narratives described themes of high agency (proactive mastery) and communion (building new sources of support), a continuity of identity (military experience integrated as foundation part of self), and low stereotype threat (military service is viewed as one part of a broader self-concept). Coping was described as planful, and problem-focused coping and challenges were framed as manageable “bumps in the road.” In stark contrast, the narrative arc from veterans in the high transition stress group was one of contamination and fragmentation. These narratives articulated themes of low agency (passive victim) and communion (profound isolation), a fractured identity (profound grief over loss of military self), and high stereotype threat (hyperawareness of negative labels and pervasive sense of being misunderstood by civilians). Coping was described as reactive and ineffective, and challenges were framed as insurmountable, leading to spirals of despair. More details on these divergent narrative themes are presented in Table 7. Illustrative text examples of 1 thematic component, narrative arc, along with the AI explanation for the categorization of that component are given in Table A1 in Multimedia Appendix 1.

A meaningful gender disparity also emerged. While the sample of female veterans was small (n=20), they were disproportionately represented in the high transition stress group. Their narratives frequently centered on military sexual trauma coupled with a profound sense of institutional betrayal by leadership, which together became the defining, unresolved theme that precipitated a severely difficult transition (see Table A2 in Multimedia Appendix 2).

One female veteran encapsulated this theme of compounded trauma:

The findings of this study offer a nuanced, dual-pathway understanding of veteran transition stress, revealing that successful and difficult transitions are not merely quantitatively different but are defined by fundamentally distinct narrative structures. A primary insight is that the simple presence of potentially traumatic events during service, such as combat exposure or the loss of comrades, does not predict a high-stress transition by itself. Rather, the crucial differentiator appears in how these experiences are processed, integrated, and framed within the veteran’s transition story. The narratives of low transition stress veterans were characterized by a sense of agency and proactive, problem-focused coping, often involving detailed pretransition planning and the leveraging of social support systems. While many of these individuals experienced significant trauma, their narratives contextualized these events as difficult past experiences that had been processed or compartmentalized, allowing for a future-oriented mindset focused on civilian opportunities. Their stories follow a classic “redemptive” arc—a problem-solution structure where initial hardships are reframed as necessary steps toward a better life.

Conversely, the narratives of high transition stress veterans were defined by compounding crises and a pervasive lack of agency and control. Their transition from the military often occurred involuntarily or unexpectedly, precipitated by medical retirement or personal trauma, and was frequently followed by a cascade of stressors, including unemployment and relationship breakdown. For this group, past traumatic experiences remain a central feature of their present-day experience, fueling a narrative of identity loss, profound social isolation, and being psychologically “stuck” in the past. These stories followed a narrative arc of contamination and fragmentation, dominated by events so powerful they psychologically tainted all other aspects of the veteran’s life. Furthermore, these analyses uncovered a critical gender dimension, where female veterans were disproportionately represented in the high transition stress group. Their narratives were uniquely characterized by military sexual trauma and institutional betrayal, suggesting that the source of stress for these veterans is often the pain of unresolved trauma perpetrated and then inadequately addressed by the institution they served.

These substantive insights were garnered through the application of a novel generative psychometrics framework, which this study served to validate by first establishing the quantitative robustness of AI-driven measurement. Operating in a 0-shot condition, the LLM achieved excellent interrater reliability with 10 human clinical experts (r=0.97). The construct validity of the AI’s ratings was further supported by theoretically congruent correlations with validated clinical measures of PTSD, depression, and loneliness, as well as a strong correlation (r=0.70) with the veterans’ own self-ratings of transition difficulty.

This framework provides a significant qualitative advance over prior automated text analysis methods in psychology. Traditional dictionary-based tools, such as linguistic inquiry and word count [21], quantify predetermined linguistic categories but cannot account for context, narrative structure, or semantic meaning—a limitation that obscures the clinically meaningful complexity of simultaneous relief and grief or the difference between effectively managed anxiety and debilitating distress. Similarly, while topic-modeling approaches (eg, Latent Dirichlet Allocation) [22] identify latent thematic clusters, they are unsupervised and atheoretical, failing to produce participant-level severity ratings anchored to external criteria. The present LLM-based approach overcomes these limitations by combining emergent thematic pattern recognition with the generation of construct-specific, ordinally scaled ratings. By processing meaning in context, the LLM approximates the integrative judgment historically requiring trained human raters.

This approach also represents a paradigm shift regarding scalability. While the human expert panel required an estimated 175 to 265 person-hours to evaluate the 106-transcript corpus, the AI completed the task in less than 30 minutes of computation time. However, this efficiency does not eliminate the need for human expertise; rather, it relies upon a rigorous upfront investment in operational definitions, expert calibration, and quality assurance to safely scale assessment across larger populations.

While LLM-based ratings achieve strong psychometric properties, responsible translation from research to clinical or programmatic use requires stringent ethical, practical, and technical safeguards. The AI rating system validated here is explicitly intended to augment, rather than replace, human clinical judgment. In clinical implementation, it should operate within a human-in-the-loop model as a scalable screening tool, facilitating timely referrals while licensed clinicians retain full decision-making authority. AI-generated scores must never serve as a sole basis for clinical diagnoses, benefits determinations, or access to services, and implementation must include strict oversight to prevent repurposing for punitive or exclusionary surveillance. Furthermore, processing sensitive veteran transcripts necessitates rigorous data privacy protocols, including comprehensive deidentification, enterprise-tier API agreements prohibiting secondary model training, and strict adherence to Health Insurance Portability and Accountability Act of 1996 and VA data security requirements. Prioritizing locally deployable, open-source models may offer the most robust solution for ensuring data sovereignty within VA and Department of Defense contexts.

Addressing the “black box” nature of LLMs is equally critical for clinical acceptability. To ensure transparency, the AI was required to generate a written rationale for each numerical rating, demonstrating that the model attended to substantively relevant narrative content (eg, employment difficulties and identity loss) rather than arbitrary textual features. Additionally, while our partial correlation analyses suggest that convergent validity associations remain stable across age and gender, LLMs may encode societal biases. Future large-scale deployments must include ongoing bias monitoring, formal fairness audits across demographic subgroups, and disaggregated performance metrics to ensure equitable assessment across racial, ethnic, and gender lines.

Several limitations warrant consideration. First, the cross-sectional design provides concurrent validation but precludes establishing predictive validity. Prospective longitudinal studies are necessary to determine whether AI-rated transition stress can accurately predict future clinical outcomes, such as the onset of PTSD or employment difficulties, before symptoms fully manifest. Second, because LLMs are inherently stochastic, minor fluctuations in ratings are possible. Although a temperature setting of 0 (greedy decoding) was utilized to minimize variability, future research must formally evaluate test-retest reliability across repeated runs to quantify stochastic variation. Third, the present findings are specific to the Gemini 2.5 Flash model. Because LLMs undergo continuous updates, “model drift” presents a unique challenge to longitudinal measurement. Our goal here was to determine LLM’s general capabilities in psychometric tasks using unstructured narratives. Future research and clinical deployments must adopt strict model versioning practices and conduct periodic revalidation. Moreover, we did not compare other model’s performance (eg, OpenAI, Anthropic, etc) as our intent was not to determine the best-in-class model or compare performance. Fourth, while the AI-generated themes align closely with established qualitative literature, future studies should incorporate independent human qualitative coding for triangulation to definitively confirm that emergent themes reflect genuine narrative patterns rather than artifacts of the model’s training data.

Regarding sample-related limitations, although we recruited a diverse sample, recruitment through online forums may have introduced sampling bias by excluding veterans who do not use social media. In addition, because the sample consisted exclusively of post-9/11 veterans, further research is needed to determine whether these findings generalize to veterans from earlier service eras or peacetime service. Additionally, while the narrative patterns observed in the low transition stress group—agency, coherence, and redemptive framing—are consistent with theoretical accounts of adaptive functioning, this cross-sectional design does not permit equating narrative style with resilience as a psychological process or outcome. Coherent, future-oriented narratives may reflect genuine psychological adaptation, but they may also be shaped by verbal fluency, self-presentational tendencies, or storytelling ability that operate independently of psychological well-being. Future research using longitudinal designs and multimethod assessment is needed to disentangle narrative coherence from underlying resilience processes. Finally, as the sample was predominantly male, further research is needed to examine whether the thematic associations observed here are consistent among female veterans.

Couched within these limitations, this study demonstrates that Generative AI, applied with psychometric rigor, can significantly advance our substantive understanding of complex psychological constructs, such as veteran transition stress, while validating a powerful new research methodology. By revealing the distinct architectures (defining narrative arcs and themes) of low and high transition stress, this approach provides a more granular picture of the veteran experience and offers a transformative paradigm for integrating quantitative and qualitative analyses in mental health research.