After predicted labels have been parsed, normalized, aligned with ground-truth labels, and validated through final integrity checks, classification performance can be quantified.

The optimal choice of performance metrics depends on the type of classification task being performed. For binary classification tasks, commonly reported measures include accuracy, precision, recall (sensitivity), specificity, and F₁-score. Precision (positive predictive value) reflects how trustworthy a positive prediction is: of all notes the pipeline flags as positive, what proportion are truly positive, with low precision indicating frequent false alarms. Recall (sensitivity) reflects how completely the pipeline captures true cases: of all notes that are truly positive, what proportion the pipeline correctly identifies, with low recall indicating missed cases. The F₁-score summarizes precision and recall into a single value. This is useful when both error types carry meaningful cost. The appropriate emphasis depends on the clinical consequences of each error type. Recall may be prioritized in safety-critical mental health detection tasks such as suicide risk screening, where missing a true positive case could delay identification of a patient in crisis, whereas a higher precision threshold may be preferred in lower-acuity screening tasks to limit clinician alert burden. The metrics should include 95% CIs (eg, bootstrapping with n=1000 resamples), with sample sizes reported for each metric computation.

A confusion matrix may be computed to examine class-level performance by cross-tabulating predicted labels against ground-truth labels to identify true positives, true negatives, false positives, and false negatives. Unlike aggregate accuracy, the confusion matrix reveals systematic error patterns and indicates which direction prompt refinement should take: a pattern dominated by false negatives suggests that the classification criteria are too strict or insensitive to subtle evidence and should be loosened, while a pattern dominated by false positives suggests criteria are too permissive and should be tightened. Reporting raw counts of misclassified results helps contextualize aggregate performance metrics. This level of detail is particularly important in clinical contexts where different error types may carry unequal consequences. Beyond summary metrics, misclassifications must also be qualitatively reviewed to determine whether errors arise from prompt ambiguity, insufficient task definition, dataset labeling inconsistencies, or genuinely ambiguous clinical documentation. If systematic errors reflect misalignment in earlier stages of the workflow, additional prompt or task refinement may be warranted. If remaining errors primarily reflect true clinical ambiguity rather than problems with the prompt, output format, or evaluation pipeline, the workflow may be considered ready for further refinement.

All inference experiments were conducted in locally deployed environments using the Ollama inference server, so clinical notes were never transmitted to external servers. Two versions of Llama were used at different stages, applied identically across both tasks. Llama 3.2 supported rapid, low-cost prompt iteration, while Llama 3.3 was reserved for final evaluation using the frozen prompts (see Refine the Model). Prompt development ran on a MacBook Pro (Apple M3 Pro, 14-core GPU, 36 GB unified memory) using quantized Llama 3.2 (~2 GB); final evaluation ran on a Mac Studio (Apple M2 Ultra, 24-core CPU, 76-core GPU, 128 GB unified memory) using quantized Llama 3.3:70B (~42 GB). Notes were processed sequentially to approximate realistic deployment conditions; Table 1 summarizes models, tasks, prompting configurations, and runtime.

We demonstrate the workflow on 2 clinically distinct tasks. Task A (pediatric SITB detection) evaluates whether a note documents suicidal ideation, suicide attempts, or nonsuicidal self-injury; Task B (antipsychotic medication nonadherence detection) evaluates whether a note documents nonadherence to prescribed antipsychotic medication in individuals with serious mental illness. Both tasks produce either a binary decision or an ordinal confidence score, which are evaluated against ground-truth labels assigned by constructions during synthetic note generation, using accuracy, precision, recall, and F₁-score for binary outputs, and Youden-optimal thresholding to convert Likert predictions to binary decisions.

Prompt development began from minimal binary baseline prompts with no additional guidance: “Determine whether the clinical note contains suicidal ideation, suicide attempts, or self-injury” (Task A) and “Determine whether the patient is adherent or nonadherent to prescribed medications” (Task B). These prompts specified no handling of borderline cases, negation, temporality, or output format and initially produced free-form responses with predictions embedded in explanatory text rather than structured labels (Figure 2).

Prompts were revised to require structured JSON output, a label field ({yes, no}) for binary prompting, or a discrete score s ∈ {−3,...,3} for Likert prompting, and automated parsing scripts extracted and normalized these fields, excluding (rather than manually correcting) malformed outputs so evaluation reflected model behavior rather than postprocessing (Figure 3). Likert scores were converted to binary predictions using a fixed threshold rule (predict yes if s≥τ), and parsed labels were aligned with ground truth via unique note identifiers.

Classification performance was evaluated using accuracy, precision, recall, and F₁-score, along with confusion matrices to examine class-level performance. Metric priorities differed by task; recall was prioritized for Task A because missing a true SITB-positive case could have significant downstream safety consequences [6], whereas precision carried relatively greater weight for Task B to limit unnecessary clinician follow-up alerts. Misclassified cases were qualitatively reviewed to distinguish errors attributable to ambiguous clinical language, prompt limitations, or label inconsistencies.

If prompt refinements fail to fully resolve classification errors, instability, or formatting inconsistencies, another component that can be adjusted is the structure of model inputs and outputs. The way information is presented to the model and the way predictions are returned can influence classification behavior and the reliability of downstream processing.

Prompt development for both tasks used a zero-shot strategy: predictions were guided entirely by natural-language instructions, with no labeled examples and no task-specific fine-tuning. Starting from the baseline prompts (see Create a Baseline Prompt), refinement addressed classification behavior, output structure, and model selection.

Classification refinement targeted early errors involving negated statements (eg, documentation denying suicidal ideation) and historical symptom references by adding explicit inclusion and exclusion criteria. The final SITB prompt specifies “Classify as No only if the patient denies suicidal ideation and no self-injury behavior is described,” and the adherence prompt classifies a case as nonadherent when “missed doses, skipped doses, or inconsistent medication use” are documented. Stability was verified by running repeated inference on the prompt development set under deterministic decoding; for the pediatric SITB task, the confusion matrix (50 true positives, 45 true negatives, 5 false positives, and no false negatives) was unchanged across repeated runs.

Prompts were also revised to require strict JSON output, replacing the free-form, explanation-embedded responses seen during early development (eg, “The patient denies suicidal ideation, so the answer is No”): {“likert_score”: −3|...|3, “likert_score_analysis”: “<one short sentence>“} for Likert mode, {“label”: “Yes”/“No”} for binary mode. Prompts were further organized into clearly labeled, delimited sections (eg, a “Clinical Note”: section with quotation-enclosed text) to separate instructions from source content and reduce parsing ambiguity across batches. For Likert mode, a binary decision threshold (τ) was selected on the prompt-development set by maximizing the Youden J statistic (sensitivity+specificity−1) across candidate thresholds s ∈ {−3,...,3} [19], fixed before evaluation and not adjusted afterward. Refinement continued iteratively until classification performance, stability, and structural compliance stabilized on the prompt development set; the frozen prompts were then applied unchanged to the evaluation datasets. Full prompt text for both tasks, including the best-practice template comparison, is provided in Multimedia Appendix 3.

For model selection, we chose Llama 3.2 and Llama 3.3 over domain-adapted clinical models (eg, BioMistral, ClinicalBERT variants). Both Llama models are available via lightweight local inference frameworks (eg, Ollama), can be deployed in quantized form on consumer and institutional hardware, and demonstrate strong zero-shot instruction-following behavior [20]. Domain-adapted clinical models, by contrast, are less broadly accessible and are not consistently maintained across institutions. Teams with access to such models may find that domain adaptation improves performance on tasks involving specialized terminology; this remains an open question for future evaluation. Smaller instruction-tuned models (eg, Phi-3 and Mistral 7B) are viable alternatives for more constrained hardware.

Llama 3.2 supported rapid iterative refinement during development. Once prompts and schemas were frozen, they were evaluated unchanged on Llama 3.3 under identical conditions so that any performance difference could be attributed to model capability rather than experimental variability. As shown in Figure 4, Llama 3.3 outperformed Llama 3.2 on both tasks. For pediatric SITB (Likert mode), accuracy rose from 65.0% to 97.5% and precision from 58.7% to 98.0%, while recall stayed stable (99.0% vs 97.0%). For antipsychotic nonadherence, accuracy rose from 92.5% to 100.0% and precision from 80.0% to 100.0%, while recall stayed unchanged at 100.0%.

Once the prompts are frozen and applied unchanged to the test data, the classification performance is measured under controlled inference conditions. The goal is to determine whether the prompt design, output schema, and model selection produce reliable predictions for the target clinical task. In this workflow, the system flags notes for clinician review; it does not make autonomous clinical determinations.

All evaluations were conducted on synthetic clinical notes under deterministic inference settings; performance estimates should be interpreted as controlled workflow-validation results rather than real-world deployment benchmarks (see Common Pitfalls and Limitations).

After prompt refinement and model selection were completed (Steps 1‐2), the finalized prompts were applied unchanged to the evaluation datasets using the Llama 3.3 model under deterministic inference settings.

Classification performance was evaluated using accuracy, precision, recall, and confusion matrices for the binary prompting condition. For Likert prompting, ordinal scores were converted to binary predictions using the Youden-optimal threshold determined during development. Likert scores were binarized using τ=−1, the Youden-optimal threshold selected on the prompt development set prior to evaluation (see Prompt Finalization).

Because the evaluation datasets consisted of synthetic notes generated under task-specific specification sheets, the following performance estimates reflect performance on controlled synthetic test sets and should not be interpreted as deployment benchmarks. Table 2 summarizes the overall classification accuracy for both tasks under binary and Likert prompting modes. Across both clinical domains, the structured prompting pipeline achieved high performance with minimal task-specific modification.

Table 2 reports accuracy, precision, recall, and F₁-score for both prompting modes, with CIs reflecting sampling uncertainty on the 200-note evaluation set. The overall pattern is more important than any single number. As shown in the confusion matrix (Figure 5), the model produced zero false negatives and a small number of false positives (6/200), meaning every true SITB-positive case was flagged for review, at the cost of roughly 1 in 17 flagged notes not meeting SITB criteria. That reflects the intended trade-off of recall-prioritized tuning; for a safety-critical task, missing a positive case is treated as more costly than an occasional unnecessary review.

For the antipsychotic medication nonadherence task, the model achieved perfect classification under both prompting modes on this evaluation set (see Table 2). This result reflects the relative simplicity of distinguishing adherence from nonadherence language in notes generated according to specification and should not be treated as a clinical benchmark. Perfect classification on a controlled synthetic set is a common finding when the evaluation set does not capture the full ambiguity of real clinical documentation; validation on real clinical notes is necessary before these results can be interpreted as estimates of deployment performance.

Figure 6 and Figure 7 illustrate the distribution of Likert scores assigned by the model for each task. In both datasets, the model assigned clearly differentiated scores to positive and negative cases. Negative examples were concentrated at the lowest Likert score (−3), indicating strong evidence against the target condition, while positive cases were predominantly assigned scores of 1‐3, indicating graded evidence in favor. The Youden-optimal threshold (τ=−1) fell between these clusters, allowing ordinal outputs to be converted into binary predictions without substantial loss of classification performance. This separation suggests the model’s confidence is well calibrated on synthetic notes.

These results indicate that the prompt-based workflow can produce consistent and well-separated classification behavior under controlled synthetic evaluation conditions using zero-shot prompting. Further validation on real clinical datasets is required to assess performance under real-world deployment conditions.

Pediatric and adult behavioral health documentation differ substantially in structure, tone, and clinical cues, yet applying the same pipeline to both tasks demonstrates that a schema-driven approach grounded in strict JSON outputs and robust parsing can support multiple clinical inference tasks with minimal modification beyond redefining schemas and label policies. This reusability has its limits; the pipeline as described is for single-construct binary or ordinal classification. Tasks requiring temporal reasoning across multiple notes, quantitative extraction (eg, medication dosing), or multilabel phenotyping would necessitate architectural modifications not covered here.

From an implementation perspective, 3 design decisions did most of the work in making this pipeline reliable. First, schema-driven task definition anchored the clinical construct, output format, and evaluation to a single specification, keeping labels, prompts, and metrics aligned with the same objective. Second, strict JSON-only output constraints, with automated extraction of the first valid JSON object, made large-scale reproducible evaluation possible when free-text outputs would not. Third, local deployment provided a privacy-preserving pattern aligned with how health systems would operationalize similar workflows on real clinical data, though in practice it requires on-premises or private-cloud GPU resources, institutional IT and security review, and verification that model licensing terms permit clinical use.

In applied settings, outputs from zero-shot LLM pipelines should be treated as decision-support signals rather than definitive clinical determinations [23]. A positive flag from this pipeline should trigger clinician review of the source note, not an automated action on the patient. No output should serve as the sole or primary basis for clinical decisions until prospective validation on real annotated notes has occurred. Practical deployment should incorporate guardrails such as (1) thresholding ordinal scores to prioritize sensitivity for safety-critical tasks, (2) human-in-the-loop workflows with explicitly defined roles and response protocols (eg, designating an individual as the reviewer, specifying a maximum review time window, defining the communication channel, and requiring documentation of the clinician’s response), (3) audit log requirements, including, at minimum, note IDs, model version, prompt version, timestamps, predictions, and applied thresholds, as well as the note text segment that triggered the extraction and prompt version change-control documentation, and (4) reference to existing clinical decision-support frameworks [24]. Further, the system should implement a fail-safe mode such that if schema validation fails or model confidence falls below the operating threshold, the output should surface as indicative of failed extraction rather than returning a null or default value that a clinician may act upon.

All performance metrics reported in this tutorial were obtained from synthetically generated clinical notes with labels assigned by construction and should be interpreted as controlled workflow-validation results rather than estimates of clinical deployment performance. Although synthetic evaluation is useful for testing schema design, prompting logic, parsing, and error-handling under reproducible conditions, it cannot fully reproduce the linguistic variability, ambiguity, documentation noise, prevalence structure, and institutional heterogeneity of real clinical records, a gap that matters more for behavioral health documentation than for many other note types, given its reliance on narrative, indirect, and negation-heavy language to convey risk-relevant information.

The synthetic notes for the medication nonadherence task specified a single antipsychotic per patient, whereas individuals with serious mental illness are frequently prescribed multiple medications, including more than one antipsychotic. We acknowledge this simplification, which was made to keep the tutorial’s worked example easy to follow rather than to reflect the full complexity of real notes. The underlying workflow does not depend on this simplification: the same process of drafting an initial prompt, refining it against a prompt development set, and then evaluating on a final evaluation dataset applies regardless of how many medications or antipsychotics appear in a note, with the schema and label logic being adjusted accordingly. The reported performance metrics, however, may well depend on it. With a single named antipsychotic per note, the model is never required to determine which agent an adherence statement refers to. In real documentation, adherence may be described generically (“has not been taking his medications”), attached to one agent while others are continued, or implied only indirectly through symptom descriptions, refill gaps, or missed long-acting injectable appointments. Resolving which drug a statement implicates and whether nonadherence to one agent constitutes nonadherence to the antipsychotic regimen is a substantially harder inference than the one evaluated here. Accuracy, precision, and recall may differ in that setting, and a multimedication application would likely require additional pipeline components, such as resolving the medication list prior to classification, which we do not evaluate here.

In practice, several additional pitfalls may arise when applying this approach to real clinical data. First, performance may degrade when encountering out-of-distribution language, including institution-specific abbreviations or documentation styles not reflected in the synthetic data. Second, class prevalence in real-world settings is often highly imbalanced, which can substantially affect precision-recall trade-offs and requires threshold calibration tailored to the deployment context. Third, LLM outputs may exhibit variability across runs or models, necessitating normalization and validation layers to ensure consistent structured outputs. Finally, prompt sensitivity and schema design choices can meaningfully impact extraction performance, highlighting the need for iterative testing and evaluation on representative datasets. For examples of application to real clinical notes, see [25,26].

Aggregate metrics do not capture differential performance across patient subgroups. LLM performance on clinical notes may vary by patient age, race, ethnicity, primary language, provider type, and documentation style, and aggregate metrics may mask disparities. Documentation culture and note type matter as much as demographics; for example, an ED note, a crisis unit note, and an outpatient therapy note may describe the same clinical presentation in structurally different language, and a prompt validated on one note type may underperform on others. Before deployment, teams should stratify evaluation metrics by relevant subgroups (eg, age, primary language, note author role, and clinical setting) and treat acceptable subgroup performance as a prerequisite for deployment, rather than a post hoc audit.

Zero-shot inference can overinterpret ambiguous language, and intermediate Likert scores in particular should be interpreted cautiously [26,27]. In the pediatric SITB cases, misclassifications often arose from subtle or indirect expressions of distress, such as passive death wishes or ambiguous emotional statements [22] that did not explicitly indicate intent or behavior. Similarly, in antipsychotic nonadherence documentation, phrases such as “missed a few doses,” “ran out of refills,” or incomplete medication histories frequently led to intermediate Likert scores, reflecting genuine uncertainty in the narrative. Teams should consider including an “indeterminate” category in the output schema to trigger human review rather than forcing the model to commit to a classification it cannot support.

This is compounded by a critical scope boundary. No clinician ratings were collected to validate these intermediate Likert judgments, and until a validated mapping between the model’s ordinal outputs and established clinical instruments is established through independent clinical rating, Likert scores from the pipeline should not be surfaced to clinicians as severity ratings or used to inform triage or level-of-care decisions. Real-world use requires clinician validation, including mapping to established clinical vocabularies with ordinal categories (eg, Columbia Suicide Severity Rating Scale).

Ensuring programmatic schema validation should be considered mandatory for workflows relying on structured outputs, as strict format compliance enables reliable parsing and evaluation at scale. Local deployment of inference models similarly supports PHI-safe workflows and aligns with institutional auditing and compliance requirements. Any change to a prompt in a deployed system should be treated as a functionally significant system update, not a low-stakes configuration change.

Once reliable extraction of signals such as SITB or antipsychotic medication nonadherence becomes feasible, multiple downstream applications become possible. These applications carry different levels of patient safety risk and must be considered in a staged manner. Lower-risk applications (eg, retrospective quality improvement) are appropriate starting points. Higher-risk applications (eg, real-time triage flagging) should be undertaken only after prospective validation with subgroup-stratified performance and formal institutional governance approvals in place.

We suggest the minimum viable local validation should include (1) a deidentified sample of at least 100 local clinical notes representative of the target documentation type; (2) independent annotation by 2 qualified clinicians using a structured rubric, with interrater reliability reported (κ ≥0.70 recommended); (3) prespecified performance thresholds (eg, sensitivity ≥0.85 for safety-critical SITB flagging) established before evaluation, not after; (4) stratified reporting of performance by relevant patient subgroups; (5) documentation of failure cases and their clinical characteristics; and (6) a defined revalidation trigger when the model version, prompt text, or target note population changes.

We present a PHI-safe, reproducible workflow for extracting clinically relevant information from behavioral health notes using open-source LLMs deployed within institutional computing environments. Across 2 clinical examples, the tutorial shows that schema-driven prompting, structured outputs, and automated validation can support controlled end-to-end workflow testing with minimal task-specific modification. Because evaluation was conducted on synthetic notes with labels assigned by constructions, the reported performance should be interpreted as proof of workflow feasibility under controlled conditions, not as evidence of real-world clinical accuracy. No clinical benefit, efficiency gain, or safety improvement can be claimed without prospective validation on real-world clinical notes. Future work should validate the same pipeline on clinician-annotated real clinical notes across institutions and documentation settings.

Rather than replacing clinical judgment, such workflows can provide scalable decision-support signals that enable safer triage, monitoring, and quality improvement efforts. We provided the synthetic clinical notes generated and used during this study in the Zenodo (European Organization for Nuclear Research [CERN]) repository [28]. By emphasizing structured outputs, automated validation, and local deployment, this tutorial aims to provide a practical foundation that clinicians and health systems can adapt to future clinical NLP tasks as LLM capabilities continue to evolve.