Automated Identification of Stroke Thrombolysis Contraindications from Synthetic Clinical Notes: A Proof-of-Concept Study

Timely stroke thrombolysis is associated with better outcomes in eligible patients [1, 2]. A key component of eligibility determination is the manual review of patients’ previous electronic medical records (EMRs) to screen for thrombolysis contraindications. This task may be time-consuming and prone to errors, potentially leading to treatment delays, inappropriate thrombolysis due to missed contraindications, or withholding of thrombolysis due to wrongfully identified contraindications. Previous studies showed that clinicians spent 2–7 min per patient on EMR review and missed 20–43% of thrombolysis contraindications [3, 4]. Recent advancements in large language models (LLM) represent a significant breakthrough in artificial intelligence and machine learning [5]. Trained on extensive textual datasets, LLMs exhibit strong capabilities in natural language processing and general reasoning, with medical applications such as diagnostic reasoning, treatment recommendations, knowledge augmentation, and clinical data summarization [6‒11]. LLMs may offer a novel opportunity to address the gap in acute stroke care related to the timely and accurate identification of thrombolysis contraindications.

This study aimed to develop a proof-of-concept LLM-based tool for the identification of stroke thrombolysis contraindications within clinical notes and test its preliminary performances using synthetic clinical notes. This tool could be used for real-time processing of a patient’s previous EMR clinical notes and to quickly generate a list of thrombolysis contraindications to enhance the accuracy and timeliness of stroke thrombolysis decision-making.

Synthetic clinical notes were generated to evaluate the LLM-based tool’s performance in identifying stroke thrombolysis contraindications (online suppl. Fig. S1; for all online suppl. material, see https://doi.org/10.1159/000545317). As no real patient data were used, IRB approval was unnecessary. Data and prompts are available upon request. This study complies with the Standards for Reporting Diagnostic Accuracy Studies (STARD) guidelines [12].

In order to mimic typical clinical notes within patients’ EMR prior to acute stroke evaluations (e.g., outpatient visit notes, operative reports, admission notes, progress notes, discharge summaries), we generated 150 synthetic clinical notes using five LLMs under default settings: ChatGPT-4o (OpenAI), Llama 3.1 70B and 405B (Meta), ChatGPT-4o mini (OpenAI), and Gemini 1.5 Flash (Google). The use of five different LLMs served to provide stylistic variety to the synthetic notes. These LLMs were stratified into three tiers based on parameter size: tier 1 (ChatGPT-4o, Llama 3.1 405B), tier 2 (Llama 3.1 70B), and tier 3 (ChatGPT-4o mini, Gemini 1.5 Flash). To emphasize higher quality notes from larger models, tier 1 generated 60 notes, tier 2 produced 50, and tier 3 contributed 40, with generation quotas predetermined prior to study initiation. A standardized prompt and a random number generator were employed to guide each LLM in generating random synthetic notes (online suppl. Table S1). The prompt specified predetermined variables, including patient age (51–100 years), sex, number of contraindications (0–3), and their nature, if applicable, to which the number generator assigned a random value. The encounter date was uniformly set one day before contraindication identification, ensuring consistency across generated notes. Each synthetic note was reviewed by a vascular neurologist to identify potential contraindications from a predefined list of 24 thrombolysis contraindications based on the American Heart Association stroke guidelines [13] (online suppl. Table S1). A contraindication was considered present if the documentation necessitated clarification or confirmation with the patient before thrombolysis administration. Identified contraindications were compared with the predefined set specified in the prompt, accounting for potential discrepancies. A final curated list of contraindications for each note was established as the ground truth for performance evaluation.

We developed an LLM-based tool, named NeuroGlimpse, for identifying stroke thrombolysis contraindications from clinical notes. This tool was powered by Llama 3.1 405B, an open-source LLM with performance comparable to ChatGPT-4o [14], in its default settings (temperature 0.2, top P 0.7, frequency and presence penalties at 0, token limit of 1,024). This LLM was accessed via an Nvidia cloud platform. A custom prompt, refined iteratively and exceeding 3,000 words, provided detailed instructions, output formats, and thrombolysis-specific clinical knowledge (online suppl. Table S2). The input consisted of this prompt and a synthetic note, and the output was a single list of thrombolysis contraindications, including the reason for contraindication, its classification (absolute or relative), pertinent clinical information, and the date and author of the source note (online suppl. Table S3). No modification to the original LLM or preprocessing of inputs or outputs was needed.

We evaluated the performance of NeuroGlimpse using the previously established ground truth. Performance evaluation included (1) number of correctly identified contraindications (true positives); (2) number of missed contraindications (false negatives); and (3) number of false positives (FPs). FPs consisted of irrelevant contraindications (items not present in the ground truth), repetitive clinical information (multiple items describing the same contraindication), and hallucinations [5] (factually incorrect or fabricated clinical information). True negatives (TNs) were contraindications not present in a clinical note and not reported by NeuroGlimpse. Given that each clinical note could contain up to 24 contraindications, a note without any contraindications was counted as 24 TNs. The primary performance measures were sensitivity and positive predictive value (PPV). Secondary measures included specificity, negative predictive value (NPV), accuracy, and F1 score.

Sensitivity, specificity, PPV, NPV, accuracy, and F1 score were calculated using standard definitions based on true positive, FP, false negative, and TN. Clopper-Pearson exact methods were employed to calculate 95% confidence intervals (CIs) for sensitivity, specificity, PPV, NPV, and accuracy. Two-sample tests for comparison of binomial proportions were utilized to compare primary performance measures across subgroups defined by sex and the number of contraindications per note. Chi-square tests were applied to evaluate differences in primary performance measures among the 24 contraindications, under the null hypothesis of equal sensitivity and PPV across all contraindications. All statistical tests were two-tailed, and a significance level of α = 0.05 was used throughout. Statistical analyses were performed using Microsoft Excel version 16.90.

A total of 150 synthetic clinical notes were generated using various LLMs (Table 1). Specifically, 40 notes (26.7%) were produced by ChatGPT-4o, 20 (13.3%) by Llama 3.1 405B, 50 (33.3%) by Llama 3.1 70B, 20 (13.3%) by ChatGPT-4o mini, and 20 (13.3%) by Gemini 1.5 Flash. Each note averaged 241.6 words (SD 110.7; range 80–549) and included an average of 1.5 contraindications (SD 1.1; range 0–5). Among these notes, 89 (59.3%) contained 0–1 contraindications, while 61 (40.7%) included 2–5 contraindications. The synthetic patient cohort had a mean age of 73.3 years (SD 14.7; range 51–100), with 72 patients (48.0%) being female.

Contraindications were randomly assigned during the synthetic note generation process. However, the predetermined set of contraindications specified in the prompt may have differed from manually established ground truth, resulting in an uneven distribution across the notes (Table 2). The most frequently appearing contraindications were major surgery within the prior 14 days (n = 31), gastrointestinal bleeding within the prior 21 days (n = 19), and recent or current use of warfarin (n = 17). The least frequent contraindications included recent or ongoing internal bleeding (n = 2), serious trauma within the prior 14 days (n = 2), gastrointestinal malignancy (n = 3), and intra-axial intracranial neoplasm (n = 3). In total, 219 contraindications were incorporated within these notes.

When evaluating all 219 contraindications collectively, NeuroGlimpse demonstrated strong performance metrics (Table 3): sensitivity of 90.9% (95% CI, 86.3–94.3%), specificity of 99.2% (95% CI, 98.8–99.5%), PPV of 87.7% (95% CI, 82.7–91.7%), NPV of 99.4% (95% CI, 99.1–99.6%), accuracy of 98.7% (95% CI, 98.2–99.0%), and an F1 score of 0.892. The primary reason for FPs was the identification of irrelevant contraindications (n = 24, 86%), such as interpreting documentation like “no clear evidence of malignancy but advanced age and multiple comorbidities may suggest limited life expectancy” as contraindications. The remaining FPs were due to repetitive clinical information (n = 4, 14%). No instances of hallucinations were observed in any of the NeuroGlimpse outputs (Table 4).

Sensitivity and PPV varied numerically across different contraindications (Table 2). Sensitivity ranged from 33.3% for intra-axial intracranial neoplasm to 100.0% for multiple contraindications, while PPV ranged from 50.0% for serious trauma within the prior 14 days to 100.0% for multiple contraindications. Chi-squared tests, assuming equal sensitivity and PPV for all contraindications, revealed no significant variation in sensitivity (p = 1.00) but a significant variation in PPV (p < 0.0001), with systematic malignancy (n = 11) representing 39.3% of all FPs.

Comparing performance between male and female patients showed no significant differences (online suppl. Table S4). For male patients, the sensitivity was 87.3% (95% CI, 79.6–92.9%) and PPV was 87.3% (95% CI, 79.6–92.9%). For female patients, the sensitivity was 94.5% (95% CI, 88.4–98.0%) and PPV was 88.0% (95% CI, 80.7–93.3%) (p = 0.06 for sensitivity; p = 0.86 for PPV).

Finally, comparing notes with 0–1 contraindication per note to those with 2–5 contraindications revealed no significant differences in performance (online suppl. Table S5). Notes with 0–1 contraindication had a sensitivity of 96.8% (95% CI, 89.0–99.6%) and PPV of 85.9% (95% CI, 75.6–93.0%), while notes with 2–5 contraindications had a sensitivity of 88.5% (95% CI, 82.4–93.0%) and PPV of 88.5% (95% CI, 82.4–93.0%) (p = 0.052 for sensitivity; p = 0.59 for PPV).

In this proof-of-concept study, we generated 150 synthetic notes mimicking real clinical notes within patients’ EMR prior to acute stroke evaluations. We also developed NeuroGlimpse, an LLM-based tool equipped with a custom prompt, for identifying stroke thrombolysis contraindications within these synthetic notes. NeuroGlimpse showed a sensitivity of 91% and PPV of 88%, with no evidence of sex bias. These results highlight the potential of this tool for identifying thrombolysis contraindications within real EMR clinical notes. This could be integrated into acute stroke workflows with real-time processing of patients’ most recent EMR clinical notes prior to acute stroke evaluations in order to quickly and accurately identify potential contraindications.

Safe and timely thrombolysis in ischemic stroke is critical, as expedited treatment improves functional outcomes [1, 2], while accurate identification of contraindications minimizes hemorrhagic risk [15‒17]. Clinicians reportedly spend 2 to 7 min reviewing EMR for contraindications, with miss rates ranging from 20% to 43%, though these estimates are based on simulated scenarios rather than real-world data [3, 4]. To enhance safety and reduce door-to-needle times, clinical decision support tools have emerged. The first, developed in 2015, relied on structured EMR data (e.g., diagnoses, medications) but was limited by data inaccuracies compared to unstructured clinical notes [3, 18]. Subsequent tools introduced in 2018 and 2023 incorporated keyword searches within clinical notes to improve information capture [4, 19]. However, these methods required manual review and were hindered by issues such as abbreviation, misspelling, and the inability to interpret contextual meanings in notes.

This proof-of-concept study showed the potential of an LLM to identify thrombolysis contraindications within EMR clinical notes. Future iterations of NeuroGlimpse (online suppl. Fig. S2) will integrate real-time EMR access. Upon activation of a stroke alert, this tool will extract the patient’s unique identifiers from paging information, retrieve the most recent EMR clinical notes prior to acute stroke evaluation for the corresponding patient, and analyze them using Llama 3.1 to generate a list of contraindications within 30 s. This list will be delivered to the clinician through mobile and desktop interfaces. By facilitating rapid and accurate identification of thrombolysis contraindications, this tool could enable faster and safer thrombolysis decision-making, reduce door-to-needle times, lower the risk of post-thrombolysis hemorrhagic complications, and increase the rate of stroke thrombolysis. These potential benefits may be even greater in rural hospitals, where lack of stroke expertise may have contributed to rural hospitals’ slower thrombolysis and lower rates of thrombolysis compared to urban hospitals [20].

NeuroGlimpse’s initial development utilized synthetic clinical notes to enable rapid generation and diverse content coverage, facilitating comprehensive prototype evaluation across varied clinical scenarios. This approach expedited preliminary validation, aligning with the rapid evolution of LLMs. Subsequent phases will incorporate retrospective analyses of real EMR data, followed by prospective randomized trials to assess NeuroGlimpse’s real-time utility in acute stroke care.

This study has several limitations. First, NeuroGlimpse was not evaluated using real patient clinical notes, potentially affecting performance metrics and limiting generalizability. For example, most of the contents within synthetic notes pertained to the past medical history and history of present illness, which may be overrepresented compared to real clinical notes. Future validation will involve a retrospective study with real-world data. Second, the model analyzed single clinical notes rather than sets, which may reduce sensitivity due to overlapping information but could also lower precision due to increased FPs. Third, the study did not demonstrate automated real-time EMR integration, a critical feature for clinical utility. Fourth, the comprehensiveness of outputs was not assessed, potentially impacting usability. Fifth, all synthetic notes were uniformly dated one day prior to contraindication identification, limiting assessment of the model’s ability to process older, less relevant information. Sixth, our current LLM-based tool was designed to only analyze past medical records for a given patient. It would not have the ability to capture live data from the acute stroke evaluation, such as blood pressure and blood glucose obtained at the time of stroke alert, due to technical difficulties in accessing very recent data (less than 24 h) from EMR. Lastly, only one vascular neurologist reviewed synthetic notes to establish the ground truth, introducing potential bias due to the lack of consensus-based validation.

We developed an LLM-based tool that showed high sensitivity and PPV in identifying stroke thrombolysis contraindications within synthetic clinical notes in this proof-of-concept study. This tool could eventually process a patient’s most recent EMR clinical notes in real time in order to inform the clinician of the presence of any thrombolysis contraindication rapidly and accurately, enabling faster and safer thrombolysis decision-making.

This study did not involve real patient data; instead, synthetic clinical notes were utilized to evaluate the performance of the developed tool. As no human participants were involved, the study was exempt from requiring Institutional Review Board (IRB) approval. The generated synthetic data were reviewed internally to ensure alignment with ethical research practices. No identifiable information was included, and the study adhered to the STARD guidelines for reporting diagnostic accuracy studies.

The authors declare no conflicts of interest regarding the publication of this manuscript. The study did not receive any specific funding, and no financial or nonfinancial relationships exist that could be perceived as a potential conflict of interest in the writing of this manuscript.

No funding was received for this study. The study was conducted independently without any financial support from external sponsors. The authors affirm that the study design, data collection, analysis, manuscript writing, and decision to publish were carried out without any involvement or influence from external funders.

Conceptualization, methodology, and formal analysis: Bing Yu Chen and Fares Antaki. Investigation, data curation, writing – original draft preparation, visualization, and project administration: Bing Yu Chen. Writing – review and editing: Fares Antaki, Marco Gonzalez Castellon, Ken Uchino, Samer Albahra, Scott Robertson, Sidonie Ibrikji, Eric Aube, Andrew Russman, and M. Shazam Hussain. Supervision: M. Shazam Hussain and Andrew N. Russman. All authors have read and approved the final version of the manuscript.

The data supporting this study consist of synthetic clinical notes generated for research purposes. These synthetic data are not based on real patient information and are available upon reasonable request. Due to their synthetic nature, there are no legal or ethical restrictions on sharing the data. Researchers interested in accessing the data and prompts used in this study may contact the corresponding author, Dr. Bing Yu Chen, at [email protected]. The data that support the findings of this study are not publicly available due to privacy reasons but are available from the corresponding author upon reasonable request.