Stratifying the Utility of Transthoracic Echocardiography for Ischemic Stroke Using a Risk Score Open Access

Stroke is one of the most common causes of neurological disability, affecting 795,000 people in the USA annually [1]. Determining ischemic stroke etiology using transthoracic echocardiography (TTE) is often considered to be a standard part of routine stroke care and secondary prevention, of stroke. However, TTEs are not always readily available, can delay hospital discharge, and increase the cost of each hospitalization. An analysis at a Canadian health center found that hospital costs from prolonged stays due to obtaining a TTE increased by an average of USD 555 [2]. There are currently no guidelines from the American Heart Association (AHA) clearly delineating which patients benefit from TTEs [3].

Some studies have attempted to address concerns regarding overutilization of TTE in stroke management, although they do present with some concerns [2, 4‒8]. First, the approaches used in some of these analyses were not able to enough to identify a substantive, low-risk cohort of patients for whom a positive TTE finding was unlikely. For instance, a closer look at the Harris et al. results shows that in the subpopulation with large artery atherosclerosis, 26/114 (23%) still have TTE findings deemed significant by their criteria. It would be difficult to begin having the conversation of deferring an inpatient TTE with these approaches if there is a nearly one-in-four chance of missing a clinically important finding. Second, those studies that could isolate such a cohort were generally limited in what they considered a “positive” TTE finding. For instance, Menon et al. [7] did not include patent foramen ovale, atrial septal aneurysm, or atrial septal defects as significant TTE findings. Other studies that have demonstrated the low utility of inpatient TTEs have also been restrictive and excluded TTE findings such as patent foramen ovale (PFO) [2, 8]. If certain stroke providers would find TTE findings (such as PFO) relevant to ischemic stroke evaluation, they would be less likely to adopt models/frameworks that exclude such findings as being important.

There are many risk scores currently used to guide medical management, such as the CHA2DS2-VASc Score for atrial fibrillation stroke risk and the 2HELPS2B score for seizure detection [9, 10]. The present study aimed to develop a risk score to achieve the following: (1) risk stratifying patients across a spectrum such that there is a cohort of patients where the risk of a positive TTE is low enough where one can begin to consider deferring a TTE, but also such that a higher risk cohort can also be identified to justify obtaining a TTE in the inpatient setting (2) having a broad definition of what constitutes a positive TTE as part of such a model, which can then give a wider array of providers more assurance that particular TTE findings important to their practice are not being missed if the model does indeed suggest the likelihood of a positive TTE is low (3) achieving this risk stratification using a model with variables that can be obtained at time of patient presentation (4) developing a model with high enough performance to ensure appropriate risk stratification (5) having this model be validated on an external data set to ensure reproducibility. While there have been several investigations into the utility of TTEs as mentioned earlier, to the author’s knowledge none of the prior literature on this subject achieves all five of these goals.

IRB approval was obtained for data queried spanning 2 years (2017 and 2018) from the University of Wisconsin Health Comprehensive Stroke Registry. Patients were added to the registry if admitted with a primary diagnosis of transient ischemic attack (TIA) or ischemic stroke. The final dataset only consisted of subjects who received a TTE and consisted of 874 subjects. Twenty-three variables were selected as potential inputs for the model – variables were selected based on prevalence in the registry and considerations of what factors might be associated with cardioembolism. These variables are shown in Table 1 along with descriptive statistics. Fisher’s exact test was used to compare categorical values and calculate p values. Data were chosen from 2017 to 2018 to exclude any confounders from the beginnings of the COVID-19 pandemic and to have a large enough sample size to ensure the events per variable to be greater than 10 for the final model.

TTE reports were reviewed and standardized by institution protocol. As discussed above, a broad definition was used for a positive TTE, as guided by the American Heart Association guidelines and work from Nakanishi et al. [3, 11]. These findings are all independently associated with increased stroke risk and included thrombus (valvular, atrial, or ventricular), dilated cardiomyopathy, left atrial enlargement, ejection fraction ≤40%, patent foramen ovale/atrial septal defect, endocarditis, atrial myxoma, papillary fibroelastoma, aortic atheroma, atrial or left ventricular aneurysm, calcified aortic valve stenosis, and Lambl’s excrescence. All patients underwent an agitated saline study to evaluate for PFO.

The data were trained on a variety of models. Because we wanted to develop a model that was easy to use, logistic regression was initially employed, after which two modifications were made. First, all variables except six were truncated from the model based on coefficient magnitude and risk calibration. Second, coefficients were adjusted to be the same low-term improper fraction for simplicity while ensuring adequate model performance. Further details regarding this modification are shown in online supplementary Table 1 (for all online suppl. material, see https://doi.org/10.1159/000544746). This modified logistic regression model was compared to more standard classifier models: Random Forest Classifier, Naïve Bayes Classifier, Support Vector Machine Classifier, and XG Boost Classifier.

We assessed model performance using a 5-fold cross validation and a mean area under the receiver operator characteristic curve (AUC). To assess model risk calibration, we employed the Brier score, which measures the mean squared difference between the predicted probability and the actual outcome. Further evaluation metrics were used to compare the models including precision, recall and accuracy.

To assess the external validity of the modified regression model, we tested the model on a sample of two-hundred patients from the Indiana University Comprehensive Stroke Registry. While in theory there is not an optimum for the sample size of the external validation set, in this study it was selected to be between 20% and 30% of the training data set sample size, a heuristic standard in machine learning. Subjects were selected sequentially from the registry from the years 2017 and 2018. All of the models applied to the initial data set were applied to the validation set, and permutation testing was used to assess the differences in AUC’s between the training and external validation data set. Comparisons between the external data set and the initial cohort are further demonstrated in online supplementary Table 2.

59% of subjects were found to have a positive TTE, whereas only 1.9% of subjects were found to have a finding of thrombus, atrial mass/myxoma, fibroelastoma, or any vegetation. Further univariate analysis across the other 23 variables is shown in Table 1.

The modified logistic regression model was generated with the following six variables: Atrial fibrillation, large artery atherosclerosis, large vessel occlusion, obesity, antihypertensive medication prescription at time of presentation, and subject age either between 18 and 39 or ≥70. Age cutoffs were determined by categorizing age by decade and then determining which age ranges optimized the coefficient for this feature while maintaining appropriate risk calibration. As a helpful mnemonic, the model was named AL₂OHA. One point was added for each risk factor present.

As outlined in Table 2, the AL₂OHA model was compared to more standard models. The AL₂OHA model had the highest AUC and lowest Brier Score when applied to the training data. After five-fold cross validation, the mean AUC was 0.78 (95% CI, 0.76–0.80), as shown in Figure 1. The mean Brier Score for AL₂OHA was 0.18 (95% CI, 0.16–0.20). The probability of a positive TTE is further demonstrated in Table 3. The probability of positive findings was 6.2% for score of 0, 23.1% for score of 1, 57.4% for score of 2, 85.8% for score of 3, 96.4% for score of 4, and 99.2% for score >5. The model predictions are also compared to the actual incidence of positive TTEs at each point level in Table 3.

To assess for external validity, the AL₂OHA and other classifier models were tested with the validation data set. Figure 2 shows the respective AUCs for all the models. The AL₂OHA model had comparable results and had an AUC of 0.73. Permutation testing was used to generate a distribution of the differences in AUC’s between the training set and external validation set. The calculated p value >0.05, thus failing to reject the null hypothesis that there was no difference in the performance of the AL₂OHA model between the training set and external validation data set.

The AL₂OHA model is a simple tool to risk stratify which patients are likely to have positive findings on TTE, which can help guide decision making about whether a TTE needs to be pursued in the inpatient setting. The model was trained and tested on an internal data set and then validated externally on a data set from another institution.

Importantly, the model’s behavior is intuitive. First, it is monotonic across all scores – as the number of risk factors goes up, the risk of a positive TTE increases. Second, it is readily apparent why most of the variables would correlate with positive TTE findings. For example, atrial fibrillation is associated with left atrial dilatation [12]. The bimodal risk profile for age makes intuitive sense as the likelihood of a cardioembolic etiology for ischemic stroke tends to be higher for younger patients [13]. Patients with age ≥70 (or those who are obese, using medication for hypertension, or have LAA) would be expected to carry some of the same cardiovascular risk factors that would lead to the TTE findings deemed positive in this study [14]. When assessing stroke etiology for patients with large vessel occlusion, cardioembolism has been documented to be the most prevalent cause of stroke [15].

Overall, the model was able to isolate a cohort of patients (AL₂OHA score of 0) for which the chance of a positive TTE was low (6.2%), while simultaneously being very comprehensive in what is considered positive on TTE. These two attributes ultimately distinguish the AL₂OHA model from prior investigations, which either could not identify a truly low-risk cohort or were too exclusive in what constituted a positive TTE. With the AL₂OHA model, no matter what a provider thinks about the utility of particular TTE findings that could be relevant in inpatient ischemic stroke evaluations (such as PFO), an AL₂OHA score of 0 can provide them reassurance that a TTE is unlikely to have a positive findings. Furthermore, for those patients with a higher AL₂OHA score admitted to a hospital in which conducting and interpreting a TTE is limited, the model could also be seen as a more quantifiable way to justify a longer inpatient stay for a TTE. Of note, of the patients with a score of 0 that did have positive TTE findings, none had findings of thrombus, atrial mass/myxoma, fibroelastoma, or any vegetation, which may allay concerns about missing critical findings in these patients.

There were some limitations to this study. First, the data were limited to patients admitted with a primary diagnosis of ischemic stroke or TIA, and so the sample did not include patients with other significant primary complications who developed strokes. Second, the variable of antihypertensive medication at the time of presentation may undermine the model’s validity for patients who may not have their hypertension treated medically. Finally, clinical judgment still has to be reserved while using this model as TTEs are in of themselves imperfect tools in assessing sources of embolus. Even with a low AL₂OHA score, further workup, such as transesophageal echocardiography, may still be warranted if there is a high clinical suspicion of embolic stroke to capture findings less optimally seen on TTE.

For the internal validation study, patient data were obtained under IRB# 2024-0166. The external validation data set was obtained under IRB #2152. IRB’s were obtained under respective Institutional Review Board offices at the University of Wisconsin, Madison and Indiana University which determined that written participant consent was not required. Data were shared between both institutions under Data Transfer and Use Agreement #212075. Informed participant consent was not required as part of both IRB’s and the Data Transfer and Use Agreement.

There are no conflicts of interest from any authors involved in this study.

No funding sources were used for this study.

N.D.J. conceptualized design of study, obtained data under appropriate IRB protocols, arranged data transfer agreement between institutions, generated models, performed and interpreted statistical analysis, and drafted manuscript. H.B. obtained data under appropriate IRB protocols, arranged data transfer agreement between institutions, reviewed and edited manuscript draft. A.M.J. provided clinical insights, reviewed and edited manuscript draft. J.A.S. conceptualized design of study, provided clinical insights, reviewed and edited manuscript draft, provided guidance throughout entirety of project’s execution. A.F.S. conceptualized design of study, supervised machine learning methods applied in study, provided guidance and interpretation for other statistical analyses, reviewed and edited manuscript draft, provided guidance throughout entirety of project’s execution.

Justin A. Sattin and Aaron F. Struck: shared senior authorship.

Corresponding author has access to all of the data in the study and takes responsibility for its integrity and the data analysis. Data cannot be made public per IRB protocols and Data Use Agreements that were used to obtain data.