Signal-Guided Multitask Learning for Myocardial Infarction Classification Using Images of Electrocardiogram

Owing to recent advancements in artificial intelligence (AI), significant opportunities and challenges emerge for clinical application in medicine. AI has been explored across medical applications, including but not limited to diagnosis, planning treatment, and risk stratifications [1‒6]. Electrocardiograms (ECGs), which are low-cost and simple diagnostic tools used to identify a range of cardiac diseases, are a popular area where AI has potential for clinical applications. Recent studies have demonstrated AI’s performance in detecting arrhythmia [7], predicting atrial fibrillation [8], identifying left ventricular dysfunction [9], and predicting heart failure [10].

In the emergency department, it is a pivotal tool for detecting acute coronary syndrome (ACS). Nonetheless, the sensitivity and specificity of a 12-lead ECG for ACS are limited owing to factors such as the lack of posterior leads, variations in coronary anatomy among individuals, collateral circulation, and the degree of infarction transmurality [11, 12]. Moreover, the proficiency of physicians in interpreting ECGs falls short of expectations [13]. An immediate reperfusion therapy is crucial in preserving the myocardium and enhancing patients’ survival in patients with ST-segment elevation myocardial infarction (STEMI) [14]. The ECG criteria for STEMI are well established for the timely management of patients. However, a considerable disparity exists among physicians in ECG interpretation that displays features indicative of STEMI. Furthermore, the majority of patients with ACS do not exhibit ST-segment elevation (NST-ACS), necessitating a longer biomarker and follow-up testing to arrive at a diagnosis.

In clinical settings, a substantial amount of standard 12-lead ECG data is commonly presented in image format. Raw data evaluation is frequently constrained by technical issues in a variety of situations, enhancing the potential for developing AI algorithms according to these images with broader applicability. We hypothesized that an AI algorithm based on deep learning (DL) could differentiate patients experiencing acute myocardial infarction (MI) from those without coronary artery disease using image-based ECG data. Furthermore, we investigated whether the algorithm can assist physicians in identifying patients with acute MI.

ECGs of patients who visited the emergency department for acute MI and those without coronary syndrome between January 2011 and December 2019 were collated. Patients with STEMI and non-ST-segment MI (NSTEMI) were defined as the MI group. ECGs of patients without coronary syndrome were randomly collected as the control group. MI diagnosis was adjudicated according to the laboratory findings, records of coronary intervention, echocardiographic data from electronic health records of the patients, and confirmation by cardiologists.

A standard supine 12-lead ECG of 25 mm/s and 10 mm/mV was used. We selected the first ECG for patients with MI when they visited the emergency department for the MI ECG. For the control group, we randomly selected patients who did not have a medical history of coronary artery disease. For the baseline ECG data, the Philips ECG system (Philips Medical System, PageWriter TC30, TC70) was used. The dataset is composed of ECG images, among which the ECG area consists of grid-like background and signal parts, as shown in Figure 1a. The signal part of the ECG area is made up of thin lines and exists sparsely. A total of 3,003 ECG images from patients with MI and 8,224 ECG images recorded in the emergency department from individuals without ischemic heart disease were collected between January 2011 and December 2019 at Kyungpook National University Hospital. Those without ischemic heart disease were defined as patients who had never been diagnosed with ischemic heart disease in the past and whose index visit was ultimately determined not to be due to ischemic heart disease.

The ECG image displays 12-lead signals on a grid-patterned background, as shown in Figure 1a. Due to the sparsity of the ECG signal parts, several studies have attempted to use the signal parts only as input of the DL algorithm instead of the ECG image itself to achieve good classification performance by extracting ECG signal parts [15]. However, only signal parts of the ECG image remained (refer it to signal image), and all the background information from the ECG image was lost, as shown in Figure 1b, and it can cause performance degradation of the DL algorithms. Thus, in this paper, we want to determine how best to use the signal images.

An ECG-digitizing open-source tool was initially applied to extract signal images from ECG images [16]. Straight background grids not removed afterward were eliminated using Hough transform, which is an image-processing technique, and the remaining noise was removed by grouping connected pixels. Figure 1 illustrates the process of extracting the processed image (signal image) from the ECG image.

In this study, four different methods were used to evaluate the performance depending on how the signal image was used. First, we used the ECG image itself as the input of the DL algorithm without the signal image. Second, the signal image was utilized without the ECG image. Third, the ECG and signal images were concatenated to emphasize the signal parts from the ECG image. Lastly, we propose to generate signal data using a DL segmentation method and guide it to highlight signal parts from the ECG image. The structure of the model using the three methods is shown in Figure 2a–d.

The proposed model comprises a segmentation module for extracting signals and a classification module for emphasizing the extracted signals to classify MI. The segmentation module is designed to automatically extract signal parts from the ECG image. The extracted signal is concatenated with the ECG input image, further highlighting these signals at the pixel level. We employed U-Net, which is known for its effective performance in segmenting biomedical images, for segmentation [17]. For classification, we used SE-ResNet, which enhances feature representation quality by highlighting more on crucial channels within learned features [18]. We fine-tuned SE-ResNet using pretrained models from the ImageNet dataset. The model input was down sampled to 384 × 384 × 3 dimensions, and data augmentation techniques such as resizing, random horizontal flipping, random brightness adjustments, and normalization were employed during training to prevent overfitting. For loss functions in model training, we used a combination of dice loss [19] and cross-entropy loss [20] for the segmentation module while only employing cross-entropy loss for the classification module. Signal images extracted through image processing served as the ground truth for dice loss calculation.

Additionally, to verify the efficacy of our proposed model, we conducted comparative experiments with various DL models. Initially, we performed experiments comparing a range of convolutional neural network-based models (ResNet [21], Wide ResNet [22], ResNeXt [23], and SE-ResNet [18]) using only ECG images as input. Subsequently, we sought to determine the impact of incorporating signal images into the input by conducting experiments using only signal images or by concatenating signal images with ECG images. The DL algorithm underwent a fivefold cross-validation, where the data were divided into five subsets. Four of these subsets were used for training, whereas one subset was reserved for validation in an alternating fashion. The model was evaluated 5 times, with each evaluation using a different validation subset. The final performance was determined by averaging the results of the five evaluations.

We examined the DL algorithm’s ability to assist physicians in distinguishing between ECGs indicating MI and those that do not to evaluate the capability of the algorithm to support medical professionals in making decisions. In this test, residents in training in internal medicine (IM) and IM specialists were included. Fifty randomly selected ECGs were presented to both the IM trainees and specialists who were then asked to determine whether each ECG indicated MI or not. Following a 2-week washout period, we presented the identical collection of ECGs to the participants with the interpretation of the DL algorithm included. We then evaluated the extent to which their judgments differed from the previous one.

In the comparative test among DL models, performance indicators for classification included sensitivity, specificity, precision, accuracy, F1 score, area under the precision-recall curve (AUPRC), and area under the receiver operating characteristic curve (AUROC). Sensitivity quantifies the proportion of true-positive cases correctly identified by the model, whereas specificity measures the proportion of true-negative cases accurately identified. Precision gauges the accuracy of positive predictions and provides a comprehensive assessment of the model’s performance. The F1 score combines precision and recall into a single metric. The AUROC, which represents the area under the curve delineated by the true positive rate and false positive rate, serves as a comprehensive metric for evaluating the performance of a model across a variety of thresholds. AUPRC is another metric that evaluates the overall precision at different levels of recall. It is calculated by determining the precision at various recall thresholds and then averaging these values. Since sensitivity, specificity, precision, accuracy, and F1 scores vary depending on threshold selection, AUROC and AUPRC – metrics less influenced by threshold – were deemed more critical indicators for evaluation [24].

In the test of the clinical availability of the DL algorithm, physicians’ decisions in detecting the ECG of MI without and with DL interpretation were collected. Data normality was evaluated using the Shapiro-Wilk test. The comparison before and after the introduction of the DL algorithm, including the comparison between physician groups, used accuracy as the metric. Paired-samples Wilcoxon test was used to compare the accuracy.

A comparison of the DL algorithms for classifying MI is illustrated in Table 1. The proposed model showed superior performance compared with other models, with an accuracy of 90.56%, sensitivity of 83.82%, specificity of 93.02%, precision of 81.44%, and F1 score of 82.61%. In particular, the AUC and AP, which are crucial evaluation indicators, recorded the best performance with values of 0.859 and 0.897, respectively. The model, leveraging DL to rectify information loss during signal image extraction and using the corrected information to guide ECG image interpretation, exhibited superior performance compared with single-task algorithms (Table 1).

A total of 51 physicians (36 trainees of IM and 15 IM specialists) were enrolled in the clinical availability test of the DL algorithm. In all participants, the median accuracy of ECG interpretation was 62%, which was significantly improved to 68% with the aid of the DL algorithm (Fig. 3a). We compared the accuracy of interpretations between IM trainees and specialists, revealing a remarkably higher accuracy among IM specialists before the aid of the DL algorithm (60% vs. 72%, p = 0.004). In both IM trainees and specialists, significant increases were observed in the accuracy of ECG interpretation after the implementation of the DL algorithm (60% vs. 66% [p = 0.040] for trainees, 72% vs. 80% [p = 0.001] for specialists) (Fig. 3b). The use of DL algorithms not only benefited the trainees but also the experienced IM specialists.

An analysis was undertaken to stratify ECGs into three subgroups: STEMI, NSTEMI, and non-MI. The aim was to evaluate the accuracy of the ECG interpretation within these delineated groups, although physicians were not surveyed for differentiating between STEMI and NSTEMI in this study. Among the test data, 8.0%, 18.7%, and 73.3% were STEMI, NSTEMI, and non-MI. The difference in the physicians’ accuracy of ECG interpretation between STEMI and NSTEMI was quite significant. The accuracy in NSTEMI was considerably lower than that in STEMI (the median accuracy of 33% vs. 80%, p < 0.001). Conversely, the accuracy of DL interpretation in STEMI and NSTEMI was 97% and 79%, respectively, with a difference not as substantial as that observed among physicians. While the ECG interpretation accuracy of physicians in STEMI was indeed at a very high level compared with that in NSTEMI, the DL algorithm continued to enhance interpretative capabilities not only in NSTEMI but also in STEMI (33–43% for NSTEMI [p = 0.010], 80–86% for STEMI [p = 0.009]) (Fig. 4a).

An all subgroups, ECG interpretation accuracy was higher among IM specialists than among trainees including STEMI, NSTEMI, and the non-MI group (Fig. 4b). As evident from the markedly lower accuracy in NSTEMI, distinguishing NSTEMI poses a considerably greater challenge for both trainees and specialists than STEMI (the median accuracy in NSTEMI vs. STEMI: 31% vs. 80%, p < 0.001 for trainees; 50% vs. 73%, p = 0.011 for specialist). Also, the DL algorithm revealed a high accuracy not only in STEMI (97%) but also in NSTEMI (79%) compared with that of physicians. The DL algorithm has proven beneficial not only for trainees but also for specialists, demonstrating significant improvement in the accuracy of ECG interpretation, particularly in patients with non-MI (without and with DL algorithm; 84% vs. 95%, p = 0.004) and STEMI (73% vs. 86%, p = 0.032).

The main findings of the present study were as follows: (1) signal-guided multitask DL algorithm demonstrated superior performance compared with previous single-task algorithm; (2) the DL algorithm supports the physicians’ decision discriminating MI ECGs from non-MI ECGs; (3) the accuracy of physicians’ interpretation in NSTEMI was significantly lower compared with that in STEMI; (4) the DL algorithm could assist the physicians clinical judgment, regardless of STEMI or NSTEMI; (5) and specifically, the DL algorithm showed a notably higher level of accuracy in NSTEMI compared with physicians. Limitations were observed in the accuracy of interpreting ECGs to discriminate MI among physicians [12, 25]. Despite the proper training and experience of healthcare professionals, the subtle and complex patterns indicative of MI can sometimes be challenging to identify accurately. Given these obstacles, there is a growing recognition of the potential for AI to considerably improve ECG interpretation in emergency settings. Leveraging advanced ML algorithms, AI systems can rapidly analyze vast amounts of ECG data, identifying nuanced patterns and anomalies that may escape human detection. Integrating AI into the diagnostic process holds promise for improving the speed and accuracy of MI detection, ultimately enhancing the decision-making process in emergency care.

To create the optimal DL model, we tested the performance of several existing models. In the SE-ResNet models [18], which were tested for developing the DL algorithm in the present study, a performance decline was observed when only signal images were used as input data or when both signal images and ECG images were used concurrently compared to solely using ECG images as input data. The cause of the decline in the performance when using signal images as input data may be unnecessary noise or signal loss due to imperfections in signal extraction. When an ECG image and a signal image are used together as input, using the signal image to guide the ECG image rather than using only the signal image may be more optimal. However, due to information loss in signal image extraction, which is not perfect, the performance could be worse than using the ECG image only. Thus, the proposed model is believed to be effective in improving performance by correcting information loss using DL in signal image extraction and allowing the corrected information to guide the ECG image. The proposed model presented superior performance compared with the other single-task models.

Some early studies using AI to identify patients with MI by ECG reported the performance of the AI model with a very high accuracy of 96% [26]. This research has revealed the potential clinical applicability of AI in interpreting the ECG. However, the studies conducted have been relatively limited, involving some participants, including only 75 cases of MI. Another early study that applied artificial neural networks to detect STEMI in ECG showed that AI interpretation was better than experienced cardiologists in detecting STEMI in ECG with higher sensitivity and discriminant power [27]. The subsequent studies consistently demonstrated a high accuracy in AI ECG interpretation. Recently, Kavak et al. [15] reported 96.3% accuracy, 96.2% sensitivity, and 89.4% specificity when using a convolutional neural network-based model to interpret STEMI ECGs. Al-Zaiti et al. [28] investigated an ML-based algorithm for predicting acute myocardial ischemia, including STEMI and NST-ACS, in individuals with chest pain. Their findings indicated that the ML model revealed a 52% improvement in sensitivity compared with commercial software and a 37% enhancement compared with experienced clinicians. They also elucidated that the predictive power of the ML model is lower in NST-ACS than in STEMI, indicating that distinguishing NST-ACS is not easier than distinguishing STEMI.

In this study, ECG interpretation accuracy using the DL algorithm was significantly higher, reaching 90%, compared with the physicians’ accuracy of 60%. In the second test, despite the support of the DL algorithm, physicians failed to enhance their accuracy to the level achieved by the AI. This is attributed to the fact that physicians make decisions by considering their own insights, and as a result, the improvement in interpretation accuracy was only modest. This may be partly attributed to the low confidence that physicians have in AI. Nevertheless, the DL algorithm has proven beneficial not only for trainees but also for experienced IM specialists, enhancing their interpretations. Hence, it could be helpful even for physicians with a certain level of experience.

Compared with STEMI, distinguishing NSTEMI based solely on the initial ECG is challenging in clinical practice because of the less pronounced ECG changes. This difficulty in differentiation was evident in our study as well, where physicians demonstrated a significantly lower accuracy in identifying NSTEMI. While the interpretation of the DL model also exhibited decreased accuracy in NSTEMI, it showed a significantly higher accuracy compared with physician interpretation. The use of the DL interpretation contributed to an improved accuracy in NSTEMI, suggesting that its application may be most beneficial in the clinical diagnosis of NSTEMI.

First, this study developed a DL model using data collected from a single center. Although strict validation was performed using separate training and validation sets, further validation under different conditions may be warranted. Nevertheless, the study employed ECG machines commonly used in current clinical practice and used digitized ECG images rather than raw data, enhancing its generalizability. Second, the accuracy in the interpretations by physicians aided by the DL algorithm did not show a dramatic improvement compared with the accuracy of the DL model in ECG interpretation. This could be attributed to the current lack of high confidence among physicians in relying on AI for ECG analysis. Third, when evaluating whether the assistance of DL algorithms enhances the accuracy in ECG interpretation by physicians, the study did not include cardiologists; therefore, it did not evaluate the improvement in a group with extensive experience in ECG interpretation. However, the test, which involved several IM specialists, demonstrated enhancement in interpretation accuracy, suggesting considerable potential for clinical utility. Lastly, this study focuses on ECG features only. We are clinically well aware of the limitations of relying solely on ECG to determine MI that requires emergency intervention. In actual clinical practice, there are far more diverse scenarios. Therefore, it is necessary to improve accuracy in the future by including symptoms and other clinical presentations in the research.

This study revealed that the signal-guided multitask DL algorithm exhibited better performance than the previous single-task algorithm, and this algorithm supported physicians in distinguishing MI ECGs from non-MI ECGs. While physicians exhibited significantly lower accuracy in interpreting ECGs for NSTEMI compared with STEMI, the accuracy of the DL algorithm was notably higher in NSTEMI than that of physicians. This aspect could be the most beneficial in clinical applications.

Prof. Se Yong Jang, the corresponding author, confirms that Prof. Dong Heon Yang and Prof. Hun Sik Park contributed to the paper, had the opportunity to review the final version prior to publication, and guarantees their co-authorship status. He also affirms the accuracy of the author contribution and conflict of interest statements, despite Prof. Yang and Prof. Park being unavailable to confirm co-authorship.

The present study was approved by the International Review Board (IRB) at Kyungpook National University Hospital (KNUH IRB No. 2020-12-022-011). Informed consent was waived by the IRB because of the retrospective nature and characteristics of the current study. The study complied with the guidelines of the 2013 Declaration of Helsinki and Good Clinical Practices.

The authors have no conflict of interest to declare.

This research is funded by biomedical research institute of Kyungpook National University Hospital.

Bo Eun Park and Byungeun Shon participated in writing the first draft of this manuscript and was involved in subsequent revisions. Jungrae Cho and Byungeun Shon participated in the analysis of this study. Min-Su Jung, Jong Sung Park, Myeong Seop Kim, Eunkyu Lee, Hyohun Choi, Hyuk Kyoon Park, Yoon Jung Park, Hong Nyun Kim, and Nam Kyun Kim participated in data collection. Myung Hwan Bae, Jang Hoon Lee, Hun Sik Park, Yongkeun Cho, and Dong Heon Yang supervised the design and analysis of the current study. Sungmoon Jeong and Se Yong Jang designed the overall process of this study and supervised its execution throughout.

Drs. Park and Shon are co-first authors.Drs. Jang and Jeong are co-corresponding authors.

The data supporting the findings of this study are not publicly accessible due to privacy concerns. Further inquiries can be directed to the corresponding author.