Computed tomography angiography (CTA) collateral scoring can identify patients most likely to benefit from mechanical thrombectomy and those more likely to have good outcomes and ranges from 0 (no collaterals) to 3 (complete collaterals). In this study, we used a machine learning approach to categorise the degree of collateral flow in 98 patients who were eligible for mechanical thrombectomy and generate an e-CTA collateral score (CTA-CS) for each patient (e-STROKE SUITE, Brainomix Ltd., Oxford, UK). Three experienced neuroradiologists (NRs) independently estimated the CTA-CS, first without and then with knowledge of the e-CTA output, before finally agreeing on a consensus score. Addition of the e-CTA improved the intraclass correlation coefficient (ICC) between NRs from 0.58 (0.46–0.67) to 0.77 (0.66–0.85, p = 0.003). Automated e-CTA, without NR input, agreed with the consensus score in 90% of scans with the remaining 10% within 1 point of the consensus (ICC 0.93, 0.90–0.95). Sensitivity and specificity for identifying favourable collateral flow (collateral score 2–3) were 0.99 (0.93–1.00) and 0.94 (0.70–1.00), respectively. e-CTA correlated with the Alberta Stroke Programme Early CT Score (Spearman correlation 0.46, p < 0.001) highlighting the value of good collateral flow in maintaining tissue viability prior to reperfusion. In conclusion, -e-CTA provides a real-time and fully automated approach to collateral scoring with the potential to improve consistency of image interpretation and to independently quantify collateral scores even without expert rater input.

Mechanical thrombectomy is the mainstay of treatment for patients with acute ischemic stroke due to large vessel occlusion, who fulfil guideline criteria [1]. Despite increasing rates of revascularisation, outcomes are not universally positive and are dependent on clinical and imaging characteristics of patients [2]. Collateral cerebral blood flow is a key determinant of both good clinical outcomes and capacity to benefit from thrombectomy, in both early and late presenting patients [3-7]. Collateral blood flow sustains tissue viability until definitive treatment can be provided, thereby averting infarction of penumbral tissue and improving outcomes [8].

Quantifying collateral cerebral blood flow can be achieved using a variety of scores in a variety of imaging modalities, including computed tomography angiography (CTA), CT perfusion (CTP), and magnetic resonance imaging-based equivalents. CT is the mainstay of acute stroke imaging, with only certain specialist centres using magnetic resonance imaging routinely at the front door. While there has been much interest in CTP to define collateral blood flow, particularly the late time window, CTA is routinely acquired to identify vessel occlusion and available in more centres worldwide. If CTA can provide information on collateral blood flow, it would provide an accessible and routinely acquired biomarker for assessing collateral status.

Several scoring systems exist for quantifying collateral cerebral blood flow on CTA [9-11]. The most commonly used is the CTA collateral score (CTA-CS) also known as the Tan score [6, 11]. The challenge for the implementation of any manual scoring system in both research trials and clinical practice is inconsistency between scorers, making robust patient characterisation less likely. Solutions to inter-rater variability include fully automated computer scoring, or software assistance to improve consistency between individuals. In this study, we evaluated the e-CTA module of the e-STROKE SUITE (Brainomix Ltd.,), a fully automated calculation of the CTA-CS using a machine learning algorithm.

The objective of this study was to validate the e-CTA software. The results of the e-CTA software were compared to a reference standard consensus opinion derived from 3 expert neuroradiologists (NRs). The agreement between individual NRs, both with and without the automated e-CTA reference was quantified, and hence the effect of e-CTA on inter-rater reliability was assessed. Finally, the agreement between e-CTA and automated Alberta stroke programme early CT score (e-ASPECTS; Brainomix Ltd.,) on non-contrast CT was quantified to explore the relationship between collateral status and maintenance of tissue viability.


The Freiburg Ethics commission granted approval for the Study (0171140). Study inclusion criteria were age ≥18 years, anterior circulation ischemic stroke with large vessel occlusion eligible for thrombectomy, and both baseline CT and CTA acquired as part of routine clinical care. Patients were excluded from analysis if there was intracranial haemorrhage on presenting CT imaging, previous territorial infarct or artefacts precluding image analysis as determined by a NR.


Imaging was acquired from 2 centres on multiple CT vendor scanners (Toshiba, General Electric, Siemens, Philips). CTA was acquired by using single-phase acquisition, and processed images had a maximum slice thickness of 1 mm.

Collateral Scoring

CTA-CS within the middle cerebral artery (MCA) territory was used to quantify the collateral blood flow status [11]. The CTA-CS comprises a numerical (ordinal) score ranging from 0 to 3: 0, no collateral filling; 1, ≤50% but >0 of the occluded MCA territory; 2, >50% but <100% of the occluded MCA; 3, 100% collateral supply of the occluded MCA territory. CTA-CS was quantified on the CTA scans independently by 3 experienced NRs (P.P., M.E., M.P.). NRs scored the extent of collaterals manually, blinded to automated e-CTA scoring and clinical information. Following initial evaluation, scorers repeated collateral scores with the e-CTA results available. Finally, a consensus score was derived between NRs (reference standard).

Automated Collateral Scoring

Automated software was used to generate the e-CTA score, a module within the e-STROKE SUITE (Brainomix Ltd., Oxford, UK). e-CTA uses a combination of classical image processing techniques and machine learning classifiers. Input DICOM data are resampled into standard resolution, with gantry tilt correction. Fast proprietary registration is used to align data with a standard coordinate system. Major vessels are segmented and labelled in this standard reference frame using a classification framework. Input arteries and output veins are analyzed to estimate acquisition phase of the CTA, to ensure early or late acquisitions are reported to the user. Vessel density is computed throughout the MCA territory, with a probability map showing absence of vessel contrast. The volume of underenhanced vessels is computed relative to the healthy hemisphere. The volume is summarized into the CTA-CS categories to generate the e-CTA score.

Ischemic Damage

Extent of ischemic damage was assessed by a NR (P.P.) on the presenting noncontrast CT using e-ASPECTS software (Brainomix Ltd., Oxford, UK). e-ASPECTS image-processing algorithm follows an artificial intelligence approach, with a combination of traditional 3D graphics and statistical methods, and machine learning classification techniques. The machine learning classifier identifies early and non-acute signs of ischemia to derive a voxel-wise probability map, which generates the e-ASPECTS [12, 13].

Statistical Analysis

The intraclass correlation coefficient (ICC) quantified agreement between scorers or automated outputs. The percentage agreement of CTA-CS was also quantified. CIs for ICCs were calculated using (bias-corrected and accelerated) bootstrap intervals, through 1,000 bootstrap resamples. To determine sensitivity and specificity to collateral status, CTA-CS were categorised into 2 groups: 0–1 (poor collateral flow) and 2–3 (good collateral flow), as has been used to select patients for clinical trials [6]. To correlate the categorical scales of CTA-CS and ASPECTS Spearman’s correlation coefficient was estimated.

The primary outcome of the study was the agreement between consensus reference standard and fully automated e-CTA quantified using the ICC and percentage of complete agreement. Additional analyses included proportion of scans where the agreement differed by >1 point, and agreements between NR scorers with and without e-CTA assistance. Sensitivity and specificity for determining favourable CTA-CS by the e-CTA were also calculated. Finally, the correlation of e-CTA with e-ASPECTS was calculated.


Ninety-eight consecutive patients with acute ischemic stroke eligible for mechanical thrombectomy were prospectively enrolled into the study and underwent CTA. No patient demographics or clinical data could be collected under the Ethics Protocol which specified evaluation of images only.

Representative CT and CTA images from patients together with the CTA-CS heat map and plain CT infarct volume can be seen in Figure 1. e-CTA score was able to successfully process all data.

Fig. 1.

Example of e-CTA. CTA imaging from a representative patient with a right MCA occlusion (e-ASPECTS and e-CTA overlay on lower row). At presentation, the e-ASPECTS is 10 with minimal ischemic change identified by e-ASPECTS infarct volume detection (red mask), but the collateral flow is poor (orange mask, e-CTA: 0). There is extensive final infarction at 24 h consistent with the paucity of collateral flow on presenting imaging. ASPECTS, Alberta stroke programme early CT score; CTA, computed tomography angiography.

Fig. 1.

Example of e-CTA. CTA imaging from a representative patient with a right MCA occlusion (e-ASPECTS and e-CTA overlay on lower row). At presentation, the e-ASPECTS is 10 with minimal ischemic change identified by e-ASPECTS infarct volume detection (red mask), but the collateral flow is poor (orange mask, e-CTA: 0). There is extensive final infarction at 24 h consistent with the paucity of collateral flow on presenting imaging. ASPECTS, Alberta stroke programme early CT score; CTA, computed tomography angiography.

Close modal

NR Scoring

The distribution of CTA-CSs is shown in Figure 2. NRs 1 and 2 generated a similar distribution of CTA-CS, whereas NR 3 tended to give lower scores. The ICC across all 3 raters was 0.58 (0.46–0.67). Between individual NRs ICC ranged from 0.42 to 0.86 and score agreements from 36.2 to 81.6% (Table 1).

Table 1.

Inter-rater comparison

Inter-rater comparison
Inter-rater comparison
Fig. 2.

Distribution of CTA-CSs. Distribution of CTA-CSs (Tan score) for each neuroradiology rater without (a) and with (b) automated CTA-CS assistance. NR, neuroradiologist 1–3.

Fig. 2.

Distribution of CTA-CSs. Distribution of CTA-CSs (Tan score) for each neuroradiology rater without (a) and with (b) automated CTA-CS assistance. NR, neuroradiologist 1–3.

Close modal

Addition of automated CTA collateral scoring to guide neuroradiological assessment reduced inter-rater variability and improved overall ICC from 0.58 to 0.77 (0.66–0.85), a difference of 0.19 (0.09–0.31, p = 0.003). Agreements between NRs when assisted ranged from 64.3 to 73.5%, with ICCs from 0.70 to 0.80 (Table 1).

Automated Scoring

Automated e-CTA score produced 90% agreement with the consensus NR scores and an ICC of 0.93 (0.90–0.95). For the remaining 10% of fully automated scores where there was no agreement with the consensus opinion the automated score was within 1 point of the consensus (Fig. 3). Sensitivity and specificity for identifying favourable collateral score compared to consensus were 0.99 (0.93–1.00) and 0.94 (0.70–1.00), respectively.

Fig. 3.

e-CTA versus consensus score. Agreement of the automated Tan score (e-CTA) with the NR consensus Tan score.

Fig. 3.

e-CTA versus consensus score. Agreement of the automated Tan score (e-CTA) with the NR consensus Tan score.

Close modal

When compared to individual NRs blinded to the automated output, the automated assessment agreed strongly with 2 individuals (ICC 0.70 and 0.70), but poorer with the third scorer (0.39). When software assisted, all experts had good agreement with the automated score (0.72, 0.84, 0.93).


Consistent laterality of ischemia was positively identified by both e-ASPECTS and CTA-CS in 95% of cases. In these examinations, the Spearman correlation coefficient was 0.46, p < 0.001, between these 2 markers of ischaemia. A similar correlation was seen between e-ASPECTS and e-CTA with a Spearman correlation coefficient is 0.46, p< 0.001.

This study demonstrated a high degree of agreement between the fully automated and objective e-CTA score and a consensus expert CTA-CS. The addition of e-CTA assisted neuroradiological grading of collateral scores and reduced variability between experts. Both the CTA-CS and e-CTA score correlated with e-ASPECTS consistent with the demonstrated deleterious effect of poor collateral blood flow on tissue survival [11].

In this study, e-CTA generated equivalent scores to consensus CTA-CS when used to quantify collateral blood flow. Although current guidelines do not advocate the use of advanced imaging to identify collateral blood flow for patients presenting within 6 h of stroke onset [1], several trials demonstrating the efficacy of early intervention did use collateral blood supply as a selection criterion, identified on either CTP or CTA-CS [6, 7, 14]. Post-hoc analysis of the MR-CLEAN trial, which did not use advanced imaging, confirmed that the treatment effect of thrombectomy was most marked in patients with good collateral status on CTA [5, 15]. For patients presenting beyond 6 h or with an uncertain time of onset, selection using good collateral blood flow has also been shown to identify those who can benefit most from thrombectomy [4], although there is discussion about whether these criteria are overly selective [16].

Widespread identification of patients with favourable imaging on CTP can be challenging as access to CTP is not universal, it increases contrast and radiation doses, and can be more vulnerable to motion artefact [17]. In contrast, CTA is acquired routinely prior to mechanical thrombectomy and is available in most centres. If collateral status on CTA is sufficient to identify patients for revascularisation therapy, this would widen access and streamline treatment pathways for this time critical intervention.

Reliable and accurate scoring is essential if CTA-CS is to be evaluated to select patients for treatment. The 90% agreement of e-CTA with consensus opinion, greater than that seen between NRs, would support the use of an automated e-CTA score to achieve this. Automated collateral scoring with e-CTA negates the demand for real-time neuroradiological input and may help accelerate referral pathways, especially in resource limited settings. This study has demonstrated the high sensitivity and specificity of e-CTA, independent of clinician input, for identifying favourable collateral status. Expert neuroradiology input is rarely available in real time at most hospitals to which stroke patients present, especially at primary stroke centres from which patients need to be transferred to a comprehensive centre. Even when specialist neuroradiology opinion is available, the e-CTA offers the opportunity to improve inter-rater reliability, an essential prerequisite to demonstrating biomarker utility in trials and routine clinical practice.

The correlation of e-CTA score and e-ASPECTS reinforces the pathological significance of collateral blood flow in sustaining tissue viability prior to reperfusion and is consistent with prior work [5, 11]. The use of an automated algorithm offers the potential to generate an objective volume of tissue without collateral blood supply and hence the further refinement of patient selection criteria, in a manner consistent with recent trials using CTP [3, 4]. Further work would be required using paired e-CTA and CTP data to determine the equivalent thresholds for these imaging modalities.

This study has several limitations. The CTA-CS was generated under experimental conditions by specialist NRs without the time pressures of real-world clinical scenarios. In many healthcare settings, ready access to specialist neuroradiology input is not available. As such, the benefit of an automated e-CTA score may be greater than demonstrated here. Although NRs in this study were blinded to clinical data all patients had a confirmed stroke and large vessel occlusion, knowledge of which was available to the expert scorers. The relationship of e-CTA with clinical outcomes and treatment effects is beyond the scope of this study, although this relationship for CTA-CS is well established [5, 11, 15, 18].

e-CTA provides automated and objective measurement of collateral score, which provides similar measures to consensus expert opinion and improves inter-rater reliability of CTA-CS. e-CTA could prove useful for consistent patient selection in clinical trials and translation into clinical practice.

The study was performed in accordance with recommendations guiding physicians in biomedical research involving human subjects adopted by the 18th World Medical Assembly, Helsinki, Finland (1964 and later revisions).

G.H.: is employed part-time by Brainomix. I.Q.G.: has equity interest in and is consultant for Brainomix. All other authors report nothing to disclose.

Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K, et al.; American Heart Association Stroke Council. 2018 Guidelines for the Early Management of Patients With Acute Ischemic Stroke: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association.
. 2018 Mar;49(3):e46–110.
Goyal M, Menon BK, van Zwam WH, Dippel DW, Mitchell PJ, Demchuk AM, et al.; HERMES collaborators. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials.
. 2016 Apr;387(10029):1723–31.
Nogueira RG, Jadhav AP, Haussen DC, Bonafe A, Budzik RF, Bhuva P, et al.; DAWN Trial Investigators. Thrombectomy 6 to 24 Hours after Stroke with a Mismatch between Deficit and Infarct.
N Engl J Med
. 2018 Jan;378(1):11–21.
Albers GW, Marks MP, Kemp S, Christensen S, Tsai JP, Ortega-Gutierrez S, et al.; DEFUSE 3 Investigators. Thrombectomy for Stroke at 6 to 16 Hours with Selection by Perfusion Imaging.
N Engl J Med
. 2018 Feb;378(8):708–18.
Boers AM, Jansen IG, Berkhemer OA, Yoo AJ, Lingsma HF, Slump CH, et al.; MR CLEAN trial investigators. Collateral status and tissue outcome after intra-arterial therapy for patients with acute ischemic stroke.
J Cereb Blood Flow Metab
. 2017 Nov;37(11):3589–98.
Goyal M, Demchuk AM, Menon BK, Eesa M, Rempel JL, Thornton J, et al.; ESCAPE Trial Investigators. Randomized assessment of rapid endovascular treatment of ischemic stroke.
N Engl J Med
. 2015 Mar;372(11):1019–30.
Campbell BC, Mitchell PJ, Kleinig TJ, Dewey HM, Churilov L, Yassi N, et al.; EXTEND-IA Investigators. Endovascular therapy for ischemic stroke with perfusion-imaging selection.
N Engl J Med
. 2015 Mar;372(11):1009–18.
Jung S, Gilgen M, Slotboom J, El-Koussy M, Zubler C, Kiefer C, et al. Factors that determine penumbral tissue loss in acute ischaemic stroke.
. 2013 Dec;136(Pt 12):3554–60.
Maas MB, Lev MH, Ay H, Singhal AB, Greer DM, Smith WS, et al. Collateral vessels on CT angiography predict outcome in acute ischemic stroke.
. 2009 Sep;40(9):3001–5.
Miteff F, Levi CR, Bateman GA, Spratt N, McElduff P, Parsons MW. The independent predictive utility of computed tomography angiographic collateral status in acute ischaemic stroke.
. 2009 Aug;132(Pt 8):2231–8.
Tan IY, Demchuk AM, Hopyan J, Zhang L, Gladstone D, Wong K, et al. CT angiography clot burden score and collateral score: correlation with clinical and radiologic outcomes in acute middle cerebral artery infarct.
AJNR Am J Neuroradiol
. 2009 Mar;30(3):525–31.
Herweh C, Ringleb PA, Rauch G, Gerry S, Behrens L, Möhlenbruch M, et al. Performance of e-ASPECTS software in comparison to that of stroke physicians on assessing CT scans of acute ischemic stroke patients.
Int J Stroke
. 2016 Jun;11(4):438–45.
Nagel S, Sinha D, Day D, Reith W, Chapot R, Papanagiotou P, et al. e-ASPECTS software is non-inferior to neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients.
Int J Stroke
. 2017 Aug;12(6):615–22.
Saver JL, Goyal M, Bonafe A, Diener HC, Levy EI, Pereira VM, et al.; SWIFT PRIME Investigators. Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke.
N Engl J Med
. 2015 Jun;372(24):2285–95.
Berkhemer OA, Jansen IG, Beumer D, Fransen PS, van den Berg LA, Yoo AJ, et al.; MR CLEAN Investigators. Collateral Status on Baseline Computed Tomographic Angiography and Intra-Arterial Treatment Effect in Patients With Proximal Anterior Circulation Stroke.
. 2016 Mar;47(3):768–76.
Fisher M, Goyal M. Variance of Imaging Protocols for Patients With Suspected Acute Ischemic Stroke Because of Large-Vessel Occlusion.
. 2018 Aug;49(8):1805–8.
Menon BK, d’Esterre CD, Qazi EM, Almekhlafi M, Hahn L, Demchuk AM, et al. Multiphase CT Angiography: A New Tool for the Imaging Triage of Patients with Acute Ischemic Stroke.
. 2015 May;275(2):510–20.
Bang OY, Saver JL, Kim SJ, Kim GM, Chung CS, Ovbiagele B, et al. Collateral flow predicts response to endovascular therapy for acute ischemic stroke.
. 2011 Mar;42(3):693–9.
Open Access License / Drug Dosage / Disclaimer
This article is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND). Usage and distribution for commercial purposes as well as any distribution of modified material requires written permission. Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug. Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.