Abstract
BACKGROUND AND PURPOSE: Automated ASPECTS has the potential of reducing interobserver variability in the determination of early ischemic changes. We aimed to assess the performance of an automated ASPECTS software against the assessment of a neuroradiologist in a comparative analysis with concurrent CTP-based CBV ASPECTS.
MATERIALS AND METHODS: Patients with anterior circulation stroke who had baseline NCCT and CTP and underwent successful mechanical thrombectomy were included. NCCT-ASPECTS was assessed by 2 neuroradiologists, and discrepancies were resolved by consensus. CTP-CBV ASPECTS was assessed by a different neuroradiologist. Automated ASPECTS was provided by Brainomix software. ASPECTS was dichotomized (ASPECTS ≥6 or <6) and was also based on the time from onset (>6 or ≤6 hours).
RESULTS: A total of 58 patients were included. The interobserver agreement for NCCT ASPECTS was moderate (κ = 0.48) and marginally improved (κ = 0.64) for dichotomized data. Automated ASPECTS showed excellent agreement with consensus reads (κ = 0.84) and CTP-CBV ASPECTS (κ = 0.84). Intraclass correlation coefficients for ASPECTS across all 3 groups were 0.84 (95% CI, 0.76–0.90, raw scores) and 0.94 (95% CI, 0.91–0.96, dichotomized scores). Automated scores were comparable with consensus reads and CTP-CBV ASPECTS in patients when grouped on the basis of time from symptom onset (>6 or ≤6 hours). There was significant (P < .001) negative correlation with final infarction volume and the 3 ASPECTS groups (r = −0.52, consensus reads; −0.58, CTP-CBV; and −0.66, automated).
CONCLUSIONS: ASPECTS derived from an automated software performs equally as well as consensus reads of expert neuroradiologists and concurrent CTP-CBV ASPECTS and can be used to standardize ASPECTS reporting and minimize interpretation variability.
ABBREVIATIONS:
- AIS
- acute ischemic stroke
- IQR
- interquartile range
The ASPECTS was initially developed in an effort to standardize assessment of the extent of early ischemic change on NCCT of the head in patients with acute ischemic stroke (AIS).1 The ASPECTS assessment has been increasingly incorporated in treatment decision-making and has been used in several randomized clinical trials for endovascular treatment decision-making.2,3 ASPECTS ≥ 6 is now included in the latest version of the American Heart Association guidelines as an imaging eligibility criterion for endovascular treatment for patients presenting in the early treatment window (<6 hours).4
The major drawback of the ASPECTS evaluation is its modest interobserver agreement and reproducibility. Early ischemic changes are often difficult to detect on NCCT, with low interobserver agreement for presence and extent.5⇓–7
One solution to decrease variability in the detection of early ischemic changes is to use artificial intelligence and deep learning techniques to minimize the variability factor related to human interaction.8,9 Today automated software programs are commercially available to automatically calculate the ASPECTS using NCCT, with promising results in comparison with human interpretation.10⇓–12
By providing hemodynamic and physiologic information, CTP delivers an improved delineation of early ischemia in comparison with NCCT. Applying ASPECTS to CTP-CBV maps has shown promising results in terms of the accuracy of detecting early ischemic changes, reduced variability, and prediction of functional outcome over NCCT ASPECTS.5,13⇓–15
In this study, we aimed to evaluate the performance of automated ASPECTS obtained from a software-based analysis (Brainomix, Oxford, UK; www.brainomix.com) against the assessment of neuroradiologists in a comparative analysis with concurrent CT perfusion in a cohort of patients with acute ischemic stroke. This study was exploratory without a prespecified hypothesis.
MATERIALS AND METHODS
Patient Selection
This study was approved by the local institutional review board (Mount Sinai Health System, New York). We retrospectively reviewed consecutive patients with AIS who presented to our institution between January 2016 and July 2018 and met the following inclusion criteria: 1) time from symptom onset <24 hours, 2) anterior circulation ischemic stroke with large-vessel occlusion (intracranial carotid artery or MCA), 3) baseline NCCT, 4) technically adequate pretreatment CTP, and 5) successful recanalization defined by TICI (thrombolysis in cerebral infarction) ≥ 2b via mechanical thrombectomy.16
We recorded clinical data, including patients’ age, sex, baseline NIHSS scores, time from stroke onset/last well known, location of large-vessel occlusion, time from CT to recanalization, data on receiving intravenous tissue plasminogen activator before mechanical thrombectomy, grade of recanalization using the TICI scale, and 90-day mRS when available. mRS scores of 0–2 were classified as indicating a good functional outcome.
Image Acquisition
CT image acquisition was performed using 2 CT scanners, including a LightSpeed VCT (GE Healthcare, Milwaukee, Wisconsin) and a Somatom Definition (Siemens, Erlangen, Germany). Helical NCCT (120 kV, 100–350 auto-mA) was performed using a 5-mm section thickness from the foramen magnum through the vertex. CTP was performed with the following scan parameters: 100-mm coverage in the z-axis, 80 kV, 150 mA, effective dose = 3.3 mSv, section thickness = 5 mm, collimation = 64 × 0.625 mm. Total acquisition time was 60 seconds (30 consecutive spiral acquisitions, each 2 seconds). A total of 50 mL of iopamidol (Isovue-370; Bracco, Princeton, New Jersey) was injected intravenously followed by a 20-mL saline flush at 5 mL/s.
Image Analysis
Two board-certified neuroradiologists independently reviewed all baseline NCCTs and assigned an ASPECTS using a 10-point scale.1 The site of arterial occlusion (right or left) was presented at the time of image interpretation. Discrepancies between 2 readers were resolved using a consensus read in a separate reading session.
In addition, an automated software–based analysis (Brainomix) was used to calculate automated ASPECTS (e-ASPECTS; https://brainomix.com/e-aspects). Axial isotropic sequences from NCCT for each patient were uploaded to the software, and automated ASPECTS was calculated without human interaction. The neuroradiologists and automated ASPECTS were then exported to an Excel spreadsheet (Microsoft; Redmond, Washington) for analysis.
Subsequently, CTP data were processed using FDA-approved postprocessing software (Olea Sphere, Version 6.0; Olea Medical, La Ciotat, France). First, the arterial input function was detected automatically using a cluster-analysis algorithm.17 This arterial input function was subsequently used by the Bayesian probabilistic method18 to generate several perfusion parametric maps including CBV and CBF. To avoid recall bias, a board-certified neuroradiologist (different from the one who determined ASPECTS on NCCT) reviewed CTP-CBV maps and assigned an ASPECTS for each patient.
The volume of final infarction was calculated by applying a volume of interest to the DWI hyperintense region using a voxel-based signal intensity method subsuming the entire region of DWI hyperintensity. When follow-up MR imaging was not available (n = 13), follow-up CT within 24–48 hours from thrombectomy was used for determination of final infarction volume. In these patients, infarct (defined as established hypodense regions) was manually delineated by a neuroradiologist with 10 years of experience.
Statistical Analysis
Clinical and demographic data were presented as mean (SD) or median (interquartile range) as appropriate. ASPECTS values were presented as median (interquartile range). Comparison of ASPECTS was performed using both the raw/original scores and dichotomized ASPECTS using ≥6 and <6 as a cutoff. Interobserver agreement between 2 neuroradiologists was performed using a weighted κ test with calculation of the 95% CI. Agreement and correlation among neuroradiologist consensus reads, CTP-CBV, and automated ASPECTS were performed using the intraclass correlation coefficient with 95% CI. Comparative analysis between ASPECTS based on the time of symptom onset (<6 or >6 hours) was also performed. Finally, clinical and imaging variables were compared between groups with good-versus-poor functional outcome (using 90-day mRS >2 as a cutoff), using a combination of t tests and χ2 tests as appropriate.
RESULTS
Demographic and Clinical Data
A total of 58 patients met our inclusion criteria (27 men, 31 women; mean age, 69.9 ± 12.5 years). The mean of time from symptom onset was 8.4 ± 5.3 hours. The median and interquartile range (IQR) of the NIHSS were 15 and 10–21. A total of 42 patients had proximal middle cerebral artery occlusion, and 16 patients had intracranial internal carotid occlusion. Thirteen of 56 patients received IV-tPA before mechanical thrombectomy (this information was not available in 2 patients). The final mean infarct volume was 41 ± 56 mL. The median and IQR of 90-day mRS available in 50 patients were 3 and 1–5. The final recanalization (TICI) scores were 2b (n = 16), 2c (n = 10), and 3 (n = 32).
Human Interpretation
The median for ASPECTS was 9 (IQR, 7–10) for reader 1 and 8 (IQR, 6–9) for reader 2. The interobserver agreement was moderate with κ = 0.48 (95% CI, 0.33–0.62). For the dichotomized ASPECTS (ASPECTS ≥6 or <6), the interobserver agreement was improved with κ = 0.64 (95% CI, 0.32–0.95). For the final consensus read, the median for ASPECTS was 7 (IQR, 7–9). In the dichotomized consensus read, a total of 54 patients had ASPECTS ≥6 and 4 patients had ASPECTS <6.
For CTP-CBV ASPECTS, the median was 8 (IQR, 7–9). A total of 50 patients had ASPECTS <6, while 8 had ASPECTS ≥6. When ASPECTS values were evaluated between CTP and NCCT (human consensus reads), there was excellent agreement (κ = 0.84; 95% CI, 0.62–1.0) for the dichotomized scores and moderate agreement (κ = 0.57; 95% CI, 0.38–0.76) for the raw scores.
Automated ASPECTS of NCCT
The median was 9 (IQR, 8–10) for automated ASPECTS. The breakdown of dichotomized ASPECTS (ASPECTS ≥6/ASPECTS <6) was 52/6 for automated ASPECTS. There was excellent agreement (κ = 0.84; 95% CI, 0.62–1.0) between automated and consensus dichotomized ASPECTS. In only 2 patients, the software overestimated the extent of early ischemic changes by providing an automated ASPECTS <6, while the score was >6 by consensus read.
There was also excellent agreement (κ = 0.84; 95% CI, 0.62–1.0) between automated and CTP-CBV dichotomized ASPECTS. In only 2 patients did the software underestimate the extent of early ischemic changes by providing an automated ASPECTS >6, while the CTP-CBV ASPECTS was <6.
Intraclass correlation coefficients were 0.84 (95% CI, 0.76–0.90) for the original ASPECTS and 0.94 (95% CI, 0.91–0.96) for the dichotomized ASPECTS across all 3 groups (consensus, CTP-CBV, and automated ASPECTS). Comparative scatterplots of ASPECTS across all 3 groups are shown in Fig 1.
Comparative illustration of the distribution of ASPECTS from consensus reads of 2 neuroradiologists, software-based automated ASPECTS, and CTP-CBV for all patients in our study population (each marker represents a patient’s score). Automated ASPECTS showed excellent agreement (κ = 0.84; 95% CI, 0.62–1.0) with both consensus and CTP-CBV ASPECTS. The intraclass correlation coefficient was 0.84 (95% CI, 0.76–0.90) across all 3 groups.
Figure 2 demonstrates an example of ASPECTS and comparative analysis among humans, software, and CTP.
An 82-year-old woman who presented with right MCA M1 occlusion and an NIHSS score of 18. She underwent successful mechanical thrombectomy (TICI 3) with a CT-to-recanalization time of 50 minutes. Axial NCCT (A), automated ASPECTS (B), axial CTP-CBV (C), and axial NCCT 48 hours after endovascular treatment (D) are shown. For the 2 human readers, one scored 6 and the other, 7 (consensus ASPECTS, 6). B, Automated software assigned an ASPECTS of 6. CTP-CBV ASPECTS was 7. There is good topographic correlation with the final infarction volume.
Comparative analysis between ASPECTS values based on the time of symptom onset (<6 or >6 hours) did not show any significant difference for neuroradiologists, CTP-CBV, or automated ASPECTS (Table 1).
Comparative analysis among ASPECTS scores categorized on the basis of time of symptom onseta
In 50 patients who had a 90-day mRS, 22 patients (44%) had good functional outcome using 90-day mRS ≤ 2, while 28 patients (56%) had poor (mRS >2) functional outcome. Patients with poor functional outcome had significantly higher NIHSS (P = .001), older age (P = .02), and higher infarction volume (P = .03). Of note, none of the ASPECTS values (consensus read, CTP-CBV, or e-ASPECTS) were discriminatory between patients with good-versus-poor functional outcome. Clinical and imaging variables compared in groups with good-versus-poor functional outcome are summarized in Table 2.
Clinical and imaging data in patients with good-versus-poor functional outcome using mRS >2 as a cutoff
All 3 ASPECTS groups had significant (P < .001) negative correlation with final infarction volume, with a correlation coefficient (r) of −0.52 for the consensus reads, −0.58 for CTP-CBV, and −0.66 for automated ASPECTS. Figure 3 shows correlation scatterplots of ASPECTS and final infarction volumes.
Scatterplots for correlation between ASPECTS and final infarction volume for all 3 ASPECTS groups showing significant (P < .001) negative correlation (r = −0.52 for the consensus reads, −0.58 for CTP-CBV, and −0.66 for automated ASPECTS).
DISCUSSION
ASPECTS has become increasingly integrated into the decision-making process for intervention in patients with AIS. According to current guidelines, absence of a large infarction core via NCCT defined by ASPECTS ≥6 is sufficient for treatment decision-making in patients with anterior circulation large-vessel occlusion within the first 6 hours from the symptom onset.4 For the same population of patients if presenting between 6 and 24 hours from symptom onset, there is a lack of data supporting the use of NCCT ASPECTS alone for treatment selection,19 and additional imaging including CTP or MR imaging is recommended.4
Our results reaffirmed the concern of interobserver variability for ASPECTS assessment by showing only fair interobserver agreement (κ = 0.48), even for experienced neuroradiologists. This has been attributed to factors such as physician training and experience, time pressure, and personal bias of expected findings (for example, from the ordering or treatment teams), among other factors that have been noted as potential reasons for the variability of ASPECTS.11,20,21 Similar to other investigators, using dichotomized ASPECTS (≥6 or <6), we showed marginal improvement in interobserver agreement (κ = 0.64).13,22 However, there remains substantial variability in ASPECTS assessment, which can introduce uncertainty for all physicians involved in the care of patients with stroke and affect the clinical management and implementation of a guideline-based approach for stroke treatment.4
With the introduction of software packages trained on deep learning algorithms, attempts have been made to use automated ASPECTS as a way to address variability associated with human interpretation with some success.10⇓–12,23⇓–25 In this study, we showed that automated ASPECTS has a similar diagnostic performance to consensus reading of experienced neuroradiologists with excellent agreement (κ = 0.84). In a recent study by Maegerlein et al11 using a different software package, similar results were shown with substantial agreement (κ = 0.9) between automated and consensus reads. Most important, we showed that automated ASPECTS performed equally well compared with consensus reads regardless of the time from onset (≥6 or <6 hours). Assuming that consensus ASPECTS of 2 neuroradiologists can be used as an “operation criterion standard” in an acute setting, it is plausible to use automated ASPECTS to standardize NCCT interpretation in the acute setting and avoid variability associated with individual human interpretation, ensuring that all patients receive equivalent care and are triaged with appropriate treatment options.
Our second finding is that the described automated ASPECTS provides similar diagnostic performance to concurrently performed CTP-CBV ASPECTS. CTP provides more sensitive assessment of early ischemic changes in AIS because it contains physiologic and hemodynamic information about the ischemia. CTP ASPECTS has been used successfully in the triage of patients with AIS with the potential added value of improving reliability and reducing variability in the assessment of early ischemic changes compared with NCCT ASPECTS, in particular in earlier treatment windows.15,26⇓⇓–29 In a study of 227 patients, Naylor et al15 showed improvement in the reliability of early ischemic changes using CTP ASPECTS over NCCT ASPECTS. In our study, having similar diagnostic performance and excellent agreement (κ = 0.84) between automated software and CTP is reassuring and strengthens the validity of the automated ASPECTS and the argument that it can be used in clinical practice with confidence. In our study using a cutoff for time from symptom onset at 6 hours that has been used in the current guidelines, we did not find any significant discrepancy among our ASPECTS groups.
Finally, we showed that none of our ASPECTS groups (human, automated, or CTP) were predictive of functional outcome as it was measured by 90-day mRS. Although there are several reports supporting a favorable association between higher ASPECTS and good functional outcome,1,30,31 others have debated the ability of ASPECTS to predict outcome.20,32 There are also some reports that showed the predictive ability of CTP ASPECTS for determination of functional outcome.27,33 In a recent study by Pfaff et al,24 automated ASPECTS was shown to be predictive of functional outcome. One reason for the lack of association between ASPECTS and functional outcome in our study could be an insufficient number of patients with ASPECTS <6 (only 6 patients using automated ASPECTS). In fact, 21/22 patients who had good functional outcome had a baseline automated ASPECTS score of ≥ 6.
Although ASPECTS values were not predictive of functional outcome in our study, they were predictive of final infarction volume concordant with prior reports.34 Using automated ASPECTS, a recent study by Demeestere et al34 showed findings similar to ours by demonstrating a lack of association with functional outcome but significant correlation in the determination of final infarction volume. We found 3 variables, including age, NIHSS, and final infarction volume, as significant contributors to predicting functional outcome. Most interesting, 2 of these (age and NIHSS) have been used in a prior established predictive score (Houston Intra-Arterial Therapy 2 score, which combines age, glucose level, NIHSS, and ASPECTS) for improved prediction of functional outcome in patients with AIS.35
There are several limitations to our study. Retrospective design can introduce unknown bias. There was selection bias because only patients with AIS were included. The sample size was relatively small, and further validation studies with larger sample sizes are required to validate the practical application of our automated software as a stand-alone tool in the triage of patients with AIS. Similarly, we had only a small group of patients with low ASPECTS, and this limits assessment of the association with functional outcome. Infarct extension and increased volume are possible between CT and the follow-up imaging after endovascular thrombectomy, which was used for final infarction determination. We tried to minimize this confounding factor by including patients with successful recanalization. Another limitation is that the final imaging study used for determination of final infarction volume was CT in a subset of patients (22%), which is less than ideal in comparison with MR imaging and can possibly introduce inconsistencies into our analysis. Finally, the CTP-CBV ASPECTS values were assessed by only 1 observer.
CONCLUSIONS
We showed that automated ASPECTS provided by the described software performs equally well compared with a consensus read of expert neuroradiologists and concurrent CTP-CBV ASPECTS in patients presenting with acute ischemic stroke. If its potential is realized, automated ASPECTS can be used as a stand-alone tool for triage and treatment decision-making in patients with acute ischemic stroke.
Footnotes
Disclosures: J. Mocco—RELATED: Board Membership: Neurotechnology Investors; Consultancy: Imperative Care, Cerebrotech inc., Viseon, EndoStream Medical, Rebound Therapeutics, Vastrax; UNRELATED: Stock/Stock Options: Neurotechnology Investors, comet inc., Neurvana inc, Cardinal Consulting, Viseon, Echovate, Serenity, RIST, Spinaker, inc., Blockade Medical. Kambiz Nael—RELATED: Board Membership: Olea Medical, Comments: Medical Advisory Board.
Indicates open access to non-subscribers at www.ajnr.org
References
- Received July 4, 2019.
- Accepted after revision September 18, 2019.
- © 2019 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- MR microscopy to assess clot composition following mechanical thrombectomy predicts recanalization and clinical outcome
- Real-world evaluation of Brainomix e-Stroke software
- Automated assessment of ischemic core on non-contrast computed tomography: a multicenter comparative analysis with CT perfusion
- Alberta Stroke Program Early CT Score and collateral status predict target mismatch in large vessel occlusion with delayed time windows
- Performance of Automated ASPECTS Software and Value as a Computer-Aided Detection Tool
- Benefit and risk of intravenous alteplase in patients with acute large vessel occlusion stroke and low ASPECTS
- Emerging Artificial Intelligence Imaging Applications for Stroke Interventions
- Innovative use of artificial intelligence and digital communication in acute stroke pathway in response to COVID-19
- ASPECTS Distorts Infarct Volume Measurement