Automated ASPECTS on Noncontrast CT Scans in Patients with Acute Ischemic Stroke Using Machine Learning
=======================================================================================================

* H. Kuang
* M. Najm
* D. Chakraborty
* N. Maraj
* S.I. Sohn
* M. Goyal
* M.D. Hill
* A.M. Demchuk
* B.K. Menon
* W. Qiu

## Abstract

**BACKGROUND AND PURPOSE:** Alberta Stroke Program Early CT Score (ASPECTS) was devised as a systematic method to assess the extent of early ischemic change on noncontrast CT (NCCT) in patients with acute ischemic stroke (AIS). Our aim was to automate ASPECTS to objectively score NCCT of AIS patients.

**MATERIALS AND METHODS:** We collected NCCT images with a 5-mm thickness of 257 patients with acute ischemic stroke (<8 hours from onset to scans) followed by a diffusion-weighted imaging acquisition within 1 hour. Expert ASPECTS readings on DWI were used as ground truth. Texture features were extracted from each ASPECTS region of the 157 training patient images to train a random forest classifier. The unseen 100 testing patient images were used to evaluate the performance of the trained classifier. Statistical analyses on the total ASPECTS and region-level ASPECTS were conducted.

**RESULTS:** For the total ASPECTS of the unseen 100 patients, the intraclass correlation coefficient between the automated ASPECTS method and DWI ASPECTS scores of expert readings was 0.76 (95% confidence interval, 0.67–0.83) and the mean ASPECTS difference in the Bland-Altman plot was 0.3 (limits of agreement, −3.3, 2.6). Individual ASPECTS region-level analysis showed that our method yielded κ = 0.60, sensitivity of 66.2%, specificity of 91.8%, and area under curve of 0.79 for 100 × 10 ASPECTS regions. Additionally, when ASPECTS was dichotomized (>4 and ≤4), κ = 0.78, sensitivity of 97.8%, specificity of 80%, and area under the curve of 0.89 were generated between the proposed method and expert readings on DWI.

**CONCLUSIONS:** The proposed automated ASPECTS scoring approach shows reasonable ability to determine ASPECTS on NCCT images in patients presenting with acute ischemic stroke.

## ABBREVIATIONS:

AIS
:   acute ischemic stroke
AUC
:   area under curve
ICC
:   intraclass correlation coefficient
IQR
:   interquartile range

Management of patients with acute ischemic stroke (AIS) relies heavily on an assessment of the extent of irreversibly injured brain at baseline. Patients with extensive early ischemic changes at presentation are unlikely to benefit from thrombolysis or thrombectomy procedures. Moreover, such patients may also be at higher risk of developing complications of treatment such as intracerebral hemorrhage. The Alberta Stroke Program Early CT Score was devised as a systematic method of assessing the extent of early ischemic change on noncontrast CT in patients with AIS.1,2 Across the years, ASPECTS has gained credence and is now used the world over for this purpose,3⇓⇓⇓–7 though it has not been proved useful for selecting patients for treatment.8,9

Although conceptually, the ASPECTS is a simple method, scoring early ischemic change on NCCT scans continues to be a challenge, especially for readers with less experience.10⇓–12 Technical factors such as peak x-ray energy (kiloelectron volt/megaelectron volt) image processing and display procedures; patient factors such as old infarcts, brain atrophy, and leukoaraiosis; and reader factors such as experience, training, and specialty, all potentially affect ASPECTS interpretation.11,12 A solution to improve ASPECTS reading is training readers to recognize these issues while providing them with strategies that can help improve the reliability and validity of these reads. Another solution is to use novel technologies such as machine learning and feature extraction to develop automated solutions to ASPECTS interpretation.13⇓⇓⇓–17

In recent years, evidence that automated ASPECTS scoring methods based on machine learning are comparable with expert reading of ASPECTS is accumulating.18⇓⇓⇓⇓⇓–24 In this study, we developed an automated ASPECTS scoring system based on machine learning and feature engineering and compared it with expert ASPECTS readings on acute DWI. We introduced multiple high-order computational textural features into our machine learning model and hypothesized that this automated method can determine ASPECTS scores accurately and reliably compared with expert ASPECTS readings on acute DWI.

## Materials and Methods

Data are from the Keimyung Stroke Registry, an ongoing single-center prospective cohort study of patients with acute ischemic stroke presenting to the Keimyung University Hospital in Daegu, South Korea. Two hundred fifty-seven patients with acute ischemic stroke presenting within 8 hours of last known well who had baseline NCCT (slice thickness, ≤ 5 mm) followed by DWI performed within 1 hour of NCCT were included in the study. An expert scored ASPECTS on DWI; any individual region with diffusion restriction occupying >20% of that region was considered affected. To assess the reliability of expert-reading DWI ASPECTS, another expert was asked to score 60 DWI scans randomly selected from the 257 patients with AIS.

Of the 257 patient images, 157 were randomly selected for training a machine learning model, while the remaining 100 images were used to evaluate the trained model. Specifically, a NCCT template with ASPECTS regions manually contoured was nonlinearly registered onto all NCCT images (Fig 1). During the training stage, 376 texture features (details of texture features are shown in the On-line Appendix) such as high-order statistics and image textural features were extracted from each ASPECTS region from the 157 patient images bilaterally after median filtering. Note that the feature extraction and classification for each ASPECTS region were performed in 3D. Information on the side of the brain affected by ischemic stroke was used as an additional input to compute difference features between ischemic and normal brain tissue. Specifically, observers first determined the ischemic hemisphere based on imaging and clinical parameters. Feature differences were then obtained by subtracting regional level values on the ischemic side from the those on the contralateral side. Sixty patients (38 in the training dataset and 22 in the testing dataset) with posterior circulation strokes were included intentionally to reflect clinical reality. In patients with posterior circulation strokes, the left side was regarded as the default ischemic side.

![Fig 1.](http://www.ajnr.org/https://www.ajnr.org/content/ajnr/40/1/33/F1.medium.gif)

[Fig 1.](http://www.ajnr.org/content/40/1/33/F1)

Fig 1. 
Examples of each ASPECTS region. L indicates lentiform; I, insula; C, caudate; IC, internal capsule; M, MCA.

The computed features were first ranked using linear discriminant analysis. The ranked features were input into a random forest model using the expert-assessed ASPECTS on DWI as a class label. We used 5-fold cross-validation on the training samples to select training hyperparameters including the number of trees in the forest, the maximum depth of trees, and also the number of ranked features. Class weight was set to deal with the imbalanced data distribution on the basis of the ratio of abnormal and normal samples in the training data. The detailed parameter settings are shown in the On-line Table. We trained a classifier for each ASPECTS region. The random forest training and testing were implemented using Scikit-learn in Python ([http://scikit-learn.org/stable/](http://scikit-learn.org/stable/)). The trained random forest classifier was then validated on the remaining 100 test patient images. A flowchart of the training and testing process of each ASPECTS region is shown in Fig 2.

![Fig 2.](http://www.ajnr.org/https://www.ajnr.org/content/ajnr/40/1/33/F2.medium.gif)

[Fig 2.](http://www.ajnr.org/content/40/1/33/F2)

Fig 2. 
A flowchart of the training and testing processes used in the study for each ASPECTS region.

### Statistical Analysis

Expert ASPECTS readings on DWI of the 100 test images were used as the ground truth to evaluate the automated ASPECTS obtained by our method. Agreement on the total ASPECTS score was measured using the intraclass correlation coefficient (ICC). Boxplots and Bland-Altman plots were used to illustrate differences in the assessment of total ASPECTS between the automated method and the ground-truth (expert-read DWI ASPECTS). The ICC analysis was also stratified by stroke onset-to-CT time (≤90 minutes, *n* = 69; 90–270 minutes, *n* = 21; and >270 minutes, *n* = 10). Because physicians use the presence or absence of extensive early ischemic changes to make clinical decisions on treatment in patients with acute ischemic stroke, we also assessed agreement on the ASPECTS interpretation between the automated method and DWI using κ statistics on a dichotomized ASPECTS threshold (>4 versus ≤4).25 κ statistics were also used to assess agreement between the automated method and expert-read DWI at each individual ASPECTS region.

Receiver operating characteristics based on the MedCalc for Windows software (MedCalc Software, Mariakerke, Belgium) were used to report the area under the curve (AUC) for the dichotomized ASPECTS (>4 versus ≤4), and individual region-level ASPECTS analysis using automated ASPECTS, as an independent variable and expert-read DWI ASPECTS, as a dependent variable. A clustered receiver operating characteristic method in R statistical and computing software ([http://www.r-project.org](http://www.r-project.org)) was used to report the AUC for grouped ASPECTS regions.26 In addition, accuracy defined as the ratio of accurately classified and total samples, sensitivity, and specificity was also calculated to further measure the performance of our proposed ASPECTS method.

A linear-weighted κ of the trichotomized ASPECTS (0–4, 5–7, 8–10) was computed. A sensitivity analysis was performed by varying threshold involvement of each ASPECTS region on expert-read DWI as >0% and >50% involvement compared with the >20% involvement used for primary analyses. Additionally, to demonstrate the efficacy of the developed automated ASPECTS method, we compared the ASPECTS reading of a stroke expert on the 100 test images with the automated ASPECTS and the expert-assessed ASPECTS on DWI. All statistical analyses were performed by using MedCalc 17.8 and Matlab (MathWorks, Natick, Massachusetts). A 2-sided α < .05 was considered statistically significant.

## Results

Of 157 patients included in the training dataset (median age, 69 years; interquartile range [IQR], 62–76 years; 54.8% male), baseline NCCT was performed within a median time of 46.5 minutes (IQR, 27–117 minutes) from last known well compared with a median baseline NCCT to baseline MR imaging time of 39.5 minutes (IQR, 30–51 minutes). Of 100 patients included in the test dataset (median age, 70 years; IQR, 64–77 years; 56% male), baseline NCCT was performed within a median time of 49 minutes from last known well (IQR, 23.8–95.5 minutes) compared with a median baseline NCCT to baseline MR imaging time of 39 minutes (IQR, 29–50.3 minutes). The median baseline ASPECTS on the training dataset using DWI was 8 (IQR, 6–9).

The κ values for regional and dichotomized ASPECTS between the 2 expert-read DWI ASPECTSs were 0.86 (95% CI, 0.81–0.91) and 0.85 (95% CI, 0.56–1), respectively. The ICC for total ASPECTS between the 2 expert-read DWI ASPECTSs was 0.90 (95% CI, 0.84–0.94).

When patients were stratified by stroke onset-to-CT time (≤90 minutes, *n* = 69; 90–270 minutes, *n* = 21; and >270 minutes, *n* = 10), the ICCs between the automated CT ASPECTS and the DWI ASPECTS for these 3 subgroups were 0.80 (95% CI, 0.69–0.87), 0.77 (95% CI, 0.52–0.90), and 0.26 (95% CI, −0.40–0.75), respectively. No evidence was found that the ICC increased in patients with longer onset-to-CT time using the automated ASPECTS method.

The median baseline ASPECTS generated by the automated method on test data (*n* = 100) was 8 (IQR, 7–9) versus a score of 7 (IQR, 6–9) on the ground truth DWI. Figure 3*A* shows a boxplot overlaid with a scatterplot showing the distribution of the automated CT ASPECTS at each individual ASPECTS on DWI. The intraclass correlation coefficient for total ASPECTS between the automated method and DWI was 0.76 (95% CI, 0.67–0.83). Figure 3*B* illustrates Bland-Altman agreement plots between the automated method and DWI for total ASPECTS. The mean difference in total ASPECTS between the automated method and DWI was minimal (0.3; limit of agreement, −3.3, 2.6).

![Fig 3.](http://www.ajnr.org/https://www.ajnr.org/content/ajnr/40/1/33/F3.medium.gif)

[Fig 3.](http://www.ajnr.org/content/40/1/33/F3)

Fig 3. 
*A*, Boxplot with a scatterplot showing the distribution of the automated CT ASPECTS at each individual ASPECTS on DWI. *B*, A Bland-Altman plot illustrating agreement between a total automated ASPECTS score and ASPECTS scores on DWI. Random jitter has been added to illustrate the number of measurements at each ASPECTS point. The *horizontal black line* represents the mean difference in the ASPECTS score between the 2 methods, while the *dotted lines* represent a 1.96 SD around the difference.

Agreement on ASPECTS between the automated method and DWI using a dichotomized ASPECTS threshold of >4 versus ≤4 was good (κ = 0.78: 95% CI, 0.57–0.99). Sensitivity (97.8%: 95% CI, 92.2%–99.7%), specificity (80%: 95% CI, 34.8%–93.3%), F1 measure (0.98), and AUC (0.89: 95% CI, 0.81–0.94) were reasonably good. When ASPECTS was trichotomized (0–4, 5–7, 8–10), the agreement between the automated method and DWI was good as well (linear weighted κ = 0.66; 95% CI, 0.53–0.80). Agreement on ASPECTS between the automated method and DWI at the individual ASPECTS region level is reported in Table 1. κ for agreement between the 2 methods ranged from 0.36 to 0.64. The automated ASPECTS method demonstrated high specificity but modest sensitivity compared with DWI at the regional level. F1 measures are also shown in Table 1.

View this table:
[Table 1:](http://www.ajnr.org/content/40/1/33/T1)

Table 1: 
κ, accuracy, F1 measure, sensitivity, specificity, and AUC on each ASPECTS region

Sensitivity analysis was attempted by varying the threshold involvement of each ASPECTS region on expert-read DWI as >0% and >50% involvement in addition to the >20% involvement used for the primary analyses. Region-level agreement between the automated method and expert-rated DWI ASPECTS for all 3 thresholds is shown in Table 2. Agreement between the 2 methods was best when DWI ASPECTS was rated using the >50% threshold method.

View this table:
[Table 2:](http://www.ajnr.org/content/40/1/33/T2)

Table 2: 
Agreement on ASPECTS interpretation at a regional level and for dichotomized ASPECTS (>4 vs. ≤4) between the automated ASPECTS method and expert-read DWI ASPECTS using different DWI ASPECTS region-involvement thresholds

The ICC for total ASPECTS between the expert-rated NCCT and DWI was 0.67 (95% CI, 0.55–0.77). The agreement between the expert-rated CT ASPECTS and DWI using a dichotomized ASPECTS threshold of >4 versus ≤4 was modest (κ = 0.42; 95% CI, 0.22–0.62). Sensitivity (81.1%: 95% CI, 71.5%–88.6%), specificity (90%: 95% CI, 55.5%–99.7%), F1 measure (0.89), and AUC (0.85: 95% CI, 0.77–0.92) were obtained.

The ICC for total ASPECTS between the expert-rated NCCT and the automated CT ASPECTS was 0.61 (95% CI, 0.47–0.72). The agreement between the expert-rated CT ASPECTS and the automated CT ASPECTS using a dichotomized ASPECTS threshold of >4 versus ≤4 was modest (κ = 0.48; 95% CI, 0.28–0.68). An example of expert-rated DWI ASPECTS, our automated CT ASPECTS, and expert-rated CT ASPECTS is shown in Fig 4.

![Fig 4.](http://www.ajnr.org/https://www.ajnr.org/content/ajnr/40/1/33/F4.medium.gif)

[Fig 4.](http://www.ajnr.org/content/40/1/33/F4)

Fig 4. 
Examples of DWI ASPECTS, the automated CT ASPECTS derived in this study, and expert-read CT ASPECTS. ASPECTS regions with ischemic changes are shown in color.

## Discussion

Results from these analyses using 100 patient images show that the automated ASPECTS method proposed in this article agrees well with expert-read DWI ASPECTS at a regional level and for the total ASPECTS. Moreover, good agreement between the automated method and expert-read DWI ASPECTS for ASPECTS cut-points (>4 versus ≤4) may help evaluate patients for the presence or absence of large infarcts at baseline. These results also show that the automated ASPECTS method is not inferior to expert-read ASPECTS on NCCT.

A commercially available automated ASPECTS scoring system (e-ASPECTS; [https://brainomix.com/e-aspects](https://brainomix.com/e-aspects)) based on a machine learning algorithm has shown an ability to detect early ischemic changes on NCCT at a level similar to that of junior stroke physicians while being noninferior to neuroradiologists.19 That study used ASPECTS on baseline and follow-up CT scans as the ground truth for comparison. e-ASPECTS was further evaluated in a small study of 34 patients in which baseline CT and DWI scans were obtained <2 hours apart.20 Another automated ASPECTS system combining filtering, bi-level and regional growth, feature selection, and a support vector machine was tested on 40 patients with AIS using DWI scans as the ground truth.24 This method obtained a κ of 0.52 for dichotomized ASPECTS (>7 versus ≤7). However, the NCCT and diffusion-weighted imaging time was not reported, making it difficult to evaluate its clinical applicability. Other methods of scoring ASPECTS automatically have mostly been tested against expert-read ASPECTS on NCCT. A major strength of this study is the use of an ASPECTS read on acute DWI by an expert as the ground truth to validate the automated ASPECTS method. This assures that the validity of the automated method was tested to a very high standard.

The proposed automated ASPECTS scoring method is based on feature engineering and random forest learning. Random forest is considered one of the most recent and popular boosting methods and has proved classification performance for difficult problems in many medical image–analysis applications compared with other classifiers.27,28 Random forest is an ensemble learning method that combines multiple weak classifiers (decision trees) and lets these decision trees vote for the most popular class. Each tree in the forest relies on a random vector sampled independently, and all trees in the forest have the same distribution. The growth of the tree is governed by random vectors. A measure of randomness is introduced into the training, which can prevent the training classifier from getting stuck at a local minimum, thereby improving the accuracy and reducing the chances of overfitting.

Some previous automated methods have used first-order image features, such as Hounsfield unit (HU) or density and HU difference between the ischemic and contralateral side as features for their algorithms. These first-order image features have limitations in patients with subtle ischemic changes and when images have low signal-to-noise ratios and motion artifacts. The use of multiple higher order computational textural features as part of the machine learning algorithms in the automated ASPECTS method proposed here helps us improve the validity of our technique.

This study has some limitations. First, of the 157 training images randomly selected, only 26.1% (410/1570) of ASPECTS regions had ischemic changes versus 73.9% (1160/1570) normal ASPECTS regions. Improving the performance by tackling the imbalance in data distribution is a goal for our machine learning algorithms. Second, this analysis used imaging data from 1 site. NCCT image acquisition and quality vary across sites; we will, therefore, need to validate the automated ASPECTS method in other data from other sites. Third, only 10 patients had ASPECTS ≤4 of 100 test patients, thus raising some valid concerns about the stability of results in the dichotomized ASPECTS analysis. Validation on a larger dataset is required to demonstrate the robustness of these results. Fourth, only a single atlas was used to localize the ASPECTS regions, which might not be optimal for all patients compared with using a method based on multiple atlases. However, localization based on nonlinear registration using a single atlas can maintain a good trade-off between computational cost and accuracy because saving time is critical in the acute stroke setting. Improving registration accuracy using a single atlas remains an open problem for brain imaging despite some existing atlas-selection techniques.29

## Conclusions

The automated ASPECTS method developed here could accurately and reliably assign ASPECTS on baseline NCCT scans in patients presenting with acute ischemic stroke. This work therefore further validates the utility of machine learning algorithms in developing software that can help and support physicians in interpreting brain scans of patients with acute ischemic stroke.

## Footnotes

*   Disclosures: Mayank Goyal—*RELATED*: *Other*: GE Healthcare, *Comments*: licensing agreement concerning systems of stroke diagnosis; *UNRELATED: Consultancy:* Medtronic, Stryker, MicroVention, Mentice, *Comments*: consulting and advice concerning acute stroke management; *Grants/Grants Pending*: Stryker, *Comments*: funding for the UNMASK EVT study.* Michael D. Hill—*UNRELATED*: *Consultancy*: Merck, *Comments*: Adjudication Board for stroke outcomes for clinical trials in diabetes; *Grants/Grants Pending*: Medtronic, Stryker, Boehringer Ingelheim, Alberta Innovates, Canadian Institutes for Health Research, *Comments*: multiple grants for stroke clinical trials*; *Stock/Stock Options*: Calgary Scientific Inc, *Comments*: stock ownership in a private image-processing company. Bijoy K. Menon—*RELATED*: *Grant*: Canadian Institute of Health Research, *Comments*: operating grant from the Canadian Institute of Health Research.* *Money paid to the institution.

*   This study was funded through an operating grant from the Canadian Institute of Health Research.

Indicates open access to non-subscribers at [www.ajnr.org](http://www.ajnr.org)

## References

1.  1. Barber PA, Demchuk AM, Zhang J, et al. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy: ASPECTS Study Group—Alberta Stroke Programme Early CT Score. Lancet 2000;355:1670–74 doi:10.1016/S0140-6736(00)02237-6 pmid:10905241
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/S0140-6736(00)02237-6&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=10905241&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000086983800008&link_type=ISI) 

2.  2. Pexman JW, Barber PA, Hill MD, et al. Use of the Alberta Stroke Program Early CT Score (ASPECTS) for assessing CT scans in patients with acute stroke. AJNR Am J Neuroradiol 2001;22:1534–42 pmid:11559501
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIyMi84LzE1MzQiO3M6NDoiYXRvbSI7czoxODoiL2FqbnIvNDAvMS8zMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.  3. Aoki J, Kimura K, Shibazaki K, et al. DWI-ASPECTS as a predictor of dramatic recovery after intravenous recombinant tissue plasminogen activator administration in patients with middle cerebral artery occlusion. Stroke 2013;44:534–37 doi:10.1161/STROKEAHA.112.675470 pmid:23212169
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjg6IjQ0LzIvNTM0IjtzOjQ6ImF0b20iO3M6MTg6Ii9ham5yLzQwLzEvMzMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

4.  4. Nezu T, Koga M, Kimura K, et al. Pretreatment ASPECTS on DWI predicts 3-month outcome following rt-PA: SAMURAI rt-PA Registry. Neurology 2010;75:555–61 doi:10.1212/WNL.0b013e3181eccf78 pmid:20697108
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1212/WNL.0b013e3181eccf78&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=20697108&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

5.  5. Goyal M, Demchuk AM, Menon BK, et al; ESCAPE Trial Investigators. Randomized assessment of rapid endovascular treatment of ischemic stroke. N Engl J Med 2015;372:1019–30 doi:10.1056/NEJMoa1414905 pmid:25671798
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1056/NEJMoa1414905&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=25671798&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

6.  6. Hill MD, Rowley HA, Adler F, et al; PROACT-II Investigators. Selection of acute ischemic stroke patients for intra-arterial thrombolysis with pro-urokinase by using ASPECTS. Stroke 2003;34:1925–31 doi:10.1161/01.STR.0000082483.37127.D0 pmid:12843342
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjM0LzgvMTkyNSI7czo0OiJhdG9tIjtzOjE4OiIvYWpuci80MC8xLzMzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

7.  7. Okazaki S, Moriwaki H, Minematsu K, et al. Extremely early computed tomography signs in hyperacute ischemic stroke as a predictor of parenchymal hematoma. Cerebrovasc Dis 2008;25:241–46 doi:10.1159/000113862 pmid:18216466
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1159/000113862&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=18216466&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

8.  8. Yoo AJ, Berkhemer OA, Fransen PS, et al; MR CLEAN investigators. Effect of baseline Alberta Stroke Program Early CT Score on safety and efficacy of intra-arterial treatment: a subgroup analysis of a randomised phase 3 trial (MR CLEAN). Lancet Neurol 2016;15:685–94 doi:10.1016/S1474-4422(16)00124-1 pmid:27302238
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/S1474-4422(16)00124-1&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=27302238&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

9.  9. Goyal M, Menon BK, van Zwam WH, et al; HERMES collaborators. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet 2016;387:1723–31 doi:10.1016/S0140-6736(16)00163-X pmid:26898852
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/S0140-6736(16)00163-X&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=26898852&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

10. 10. Grotta JC, Chiu D, Lu M, et al. Agreement and variability in the interpretation of early CT changes in stroke patients qualifying for intravenous rtPA therapy. Stroke 1999;30:1528–33 doi:10.1161/01.STR.30.8.1528 pmid:10436095
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjMwLzgvMTUyOCI7czo0OiJhdG9tIjtzOjE4OiIvYWpuci80MC8xLzMzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

11. 11. Menon B, Smith E, Modi J, et al. Regional leptomeningeal score on CT angiography predicts clinical and imaging outcomes in patients with acute anterior circulation occlusions. AJNR Am J Neuroradiol 2011;32:1640–45 doi:10.3174/ajnr.A2564 pmid:21799045
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYWpuciI7czo1OiJyZXNpZCI7czo5OiIzMi85LzE2NDAiO3M6NDoiYXRvbSI7czoxODoiL2FqbnIvNDAvMS8zMy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

12. 12. Menon BK, Puetz V, Kochar P, et al. ASPECTS and other neuroimaging scores in the triage and prediction of outcome in acute stroke patients. Neuroimaging Clin N Am 2011;21:407–23 doi:10.1016/j.nic.2011.01.007 pmid:21640307
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.nic.2011.01.007&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21640307&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

13. 13. Bentley P, Ganesalingam J, Carlton Jones AL, et al. Prediction of stroke thrombolysis outcome using CT brain machine learning. Neuroimaging Clin 2014;4:635–40 doi:10.1016/j.nicl.2014.02.003 pmid:24936414
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.nicl.2014.02.003&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=24936414&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

14. 14. Przelaskowski A, Sklinda K, Bargieł P, et al. Improved early stroke detection: wavelet-based perception enhancement of computerized tomography exams. Comput Biol Med 2007;37:524–33 doi:10.1016/j.compbiomed.2006.08.004 pmid:16999952
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.compbiomed.2006.08.004&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=16999952&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

15. 15. Rajini NH, Bhavani R. Computer aided detection of ischemic stroke using segmentation and texture features. Measurement 2013;46:1865–74 doi:10.1016/j.measurement.2013.01.010
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.measurement.2013.01.010&link_type=DOI) 

16. 16. Tang FH, Ng DK, Chow DH. An image feature approach for computer-aided detection of ischemic stroke. Comput Biol Med 2011;41:529–36 doi:10.1016/j.compbiomed.2011.05.001 pmid:21605853
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.compbiomed.2011.05.001&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=21605853&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

17. 17. Chawla M, Sharma S, Sivaswamy J, et al. A method for automatic detection and classification of stroke from brain CT images. Conf Proc IEEE Eng Med Biol Soc 2009;2009:3581–84 pmid:19965232
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=19965232&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

18. 18. Hampton-Till J, Harrison M, Kühn AL, et al. Automated quantification of stroke damage on brain computed tomography scans: e-ASPECTS. EMJ 2015;3:69–74 [https://www.emjreviews.com/neurology/article/automated-quantification-of-stroke-damage-on-brain-computed-tomography-scans-e-aspects/](https://www.emjreviews.com/neurology/article/automated-quantification-of-stroke-damage-on-brain-computed-tomography-scans-e-aspects/). Accessed August 6, 2015. 
    
    

19. 19. Nagel S, Sinha D, Day D, et al. e-ASPECTS software is non-inferior to neuroradiologists in applying the ASPECT score to computed tomography scans of acute ischemic stroke patients. Int J Stroke 2017;12:615–22 doi:10.1177/1747493016681020 pmid:27899743
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1177/1747493016681020&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=27899743&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

20. 20. Herweh C, Ringleb PA, Rauch G, et al. Performance of e-ASPECTS software in comparison to that of stroke physicians on assessing CT scans of acute ischemic stroke patients. Int J Stroke 2016;11:438–45 doi:10.1177/1747493016632244 pmid:26880058
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1177/1747493016632244&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=26880058&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

21. 21. Stoel BC, Marquering HA, Staring M, et al. Automated brain computed tomographic densitometry of early ischemic changes in acute stroke. J Med Imaging (Bellingham) 2015;2:014004 doi:10.1117/1.JMI.2.1.014004 pmid:26158082
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1117/1.JMI.2.1.014004&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=26158082&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

22. 22. Shieh Y, Chang CH. Automated ASPECTS scoring system as a clinical support system for acute stroke care. In: Proceedings of 2012 IEEE-Engineering in Medicine and Biology Society International Conference on Biomedical and Health Informatics, Hong Kong, China. January 5–7, 2012:691–94
    
    

23. 23. Shieh Y, Chang CH, Shieh M, et al. Computer-aided diagnosis of hyperacute stroke with thrombolysis decision support using a contralateral comparative method of CT image analysis. J Digit Imaging 2014;27:392–406 doi:10.1007/s10278-013-9672-x pmid:4464216
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1007/s10278-013-9672-x&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=4464216&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

24. 24. Su JL, Chan L, Huang SY. Development of computer aids ASPECTS system for acute ischemic stroke patient: a preliminary study. In: Proceedings of the 2017 2nd International Conference for Innovation in Biomedical Engineering and Life Sciences, Pantang, Malaysia. December 7, 2017:203–07
    
    

25. 25. Inoue M, Olivot JM, Labreuche J, et al. Impact of diffusion-weighted imaging Alberta stroke program early computed tomography score on the success of endovascular reperfusion therapy. Stroke 2014;45:1992–98 doi:10.1161/STROKEAHA.114.005084 pmid:24923724
    
    [Abstract/FREE Full Text](http://www.ajnr.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjQ1LzcvMTk5MiI7czo0OiJhdG9tIjtzOjE4OiIvYWpuci80MC8xLzMzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

26. 26. Obuchowski NA. Nonparametric analysis of clustered ROC curve data. Biometrics 1997;53:567–78 pmid:9192452
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.2307/2533958&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=9192452&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=A1997XE96500014&link_type=ISI) 

27. 27. Criminisi A, Shotton J. Decision Forests for Computer Vision and Medical Image Analysis. London; Springer-Verlag; 2013
    
    

28. 28. Maier O, Schröder C, Forkert ND, et al. Classifiers for ischemic stroke lesion segmentation: a comparison study. PLoS One 2015;10:e0145118 doi:10.1371/journal.pone.0145118 pmid:26672989
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1371/journal.pone.0145118&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=26672989&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 

29. 29. Aljabar P, Heckemann RA, Hammers A, et al. Multi-atlas-based segmentation of brain images: atlas selection and its effect on accuracy. Neuroimage 2009;46:726–38 doi:10.1016/j.neuroimage.2009.02.018 pmid:19245840
    
    [CrossRef](http://www.ajnr.org/lookup/external-ref?access_num=10.1016/j.neuroimage.2009.02.018&link_type=DOI) 
    
    [PubMed](http://www.ajnr.org/lookup/external-ref?access_num=19245840&link_type=MED&atom=%2Fajnr%2F40%2F1%2F33.atom) 
    
    [Web of Science](http://www.ajnr.org/lookup/external-ref?access_num=000265938700020&link_type=ISI) 

*   Received June 20, 2018.
*   Accepted after revision October 8, 2018.


*   © 2019 by American Journal of Neuroradiology