Abstract
BACKGROUND AND PURPOSE: Many small, regularly shaped cerebral aneurysms rupture; however, they usually receive a low score based on current risk-assessment methods. Our goal was to identify patient and aneurysm characteristics associated with rupture of small, regularly shaped aneurysms and to develop and validate predictive models of rupture in this aneurysm subpopulation.
MATERIALS AND METHODS: Cross-sectional data from 1079 aneurysms smaller than 7 mm with regular shapes (without blebs) were used to train predictive models for aneurysm rupture using machine learning methods. These models were based on the patient population, aneurysm location, and hemodynamic and geometric characteristics derived from image-based computational fluid dynamics models. An independent data set with 102 small, regularly shaped aneurysms was used for validation.
RESULTS: Adverse hemodynamic environments characterized by strong, concentrated inflow jets, high speed, complex and unstable flow patterns, and concentrated, oscillatory, and heterogeneous wall shear stress patterns were associated with rupture in small, regularly shaped aneurysms. Additionally, ruptured aneurysms were larger and more elongated than unruptured aneurysms in this subset. A total of 5 hemodynamic and 6 geometric parameters along with aneurysm location, multiplicity, and morphology, were used as predictive variables. The best machine learning rupture prediction-model achieved a good performance with an area under the curve of 0.84 on the external validation data set.
CONCLUSIONS: This study demonstrated the potential of using predictive machine learning models based on aneurysm-specific hemodynamic, geometric, and anatomic characteristics for identifying small, regularly shaped aneurysms prone to rupture.
ABBREVIATIONS:
- AUC
- area under the curve
- F1 score
- harmonic mean of precision and recall
- FPR
- false-positive rate
- PHASES
- Population, Hypertension, Age, Size, Earlier subarachnoid hemorrhage, and Site
- PPV
- positive predictive value
- ROC
- receiver operating characteristic
- SVM
- support vector machine
- TPR
- true-positive rate
- WSS
- wall shear stress
Cerebral aneurysms are a common vascular disease affecting about 2%–5% of the general population.1,2 Many studies have focused on identifying risk factors for the rupture of cerebral aneurysms.3⇓-5 Aneurysm size and shape irregularity determined by the presence of blebs have been identified as risk factors for future aneurysm rupture.6⇓-8 In addition, risk-scoring scales such as Population, Hypertension, Age, Size, Earlier subarachnoid hemorrhage, and Site (PHASES)5 assign a higher risk of rupture to aneurysms larger than 7 mm or aneurysms that have blebs.
However, it is well-known that a large number of aneurysms presenting with rupture are small (<7mm),9,10 and many of them have regular shapes and do not have any blebs or daughter sacs. Thus, these aneurysms would, in general, receive a low score for rupture risk based on the current scales, leading to undertreatment and unnecessary patient mortality and morbidity. Therefore, it would be of great importance to identify those small, regularly shaped aneurysms that are prone to rupture to recommend them for treatment and prevent devastating consequences, while, at the same time, minimizing the number of unnecessary interventions.
Thus, the objective of this study was to identify patient and aneurysm characteristics associated with the rupture of small, regularly shaped aneurysms, train machine learning models using those characteristics to identify rupture-prone aneurysms in this important subpopulation, and evaluate the predictive performance of the trained models on an external data set to assess their potential clinical utility.
MATERIALS AND METHODS
Overview of Methodology
This was a cross-sectional study that used several data sets generated from populations from multiple geographic regions. The analysis was restricted to small (<7mm), regularly shaped aneurysms (ie, without blebs). The study was conducted in 3 steps: 1) identify the distinguishing characteristics between ruptured and unruptured small, regularly shaped aneurysms, 2) develop predictive models of rupture using machine learning techniques, and 3) validate the model performance using a separate independent data set from different hospitals. To achieve these goals, we used 2 independent cross-sectional data sets. The first data set was used for identifying differences between ruptured and unruptured small, regularly shaped aneurysms and to train the predictive models. The second data set was used as an external data set to validate the predictions of these models and determine their predictive power and generalizability. Protocols for consent, data handling, and analysis were approved by the institutional review boards at the University of Pittsburgh and George Mason University.
Patients and Data
The first data set used for machine learning model development and internal cross-validation (training set) had a total of 1079 aneurysms in 764 patients. These aneurysms corresponded to patients referred for diagnostic angiography and imaged with 3D rotational angiography or CTA. This set included data from different populations, including the United States, South America, Europe other than Finland, Finland, and Japan. Of these aneurysms, 197 were ruptured (18.3%) and the remaining 882 aneurysms (81.7%) were unruptured. Patients ranged from 12 to 90 years of age, with a mean of 56.4 years. There were 570 (75%) females and 194 (25%) males, and the mean aneurysm size was 4.1 mm (range, 1.1–6.9 mm). Multiple aneurysms were present in 344 patients (45%).
The second data set (validation set) contained 102 aneurysms in 63 patients selected for surgical clipping and imaged with 3D rotational angiography or CTA before surgery. This set contained data from the United States and Finland populations. Of these aneurysms, 14 were ruptured (13.7%) and the remaining 88 aneurysms (86.3%) were unruptured. The average patient age was 53.3 years (median, 54 years; range, 25–69 years). There were 49 (78%) women and 14 (22%) men, and the mean aneurysm size was 4.4 mm (range, 1.3–6.9 mm). Multiple aneurysms were present in 23 patients (37%). For this study, de-identified vascular geometries and patient demographic information were obtained from our data base. The details about these data sets are summarized in the Online Supplemental Data.
Aneurysm Characterization
Patient-specific 3D vascular models were constructed for all aneurysms in the training and testing sets from the available 3D rotational angiography or CTA images as previously described.11 To characterize the hemodynamic environment of the aneurysm, we performed computational fluid dynamics simulations. Similar to previous studies,12,13 blood was modeled as a Newtonian incompressible fluid with a density of 1.0 g/cm3 and viscosity of 0.04 Poise, and the unsteady Navier-Stokes equations were numerically solved using finite elements. Vascular walls were approximated as rigid, and pulsatile inflow conditions were imposed by scaling a representative flow waveform with an empiric law relating flow rate and cross-sectional vessel area in internal carotid and vertebral arteries. Outflow conditions were imposed by dividing flows consistent with the Murray law. Simulations were performed for 2 cardiac cycles with a heart-beat rate of 60 beats per minute using 100 time-steps per cardiac cycle, and data from the second cycle were used to characterize the flow conditions in the aneurysm.
To characterize the aneurysm hemodynamics and geometry, we computed 15 flow variables and 10 geometric parameters defined on the aneurysm region.14,15 Additionally, the aneurysm anatomic characteristics (location, morphology, and multiplicity) and patient demographics (population, sex, and age) were described by 6 categoric variables (numeric in the case of age).
Postprocessing and Construction of Rupture-Predictive Models
Patient and aneurysm characteristics associated with rupture of small, regularly shaped aneurysms were identified in a data set 1 by performing contingency table analysis and the Pearson χ2 test for categoric variables. For continuous variables, tests for differences in the median of the 2 populations defined by the rupture status were performed via the 2-sample unpaired Wilcoxon (Mann-Whitney) test. All statistical analyses and machine learning modeling were performed in R statistical and computing software (http://www.r-project.org/). Comparisons between multiple groups were adjusted using the Benjamini-Hochberg method, and associations were considered significant with P < .05 after adjustment.
Various machine learning methods for supervised classification were used to identify the best predictive power. These included logistic regression, support vector machine (SVM), K-nearest neighbor, random forest, and bagging or bootstrap aggregating. Data from all 1079 aneurysms (no missing data) of the first data set were used for model training.
The columns of the feature matrix of the continuous predictor variables were standardized so that the attributes would have a mean value of zero and an SD of 1. Categoric variables were encoded as dummy variables. One hundred repetitions of nested 10-fold (internal) cross-validation, yielding 100 random partitions of the original training sample, were used to train the models, estimate the tuning parameters, and identify the important predictor variables. In this step, data set 1 was split into training and testing subsets for each of the 10 folds, and the optimal value of each tuning parameter related to the training process was determined via a grid search to achieve the largest area under the curve (AUC) of the receiver operating characteristic (ROC). The 100 results were combined (averaged) to produce a single estimation.
Feature Selection
To identify features to be used to predict rupture in aneurysms, we performed the recursive feature elimination16 technique. Briefly, an initial model was built on the basis of the entire set of predictors, and an importance score was computed for each predictor. Then, the least important predictors were recursively removed while maintaining the overall model accuracy. The optimal subset of features was then used to train the final model.
Because only about 20% of small, regularly shaped cerebral aneurysms rupture, the data sets are unbalanced, negatively impacting the model fitting and performance. To deal with this problem during the internal cross-validation of the model training process, we used a down-sampling approach, in which data from most classes (unruptured aneurysms) were randomly removed to achieve a balanced class distribution and consequently mitigate this issue.
Model Performance and Validation
First, the performance of the different machine learning models of aneurysm rupture was evaluated internally (ie, on the same training data set) and compared. For this purpose, the AUC of the ROC, the true-positive rate (TPR), the false-positive rate (FPR), and the misclassification error were calculated. Pair-wise comparisons of these performance metrics between machine learning models were performed using the built-in summary function of the caret package available in R (based on a 1-sample t test) to find differences in the performance of various models and to determine whether these differences were statistically significant.
Second, the predictive models were externally validated on the independent validation data set 2 containing data from 102 aneurysms that were not used during training, parameter tuning, and model selection. In addition to the AUC, the accuracy of the model was assessed in terms of the true-positive rate (TPR or recall), FPR, positive predictive value (PPV or precision), negative predictive value, harmonic mean of precision and recall (F1 score), and balanced accuracy.
RESULTS
Patient and Aneurysm Characteristics Associated with Rupture
Statistical comparisons between patient and aneurysm characteristics among ruptured and unruptured small, regularly shaped aneurysms are presented in the Online Supplemental Data. Aneurysm rupture was significantly associated with patient sex (the ratio of males to females was higher in the ruptured group compared with the unruptured group, P = .01), age (patients with ruptured aneurysms were, in general, younger, P = .003), and population (the Finnish population had a higher proportion of ruptured aneurysms compared with the US and European populations, P < .001). In addition, aneurysm rupture was significantly associated with aneurysm multiplicity (most ruptured aneurysms were single aneurysms, P < .001), morphology (ratio of bifurcation to lateral aneurysm was higher in the ruptured group, P < .001), and location (the anterior communicating artery was the location with the higher proportion of ruptured aneurysms, P < .01).
Hemodynamic and geometric differences between ruptured and unruptured small, regularly shaped aneurysms are presented in the Online Supplemental Data. As shown in these data, most of the hemodynamic and geometric variables were significantly different between the 2 groups of aneurysms, even after adjustment for multiple testing.
Hemodynamically, ruptured aneurysms had stronger (Q, P = .02) and more concentrated inflow jets (ICI, P = .002) and more complex (corelen, P < .001) and unstable (podent, P < .001) flow patterns than unruptured aneurysms. They also had larger maximum wall shear stress (WSSmax) (P < .001; MaxWSSnorm, P = .003), more concentrated (SCI, P < .001) and oscillatory (OSImax, P < .001; OSImean, P < .001) WSS distribution, and a larger number of critical points of the WSS field (nr.critical, P < .001).
Geometrically, ruptured aneurysms were larger (Asize, P < .001; SR, P < .001; GAA, P < .001), more elongated (AR, P < .001; VOR, P < .001; BF, P < .001), and had larger shape distortion (CR, P < .001; NSI, P < .001; UI, P < .001) than unruptured aneurysms in this subpopulation.
Variables Retained in the Model
Thirty-one variables were used to build the models and select the optimal set of predictive features. The final model retained the following 14 predictive variables from the 3 different domains: 1) hemodynamic: corelen, WSS, OSImax, OSImean, nr.critical; 2) geometric: Asize, SR, AR, VOR, BF, NSI; and 3) aneurysm: location, multiplicity, and morphology. As illustrated in the Online Supplemental Data, accuracy reached the maximum level when 14 variables were retained in the model, with a noticeable decrease in accuracy beyond 14 variables. See the Online Supplemental Data for the complete list of the variables considered and retained.
Performance of Different Predictive Models
The best predictive model for internal cross-validation was the SVM, with a mean AUC = 0.85, TPR = 0.82, FPR = 0.26, and misclassification error = 0.23 (see Table 1 for performance metrics for all machine learning models considered, and the Online Supplemental Data to visualize the spread of their AUC, sensitivity, and specificity). Pair-wise comparisons of different performance metrics among machine learning models are presented in the Online Supplemental Data.
Summary of evaluation metrics for machine learning predictive models during repeated internal 10-fold cross-validation
External Validation
Table 2 presents the performance of different machine learning classifiers when applied to the independent external testing data set. It can be seen that the best performance was achieved by the SVM model, with consistently the largest AUC = 0.84 (95% CI, 0.80–0.89), TPR = 0.93, PPV = 0.30, NPV = 0.98, F1 score = 0.45, and balanced accuracy = 0.79, and the smallest misclassification error = 0.31. However, the random forest model had the smallest FPR = 0.32, slightly smaller compared with FPR = 0.35 of the SVM model. The performance of different machine learning models on the external data set is graphically presented in Fig 1, which shows the ROC and the precision/recall curves.
Performance measures for each machine learning model applied to the external testing data seta
Performance of different machine learning models on an independent, external validation data set: A, ROC curves. B, Precision/recall curves. The best performance is achieved by the SVM model (red curves). BG indicates bagging or bootstrap aggregating; RF, random forest; SVM, support vector machine; KNN, K-nearest neighbor; LR, logistic regression.
These results suggest that the SVM model was able to correctly identify 93% of small, regularly shaped aneurysms that ruptured. On the other hand, 35% of unruptured aneurysms were identified as at risk of rupture, which may result in overtreatment for these lower-risk aneurysms. Of all the aneurysms classified as unruptured, 98% were correctly classified. This finding indicates that the model is generalizable to independent data sets from different centers.
The most important variables (n = 14) for discriminating between small, regularly shaped aneurysms prone to rupture and aneurysms less likely to rupture were determined using an algorithm that excludes variables from the model, one at a time, and ranks the importance of the excluded variable on the basis of the decrement in the AUC of the ROC (as the measure of variable importance), in which a larger decrement indicates a more important feature. The most important variables (Fig 2) comprised geometric shape and size factors, flow complexity, and WSS parameters, as well as aneurysm multiplicity.
Variable importance for the SVM model applied to the validation data set. Variables are ordered from top to bottom according to their importance determined by the AUC of the ROC as the measure of variable importance. See the Online Supplemental Data for an explanation of the terms.
Examples of aneurysms from the external validation data set that were correctly classified by the SVM model are illustrated in the Online Supplemental Data. The upper panel of the Online Supplemental Data shows a small (<7mm) and regularly, shaped ruptured aneurysm at the anterior communicating artery. Visualizations illustrate the hemodynamic environment characterized by a strong inflow jet, elevated WSS, and a complex flow structure. The SVM model assigned a probability of 92% to this aneurysm being ruptured. The lower panel shows an unruptured, small, regularly shaped aneurysm at the anterior communicating artery and its hemodynamics characterized by a slow, smooth, and simple flow pattern with uniformly low WSS. This aneurysm was assigned a probability of 90% of not being ruptured. The upper panel of the Online Supplemental Data shows a small, regularly shaped ruptured aneurysm at the MCA and its hemodynamics environment characterized by a strong inflow jet, elevated and heterogeneous WSS, and a complex flow structure. The SVM model assigned a probability of 88% of this aneurysm being ruptured. The lower panel shows an unruptured small, regularly shaped aneurysm at the MCA and its hemodynamics, characterized by a simple flow pattern with fairly uniform WSS. This aneurysm was assigned a probability of 86% of not being ruptured. These examples highlight the generalizability of the predictive model.
DISCUSSION
The focus of this study was to identify patient and aneurysm characteristics associated with rupture of small and regularly shaped aneurysms; because this is a cross-sectional study, it is not the authors’ intention to establish causal associations. Several patient and aneurysm characteristics were different between ruptured and unruptured aneurysms of this subset. In particular, our results indicate that when restricting the analysis to small aneurysms (<7mm) without blebs, ruptured aneurysms have higher flow conditions characterized by high, concentrated inflow jets; complex, unstable flow patterns; and concentrated, complex, and oscillatory WSS distribution compared with unruptured aneurysms. Additionally, ruptured aneurysms were larger and more elongated and had larger shape distortion than unruptured aneurysms in this subpopulation. Furthermore, aneurysm rupture was significantly associated with patient sex, age, population, as well as aneurysm multiplicity, morphology, and location.
Machine learning models based on patient and aneurysm characteristics are capable of identifying rupture-prone small, regularly shaped aneurysms with an accuracy of approximately 79%, a sensitivity of 93%, and a specificity of 65%. This feature is important to improve the management of patients presenting with small, regularly shaped aneurysms. To interpret the meaning of these numbers, one should consider, for example, that in a sample of 100 small, regularly shaped aneurysms, approximately 18 would rupture (based on a rupture rate of 17.6% from our data base, which, of course, is higher than the annual rupture rates reported from highly selected series of longitudinally followed aneurysms17). On one extreme, one could decide to treat all aneurysms. In this case, all hemorrhages would presumably be avoided, but the expected number of complications would be approximately 10, assuming a 10% risk of complications.18,19 On the other extreme, if one decided not to treat any of these small, regularly shaped aneurysms, there would be no treatment complications, but a total of 18 hemorrhages. Even recognizing the inaccuracies of the predictive model presented here, which would correctly identify 17 of the 18 aneurysms prone to rupture and thus avoid their bleeding and misidentify as potentially risky another 29 that would probably not need intervention, this method could be useful for clinicians. For instance, with the use of the machine learning model, the number of expected treatment complications would be around 5 and the expected number of ruptures would be around 1, with a reduction in the total number of treatments to about 46 compared with the 100 treatments if one decided to treat everybody. Considering the high complication rate of unruptured cerebral aneurysm intervention, around 10% by any method, added to the low rupture rate of small, regularly shaped aneurysms, it is very important to identify those patients with aneurysms that rarely rupture for whom the intervention is probably unnecessary and thus the number of avoidable complications that lead to brain damage. These complications also have an impact on the health system because it is well-known that one of the most expensive items in attending cerebral aneurysms, both socially and economically, is the treatment of perioperative complications, which would be minimized.
Machine learning algorithms have recently gained attention for assessing rupture risk of cerebral aneurysms. One study20 used 12 morphologic features in addition to clinical and anatomic characteristics to predict aneurysm stability in a sample of 420 aneurysms of <8 mm (without excluding aneurysms with blebs) from a single institution and found an aneurysm irregularity metric to be the most important predictor. Although our models incorporate hemodynamic features and did not include some of their clinical parameters (eg, hyperlipidemia), the results are consistent and the predictive power in internal cross-validation is similar (AUC = 0.85 for both). Furthermore, our external validation indicates that our models are generalizable to other populations. Another study21 used cross-sectional data of 374 aneurysms <8 mm (18% of which were ruptured) from a single center to build machine learning models of rupture based on patient demographics, life behaviors, clinical histories, lipid profiles, and aneurysm morphology. Their predictive power in internal 10-fold cross-validation was slightly higher than ours (AUC = 0.88) and significantly higher than the PHASES score (as expected). While our findings are consistent with this study in that aneurysm irregularity, location, multiplicity, and size ratio are important predictors, 1 important difference is that our study considered only aneurysms without blebs, while in this other study, most ruptured aneurysms had blebs.
Another recent study22 proposed the use of fluid-structure interaction for aneurysm risk assessment, by first estimating regions of thin walls from computational fluid dynamics analysis and then using fluid-structure interaction to estimate wall strain for varying wall stiffness. Although this work made several important assumptions and was based on a small (but longitudinal) sample from a single institution, it proposes an intriguing approach. Our models included several hemodynamic features that were used in this study for estimating local wall thinning, which subsequently affects wall strain and rupture risk, and thus are expected to be important predictors as shown in our study. Additionally, more advanced machine learning and artificial intelligence algorithms have also been used to predict treatment outcomes after aneurysm clipping23 and for the detection and measurement of aneurysms on MR images,24,25 showing that these methods are quite effective for multivariate prediction in a wide variety of applications but need careful interpretation.
We believe that ultimately, the aneurysm risk assessment should be conducted in stages. For example, in the first stage, aneurysm size, shape irregularity, and patient characteristics (as in PHASES5 or unruptured intracranial aneurysm treatment score [UIATS]4) could be used to pick up the most dangerous aneurysms. In the second stage, additional anatomic, geometric, and hemodynamic parameters derived from precise diagnostic angiography images (3D DSA or CTA) could be used to identify rupture-prone aneurysms within the small, regularly shaped subset as in the current study. Perhaps in a third stage, further characteristics related to the status of the aneurysm wall (for example derived from MR vessel wall imaging and/or aneurysm wall-enhancement images) could be used to further identify unstable aneurysms that should be recommended for immediate treatment and those that could be safely monitored without treatment. Thus, aneurysm evaluation would be performed by first applying the existing scales based on demographics and basic imaging; second, performing a more in-depth aneurysm-specific analysis of shape and flow; and finally, studying the aneurysm wall with additional specialized imaging.
The current study has several limitations. In addition to the usual limitations of the computational fluid dynamics modeling approach (Newtonian flow, rigid walls, estimated flow rates, and so forth), in this study, the sample size was limited and there were only 14 ruptured, small, regularly shaped aneurysms in the external validation data set. Also, to deal with the inherent imbalance in the training data set, we performed down-sampling in the cross-validation process. Moreover, the internal and external validations were performed using retrospective cross-sectional data sets. A recent study26 showed that geometric and hemodynamic characteristics were not significantly different between unstable (growing or symptomatic) and already ruptured aneurysms but were significantly different from unruptured aneurysms, thus providing support to the assumption that predictive models based on cross-sectional data are useful to identify aneurysms at risk of destabilization and rupture. Nevertheless, further evaluation and validation with longitudinal data sets are required and will be the focus of future studies. Finally, in the construction of the predictive models of this study, important clinical variables that have been previously recognized as rupture risk factors such as hypertension, smoking, hyperlipidemia, and family background were omitted, possibly inducing some biases in the present work.
CONCLUSIONS
Hemodynamic conditions characterized by strong, concentrated inflow jets; complex, unstable flow patterns; and concentrated and oscillatory WSS patterns are associated with aneurysm rupture in small, regularly shaped cerebral aneurysms. Additionally, ruptured aneurysms are larger, more elongated, and have larger shape distortion than unruptured aneurysms in this subset. Predictive models based on aneurysm characteristics are capable of identifying small, regularly shaped aneurysms prone to rupture. This is an important finding because we could potentially understand which parameters predispose cerebral aneurysms in the subpopulation of small, regularly shaped aneurysms to rupture, and the findings could lead to improved patient selection for treatment or monitoring.
Footnotes
This work was supported by the National Institutes of Health grant R01NS097457.
Disclosure forms provided by the authors are available with the full text and PDF of this article at www.ajnr.org.
Indicates open access to non-subscribers at www.ajnr.org
References
- Received October 26, 2021.
- Accepted after revision January 20, 2022.
- © 2022 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- No citing articles found.