Abstract
BACKGROUND AND PURPOSE: Conventional angiography is the benchmark examination to diagnose cerebral vasospasm, but there is limited evidence regarding its reliability. Our goals were the following: 1) to systematically review the literature on the reliability of the diagnosis of cerebral vasospasm using conventional angiography, and 2) to perform an agreement study among clinicians who perform endovascular treatment.
MATERIALS AND METHODS: Articles reporting a classification system on the degree of cerebral vasospasm on conventional angiography were systematically searched, and agreement studies were identified. We assembled a portfolio of 221 cases of patients with subarachnoid hemorrhage and asked 17 raters with different backgrounds (radiology, neurosurgery, or neurology) and experience (junior ≤10 and senior >10 years) to independently evaluate cerebral vasospasm in 7 vessel segments using a 3-point scale and to evaluate, for each case, whether findings would justify endovascular treatment. Nine raters took part in the intraobserver reliability study.
RESULTS: The systematic review showed a very heterogeneous literature, with 140 studies using 60 different nomenclatures and 21 different thresholds to define cerebral vasospasm, and 5 interobserver studies reporting a wide range of reliability (κ = 0.14–0.87). In our study, only senior raters reached substantial agreement (κ ≥ 0.6) on vasospasm of the supraclinoid ICA, M1, and basilar segments and only when assessments were dichotomized (presence or absence of ≥50% narrowing). Agreement on whether to proceed with endovascular management of vasospasm was only fair (κ ≤ 0.4).
CONCLUSIONS: Research on cerebral vasospasm would benefit from standardization of definitions and thresholds. Dichotomized decisions by experienced readers are required for the reliable angiographic diagnosis of cerebral vasospasm.
Cerebral vasospasm remains a major cause of poor outcome after aneurysmal SAH.1 Timely identification of vascular constriction to avoid delayed cerebral ischemia can result in escalation of therapy and avoiding vasospasm-related strokes, disability, and death.2 Endovascular rescue procedures such as intra-arterial infusion of calcium channel blockers and balloon angioplasty can be used to dilate constricted vessels and prevent strokes3⇓-5 but these treatments carry a risk.6,7 The appropriate timing and indications for endovascular treatment in this setting remain unclear.1,8
Even though various noninvasive imaging methods have been proposed, conventional angiography remains the gold standard for evaluating whether a vessel is sufficiently narrowed to qualify as significant vasospasm and whether to proceed with endovascular treatment. However, the reliability of the diagnosis of vasospasm by conventional angiography has, in fact, rarely been studied.9 There are few reported reliability studies to date and usually in the context of comparing a novel, non- or less-invasive test with conventional angiography, with a small number of patients and a limited number of raters.9⇓⇓⇓-13
Here, we first systematically reviewed the literature on classification systems used to describe vasospasm after aneurysmal SAH using conventional angiography, with particular attention paid to classifications tested for interobserver reliability. We then sought to rigorously evaluate the reliability of the assessments made by a substantial number of interventionists of various backgrounds and years of experience regarding the degree of vasospasm and the perceived need for endovascular treatment in a substantial number of clinical cases.
MATERIALS AND METHODS
A detailed protocol for the literature search strategy (Online Supplemental Data) was designed and implemented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis statement.14 The article was prepared in accordance with the Guidelines for Reporting Reliability and Agreement Studies.15 The research ethics board at the University of Alberta provided consent to access patient clinical and angiographic data.
The electronic search was performed using the Cumulative Index of Nursing and Allied Health Literature, EMBASE, EBM, Cochrane Database of Systematic Reviews, and MEDLINE data bases with no start date specification, capturing English and French publications up to August 28, 2019. One author tested the ability of the search strategy to recover pertinent articles. The data were collected and reviewed in detail independently by 2 authors (B.F., T.E.D.), with 6 and 12 years’ experience, respectively, and discrepancies were resolved by consensus.
Patients
We assembled an anonymized portfolio of 221 patients with SAH hospitalized at the University of Alberta from January 2007 to December 2017, identified through a prospectively maintained transcranial Doppler case log. The case log excluded patients with high-grade aneurysmal SAH who were considered moribund. Patients who had undergone catheter cerebral angiography during the vasospasm risk period (3–17 days post-SAH) with injections of the left and right ICAs and at least 1 vertebral artery injection were included (257 patients). Thirty-six cases were subsequently excluded because ≥1 of the angiographic images was judged too poor to evaluate, leaving 221 cases for assessment of the degree of vasospasm. Demographic and aneurysm characteristics of the patients included in the portfolio, including the proportion actually treated with hypertensive therapy or balloon angioplasty for vasospasm, are presented in Table 1.
Characteristics of patients and aneurysms included in the portfolio
Clinician Responders
The angiographic portfolio was sent to selected clinicians from different training backgrounds, working in different countries, with differing levels of experience. Responders provided some demographic information but were assured of anonymity. Senior raters were those with >10 years’ experience in treating vasospasm. Raters were not provided with any clinical information regarding the case and were blinded to: i) the official radiology report, ii) whether the patient was receiving hypertensive therapy, iii) the scores given by the other responders.
For each of the 221 cases, clinicians were provided with anterior-posterior projections for 3 injections (left ICA, right ICA, and 1 vertebral artery) and were asked to provide their visual judgment on the degree of vessel narrowing at the level of the supraclinoid ICA, M1, A1, and basilar segments, for a total of 7 assessments per case (1547 assessments per responder per reading session). Caliper measurements were not required. Baseline angiographic images were available for comparison in 342/663 (51.6%) injections. The categories and thresholds were the following—none/mild vasospasm: <33% vessel narrowing; moderate vasospasm: 33%–50% narrowing; or severe vasospasm: >50% narrowing, according to a modified scheme used at our institution.13 Clinicians were then asked, for each case, if the patient is symptomatic without another cause, would you proceed with endovascular treatment of vasospasm? For the intrarater evaluation, raters were asked to independently evaluate the same-but-permutated portfolio twice, >4 weeks apart.
Finally, to assess whether a different defining threshold for severe vasospasm influenced agreement, we asked 4 clinicians to evaluate the same 221 cases 2 more times, at least 4 weeks apart using a different dichotomized threshold (not severe vasospasm: less than two-thirds narrowing; or severe vasospasm: equal or more than two-thirds vessel narrowing) and to consider only the ICA, M1, and basilar segments (1105 assessments per responder per reading session). An illustrative case from the portfolio is presented in Fig 1, and the questionnaire is available in the Online Supplemental Data.
Illustrative case from the portfolio. The degree of vessel narrowing was assessed on the right-sided Index images (A2, B2, C2).
Statistical Analysis
Results were analyzed for all raters according to the arterial segment, first using all 3 categories and then dichotomized into nonsevere/severe using 50% vessel narrowing as the threshold. This analysis was then repeated for the raters who responded to the portfolio with the dichotomized higher (two-thirds) narrowing threshold. An overall per-patient assessment of severe vasospasm was considered to have been made when the rater assessed narrowing to be “severe” for at least one of the ICA, M1, or basilar segments. Isolated A1 segment vasospasm was not included in this definition to avoid the uncertainty regarding pre-existent A1 hypoplasia. Responses were then stratified according to training background and years of experience.
Interrater and intrarater agreement regarding the degree of vasospasm for each arterial segment was assessed using Fleiss κ statistics for dichotomized data using 95% bias-corrected confidence intervals obtained with 1000 bootstrap resampling. For nondichotomized data, agreement was determined using the Krippendorff α. The strength of agreement was interpreted according to Landis and Koch (0–0.2, slight; 0.21–0.4, fair; 0.41–0.6, moderate; 0.61–0.8, substantial; 0.81–1.0 almost perfect). Differences between point estimates or between κ values were considered to exist when confidence intervals did not overlap. All analyses were performed by statisticians (M.C., J.Z.) using STATA, Version 16.0 (StataCorp) and SPSS Version 25 (IBM), with a significance level set at .05.
RESULTS
Systematic Review
A total of 1892 titles were reviewed, 683 abstracts were examined, 328 full-text articles were read in detail, and 140 eligible articles that used classification systems to evaluate cerebral vasospasm were included (flow chart, Online Supplemental Data). Articles described percentage luminal narrowing (91 articles), measured millimetric narrowing (11 articles), categorized the degree of vasospasm without quantification (9 articles), or did not categorize or quantify the degree of vasospasm (simple yes/no presence) (29 articles). Articles used a different number of tiers to grade the degree of vasospasm: 2 tiers (27 studies), 3 tiers (16 studies), 4 tiers (42 studies), 5 tiers (5 studies), and 6 tiers (1 study); no tiers were specified in 37 studies (Online Supplemental Data). Sixty different nomenclatures were used to define the degree of vasospasm. Measurement calipers (37 studies) or simple visual judgment (ie, eyeballing) (104 studies) were used to evaluate vessel diameter. Most (126/140, 89%) of the articles were diagnostic accuracy studies comparing how a second (noninvasive) imaging technique compared with conventional angiography. None of the published articles examined the same cases twice, with different thresholds to evaluate the consequence of changing thresholds on agreement.
The systematic review identified 5 interobserver variability studies, all of which had been published before the publication of the Guidelines for Reporting Reliability and Agreement Studies recommendations (2011).15 The 5 articles each used a different classification system to evaluate the degree of vasospasm (with 2–4 tiers) using various arbitrary thresholds (Online Supplemental Data). All included an assessment of interobserver agreement on cerebral vasospasm using conventional angiography, but this was the main focus of only one of the articles.11 The number of patients ranged from 22 to 101, and the number of raters, from 2–4. The degree of blinding of the raters was not reliably reported. The paucity of data and the heterogeneity of methods and end points precluded meta-analysis.
Agreement Study
The 221-patient portfolio included 52 patients (23.5%; 95% CI, 18.4%–29.5%) actually treated for vasospasm with vasopressor medications and 26 patients (11.8%; 95% CI, 8.1%–16.7%) who received balloon angioplasty for vasospasm. Six patients with high-volume subarachnoid hemorrhages were never found to have an aneurysm but were included in this series because they were considered at risk of developing symptomatic vasospasm.
The portfolio was sent to 20 clinicians, and 17 (85%) responded. All respondents (17/17) currently perform balloon angioplasty for cerebral vasospasm as part of their practice. Their training backgrounds were the following: interventional neuroradiology (n = 10), endovascular-trained cerebrovascular surgery (n = 4), and interventional neurology (n = 3). Raters were from Canada (n = 9), the United States (n = 5), or Europe (n = 3). Junior raters had ≤5 years’ (n = 3) or 6–10 years’ (n = 6) experience, while the senior raters had 11–20 years’ (n = 4) or >20 years’ experience (n = 4). Nine raters (53%) completed the survey twice (cases were permuted between the 2 reading sessions) >4 weeks apart for the intrarater evaluation. Four raters (24%) completed the survey 2 more times using the two-thirds vessel narrowing threshold for severe vasospasm, again >4 weeks apart.
The prevalence of severe vasospasm in the portfolio, as assessed by the 17 readers, ranged from 7.4% to 40.3% (mean, 26.3% [SD, 9.7%]). The nondichotomized interrater agreement for the diagnosis of severe vasospasm using conventional angiography is available in the Online Supplemental Data, while the dichotomized agreement results are presented in Table 2. Interpretation of the dichotomized (nonsevere/severe) data, according to Landis and Koch, showed moderate agreement for the supraclinoid ICA (κ = 0.568; 95% CI, 0.489–0.646), M1 (κ = 0.576; 95% CI, 0.517–0.629), A1 (κ = 0.511; 95% CI, 0.460–0.562), and basilar (κ = 0.533; 95% CI, 0.413–0.646) arterial segments. Having a training background in radiology, neurosurgery, or neurology did not significantly influence agreement, but responders with >10 years’ experience had significantly better agreement than junior readers when considering the presence of severe vasospasm in the M1 segment (κ = 0.638; 95% CI, 0.575–0.693 versus κ = 0.512 (95% CI, 0.444–0.574) and also agreed more on the per-patient assessment of severe vasospasm (κ = 0.629; 95% CI, 0.566–0.686) compared with junior readers (κ = 0.483; 95% CI, 0.414–0.552).
Interrater agreement for the per-patient diagnosis of severe vasospasm on DSA using 50% vessel narrowing threshold (dichotomized)
Perfect agreement (17/17 raters) on the presence of severe vasospasm in at least 1 arterial segment existed for 9 cases (4.0%). For 30/221 (14%) cases, 14/17 (80%) respondents considered the degree of vasospasm to be severe using a 50% narrowing threshold. Illustrative cases of maximal and minimal agreement for the diagnosis of severe vasospasm are presented in Fig 2.
Illustrative cases of maximal agreement and maximal disagreement. Two cases showing A) maximal agreement (17/17 readers rated vasospasm to be “severe” with 50% narrowing threshold), B) maximal disagreement (only 8/17 readers rated vasospasm to be “severe”).
The interobserver agreement for all raters on whether to proceed with endovascular treatment if the patient were symptomatic without another cause was fair (κ = 0.39) and was not significantly better when considering years of experience or training background. There was a strong association between the determination of severe vasospasm and the choice to proceed with endovascular treatment (OR = 20.57; 95% CI, 16.3–25.89; P < .001). Patients who actually underwent angioplasty in real life obtained “yes” votes for endovascular treatment 348 of 406 (86%) times, while patients who did not undergo angioplasty obtained 1450/3326 (44%) yes votes (P < .000). The assessments of the 17 readers on the 221 cases are ordered and presented in Fig 3.
Total number of dichotomized assessments of severe vasospasm and choice for endovascular treatment. Along the x-axis, the graph shows patients primarily ordered according to increasing proportion of severe vasospasm votes. Patients are secondarily ordered within each bracket according to number of votes for endovascular treatment. Circles show patients who received angioplasty in real life. Note that perfect agreement occurs only at the tails of the distribution: cases 1–92 (all no) and 212–221 (all yes).
Nine clinicians answered the survey twice to obtain intraobserver agreement (Online Supplemental Data). Intraobserver agreement for the dichotomized evaluation of whether the patient had severe vasospasm was at least substantial for all raters (κ = 0.639–0.802). The range of intraobserver agreement on the decision of whether to proceed with endovascular management was at least substantial for 7 of 9 raters, varying from moderate (κ = 0.464) to almost perfect (κ = 0.828).
To test the impact of changing the threshold definition of severe vasospasm from 50% to 66.6% vessel narrowing, 4 readers completed the survey 4 times each, twice with the 50% threshold and twice with the two-thirds threshold. For these 4 readers, altering the threshold definition did not significantly change the agreement; interobserver agreement was substantial using 50% and remained at a substantial level using the two-thirds narrowing threshold for carotid and M1 segments and for per-patient assessments as well (Online Supplemental Data). The range of intrarater agreement was also substantial (κ = 0.723–0.784) at 50% and from moderate-to-almost-perfect agreement at two-thirds (range, κ = 0.596–0.834) (Online Supplemental Data).
DISCUSSION
The present work specifically addresses the angiographic diagnosis of vasospasm, which is only 1 aspect of the physiologic derangements that occur after aneurysmal SAH and which are collectively referred to as delayed cerebral ischemia/vasospasm.1,2 The pathophysiologic association between the caliber of cerebral vessels and delayed cerebral ischemia was historically made with conventional angiography, a method that remains the gold standard, even if angiography is currently increasingly replaced by various noninvasive diagnostic alternatives.9,10,16 The diagnosis of vasospasm on conventional angiography is, nevertheless, beset by a number of unresolved problems. There is no consensus in the literature on the diagnostic criteria for angiographic vasospasm, and our systematic review showed such heterogeneity in the way cerebral vasospasm is defined and evaluated that the comparison of results from 1 article to the next is difficult or impossible.
One major difficulty is that there is no way to verify whether the results of conventional angiography are accurate: There is no additional gold standard with which angiography can be verified, unlike appendicitis, for example, for which the accuracy of a sonographic diagnosis can be verified by surgical exploration. The best that can be done is to evaluate the reliability of the evaluations obtained with conventional angiography. However, because angiography often serves as the gold standard to evaluate other noninvasive tests, this evaluation becomes crucial. It is, thus, somewhat surprising that reliability has been so infrequently assessed, and never with the benefit of modern computing power and statistical techniques.11 One explanation could be that in a study dedicated to the diagnostic accuracy of a new technique, there is little incentive to put the gold standard conventional angiography to a severe test.
We thus endeavored to address this gap by assembling a large number of cases and recruiting a large number of clinicians, hoping to establish that the best test of vessel narrowing was actually reliable. However, the reliability of the diagnosis of severe vasospasm remains borderline for a number of methodologic and practical clinical reasons. Variability in intracranial vessel diameters is inevitably present across individuals, and baseline catheter angiograms in the same patient are not always available for comparison (ie, for clipped aneurysms which are treated on the basis of CT angiography). The determination of spasm can sometimes be made only by comparison with a noninvasive test or through a comparison with ipsilateral or contralateral presumably uninvolved arteries. Well-defined, precise measurement protocols, including which vessels should be measured (and how), remain elusive.
Second, whether readers can reliably differentiate 25% or 30% vessel narrowing is questionable, and even if precise and reliable measures were possible, there is no consensus on what baseline reference values should be used on which or how many vessels. The reasons for the various grading scales and the exact procedural methodology are frequently not clearly stated or validated, with some authors choosing to measure vessel diameter with calipers, while others use simple visual assessment.
Our findings have a number of important implications for vasospasm research. Correlative research on alternative (noninvasive) diagnostic tests such as transcranial Doppler,13 CTA,10,17,18 or MRA18 that commonly use conventional angiography as the comparative benchmark examination should bear in mind the suboptimal reliability of the gold standard diagnostic test. All by itself, the lack of reliability of the diagnoses may explain how pharmacologic treatments such as nimodipine could be shown to be clinically effective despite the lack of a measurable effect on angiographic vessel narrowing;19 this explanation further reinforces the notion that hard clinical outcomes should be selected for clinical trials on vasospasm.19⇓⇓⇓-23
The most important questions following conventional angiography pertain to the clinical relevance of the interpretation. The clinical decision of whether to proceed with endovascular management of vasospasm is based primarily on the clinical and neurologic status, but it is also closely related to the judged degree of angiographic vasospasm, particularly for sedated or comatose patients. Currently, the use of conventional angiography is often restricted to patients already identified by noninvasive tests, and interpretation is often biased toward intervening. We attempted, in this survey, to eliminate any decisional variability due to differences in clinical status by not mentioning when the patients were already being treated with vasoactive drugs and by asking responders to consider, for each case, that the patient was symptomatic without another cause. This word choice may have influenced readers to proceed with endovascular treatment even when the degree of vasospasm was judged not severe. The wording of the questions may also have influenced results because the choice of whether to proceed with endovascular management could have been variously interpreted as including infusion of vasoactive medications or balloon angioplasty, which may have different thresholds for decisions to treat.
Limitations of this work include the artificially constructed portfolio and the self-selected readers. Different cases and readers could have produced different results. The choice of 50% and two-thirds thresholds for severe vasospasm were arbitrarily determined after a review of the literature. We did not ask readers to differentiate focal from diffuse vasospasm or to fastidiously evaluate spasm in more distal vessels such as the M2, A2, or P2 segments because we were mostly interested in an evaluation of vessel segments that are more reliably examined by noninvasive methods such as transcranial Doppler and CTA and are safely treatable with balloon angioplasty. The representative images evaluated by the raters were only 3 still images; rating the entire dynamic angiographic study on a PACS system may have led to different results. Finally, evaluating a portfolio of images and making a case-by-case evaluation of real patients are different contexts, and we can only hope that readers took the same amount of time and care to respond as they would in a normal clinical setting.
To improve the ability to make meaningful comparisons regarding the degree of vessel narrowing in future research, standardization of thresholds and nomenclatures is desirable. We propose that core labs using angiographic vasospasm as an end point measure should use senior raters using a dichotomized assessment (presence or absence of severe vasospasm defined as >50% luminal narrowing) limited to the supraclinoid ICA and M1 segments.
CONCLUSIONS
Cerebral vasospasm research would benefit from standardization of definitions and thresholds. Dichotomized decisions by experienced readers are required for the reliable angiographic diagnosis of cerebral vasospasm.
Footnotes
Disclosures: Anne-Christine Januel—UNRELATED: Consultancy: Congress reporting for Balt. Jean-Christophe Gentric—UNRELATED: Consultancy: Balt, Medtronic, MicroVention, Stryker; Stock/Stock Options: INTRADYS.
References
- Received July 9, 2020.
- Accepted after revision September 24, 2020.
- © 2021 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- Botulinum Toxin Application for Treatment of Graft Vasospasm: A Reverse Translational Study
- Reply:
- CTA Supplemented by CTP Increases Interrater Reliability and Endovascular Treatment Use in Patients with Aneurysmal SAH
- Reply:
- Transcranial Doppler Velocities and Angiographic Vasospasm after SAH: A Diagnostic Accuracy Study