Abstract
BACKGROUND AND PURPOSE: Both the Meyer and Raymond scales are commonly used to report angiographic outcomes following coil embolization of intracranial aneurysms. The objectives of this study were the following: 1) to assess the interobserver agreement of the Meyer and Raymond scales, and 2) to evaluate and compare their performance in predicting major recurrence at follow-up.
MATERIALS AND METHODS: A retrospective series of 120 coiled aneurysms was included. Four investigators independently graded DSA images immediately posttreatment and at follow-up according to the Meyer and Raymond scales. On follow-up DSA, readers also evaluated recurrence outcome. Interobserver agreement was assessed via the intraclass correlation coefficient. The ability of posttreatment Meyer and Raymond scales to predict major recurrence was modeled by using logistic regression and assessed by using receiver operating characteristic analysis.
RESULTS: For the Meyer scale, interobserver intraclass correlation coefficients were 0.58 (95% CI, 0.46–0.68) on posttreatment and 0.78 (95% CI, 0.72–0.83) on follow-up evaluations. For the Raymond scale, interobserver intraclass correlation coefficients were 0.50 (95% CI, 0.39–0.61) and 0.69 (95% CI, 0.62–0.76), respectively, for posttreatment and follow-up. The areas under the curve for the receiver operating characteristic analyses regarding the performance to predict major recurrence at follow-up were 0.69 (95% CI, 0.60–0.79) for the Meyer and 0.70 (95% CI, 0.61–0.78) for the Raymond scale.
CONCLUSIONS: The Meyer scale appears consistent and reliable with observer agreement as high or higher than that of the Raymond scale. Performance of both scales in predicting the risk of major recurrence at follow-up is adequate, with no statistical difference between the scales.
ABBREVIATIONS:
- ICC
- intraclass correlation coefficient
- ROC
- receiver operating characteristic
The most commonly used grading scale to assess aneurysm occlusion after coiling is the Raymond and Roy scale.1 This grading scale divides the arteriographic outcomes of endovascularly treated saccular aneurysms into 3 categories: 1) complete occlusion, 2) residual neck, and 3) residual aneurysm.1 However, despite its simplicity, its application in large clinical registries demonstrated important interobserver variability, which may limit its application and efficacy as an outcome-assessment tool.2 Moreover, important variability in aneurysm morphology and occlusion may occur within the same category by using this 3-category scale. To standardize the reporting on the degree to which volumetric occlusion has been achieved, a new consensual 6-point grading system, the Meyer scale, has been proposed jointly by multiple societies across neurointerventional radiology, neurosurgery, and neurology.3⇓⇓–6 This proposed scale, published in 2012, was put forth to standardize the reporting of the degree of volumetric occlusion and to properly assess technical outcomes in a standardized report. The rationale for this new scale was that the commonly used Raymond scale demonstrated important interobserver variability that may limit its application and efficacy as an outcome-assessment tool.2 However, despite the necessity for high reproducibility and interobserver rates, this new grading scale has never been evaluated, to our knowledge. As stated by the consensus writing group, there is no literature support for the proposed grading system, though this writing group believes it is the best available grading scale and offers a greater likelihood of reporting additional degrees of aneurysm recurrence.3⇓⇓–6
In addition, none of the scales have been validated as predictive of aneurysm recurrence.7,8 Possible risk factors for reopening of a coiled aneurysm with time, such as the initial degree of occlusion,9 seem to have an influence on the likelihood of recurrence, with subtotal initial treatment leading to remnant regrowth.10,11
Using a large retrospective series of patients with coiled aneurysms, the objectives of this current study were the following: 1) to assess interobserver agreement of the Meyer scale, and 2) to evaluate its performance in predicting recurrence at follow-up and compare the results with those of the Raymond scale.
Materials and Methods
Study Cases
Mayo Clinic institutional review board approval was obtained for this retrospective study. Records for patients treated with endovascular coil embolization for either ruptured or unruptured aneurysms at our institution were evaluated. Some of the cases were used in a prior publication, but there is no overlap in scope with the current article.12 These images of 120 cases of coiled aneurysms were compiled, de-identified, and placed into a 120-page PDF file. Each page included a DSA of the initial uncoiled aneurysm if available, the final postcoiling working-projection DSA, and the follow-up DSA. The PDF file was then distributed to 4 interventional neuroradiologists in different international centers. Two readers had 6 years' experience, and 2 readers had 4 years' experience.
Angiographic Evaluations
The Meyer scale is a 6-point grading scale based on the percentage of the aneurysm filled by contrast on DSA. Grade zero indicates complete and total aneurysm occlusion without remnant or interstitial filling within the aneurysm. Grade 1 represents >90% volumetric occlusion of the aneurysm based on planar imaging assessment; grade 2, 70%–89% aneurysm occlusion; grade 3, 50%–69%; grade 4, 25%–49%; and grade 5, <25% volumetric aneurysm occlusion. Two other “modifier” criteria are combined with this scale: The modifier “I” may be used to describe interstitial opacification within the confines of the coil mass and the modifier “G” may be used to describe interval growth in the overall dimensions of the aneurysm as a separate phenomenon from coil compaction with recurrence of the aneurysm.
The Raymond scale is a commonly used outcome grading scale that divides the angiographic outcomes of endovascularly treated saccular aneurysms into 3 categories: complete occlusion, residual neck, and residual aneurysm.1 Complete occlusion and residual neck are considered adequate occlusion; and residual aneurysm, inadequate occlusion.13⇓⇓–16
The 4 investigators independently and retrospectively examined each set of images of the posttreatment and follow-up DSAs to grade occlusion status according to the Raymond scale, the Meyer scale, and the modifier criteria. The investigators also evaluated the recurrence status of the aneurysm at follow-up DSA as follows: no recurrence, minor recurrence, or major recurrence. Major recurrence was defined as a theoretically or technically retreatable aneurysm and was considered an unfavorable outcome.
To evaluate the performance of posttreatment DSA evaluation to predict the risk for major recurrence, we used a consensus rating based on the most common grade among all readers. In case of disagreement among readers, 2 readers (A.R., W.B.) did a consensus reading.
Statistical Analysis
Reader agreement was assessed via the intraclass correlation coefficient (ICC) by using the ICC method (2,1), as specified by Shrout and Fleiss,17 for each of the 7 outcomes (posttreatment and follow-up Raymond scale, Meyer scale, Meyer modifier criteria, and recurrence status at follow-up). For the ICC description, we used the verbal description proposed by Portney and Watkins18: 0–.2, poor agreement; 0.3–0.4, fair agreement; 0.5–0.6, moderate agreement; 0.6–0.7, substantial agreement; 0.7–0.8, strong agreement; and >0.8, almost perfect agreement.
Aneurysm recurrence at follow-up was modeled as a function of either the Raymond scale or the Meyer scale (including the Meyer modifier criteria). Univariable (for Raymond) or multivariable (for Meyer) logistic regression by using the Firth penalized likelihood was performed by using the consensus rating of favorable or unfavorable recurrence at follow-up as the outcome. Aneurysm recurrence was modeled and receiver operating characteristic (ROC) curves were constructed both for each of the 4 raters individually and by using a consensus rating for each scale.
Statistical analyses for ICC were performed in SAS (Version 9.3; SAS, Cary, North Carolina). All other analyses were performed in R (Version 3.1.1; http://www.r-project.org/). Penalized logistic regression was performed by using the logistf package (version 1.21; https://cran.r-project.org/web/packages/logistf/logistf.pdf). ROC, area under the curve, and confidence interval estimates were obtained by using the pROC package (Version 1.7.3; http://www.rdocumentation.org/packages/pROC/functions/pROC-package).
Results
Posttreatment Results
With the Raymond scale, we classified 165 (34%) of 480 readings as complete occlusion, 163 (34%) as neck remnants, and 152 (32%) as aneurysms remnants. The ICC for the Raymond scale was 0.50 (95% CI, 0.39–0.61), indicating moderate agreement among readers. Using the Meyer scale, we classified 159 (33%) of 480 readings as complete occlusion, 152 (32%) as grade 1, 112 (23%) as grade 2, 43 (9%) as grade 3, 11 (2%) as grade 4, and 3 (1%) as grade 5. The ICC for the Meyer scale was 0.58 (95% CI, 0.46–0.68), indicating moderate agreement among readers. Using the modifier criteria, we classified 213 (44%) of 480 readings as having interstitial opacification (modifier I). The ICC was 0.36 (95% CI, 0.26–0.47), indicating fair agreement among readers (range, 0.24–0.56).
Follow-Up Results
The mean time between treatment and follow-up DSA was 37.8 months, with a median of 30.5 months (range, 0.37–166.0 months). Using the Raymond scale, we classified 138 (29%) of 480 readings as complete occlusion, 147 (31%) as neck remnants, and 195 (41%) as aneurysms remnants. The ICC for the Raymond scale was 0.69 (95% CI, 0.62–0.76), indicating substantial agreement among readers. Using the Meyer scale, we classified 126 (26%) of 480 readings as complete occlusion, 107 (22%) as grade 1, 128 (27%) as grade 2, 82 (17%) as grade 3, 29 (6%) as grade 4, and 8 (2%) as grade 5. The ICC for the Meyer scale was 0.78 (95% CI, 0.72–0.83), indicating strong agreement among readers. Using the modifier criteria, we classified 113 (24%) of 480 readings with the interstitial opacification modifier (modifier I) and 35 (7%) with interval aneurysm growth (modifier G). The ICC was 0.37 (95% CI, 0.27–0.47), indicating fair agreement among readers.
Assessment of Recurrence Status
The evaluation of the recurrence status yielded 43.8% (n = 210/480 readings) of cases with no recurrence, 20.6% (n = 99) with minor recurrence, and 35.6% (n = 171) with major recurrence (unfavorable outcome). The ICC was 0.72 (95% CI, 0.66–0.79), indicating strong agreement among readers. ICCs are presented in the Table.
Interreader agreements
After consensual assessment, dichotomized evaluation yielded 65.0% (n = 78/120 cases) good outcomes (no or minor recanalization) and 35.0% (n = 42/120) bad outcomes (major recanalization).
Prediction of Recurrence According to the Posttreatment Angiographic Evaluation
For the Raymond and Meyer evaluations, individual readers' ROC evaluations with areas under the curve are presented in Fig 1, and the ROC curve for the model by using the consensus reading is presented in Fig 2. According to the consensus reading, the area under the curve for the model using the Raymond scale was 0.70 (95% CI, 0.61–0.78). The area under the curve for the model using the Meyer scale was 0.69 (95% CI, 0.60–0.79). The confidence interval estimates overlap, and both of the scales have a fair performance to predict major recurrence.
ROC curves per reader for the prediction of unfavorable outcome at follow-up (major recurrence) according to the posttreatment Raymond and Meyer scale evaluations.
ROC curves for the prediction of unfavorable outcome at follow-up (major recurrence) according to the consensus posttreatment Raymond and Meyer scales evaluations.
Discussion
Our current study demonstrated that the Meyer scale has moderate interreader agreement immediately after treatment and strong agreement at follow-up, and for both immediate posttreatment and follow-up, the agreement for the Meyer scale is comparable with that of the Raymond scale. Furthermore, the performance of the Meyer and Raymond scales for predicting major recurrence risk, based on immediate posttreatment results, was fair and similar between scales. These data indicate that the Meyer and Raymond scales have similar performance and consistency levels for aneurysm occlusion evaluation and prediction of a major recurrence.
The interobserver agreement in our study for the Raymond scale is higher than that previously reported by Tollard et al,19 who reported a fair κ statistic at 0.276. Interobserver agreement statistics for the Raymond scale ranged from 0.28 to 0.83 in prior studies.19⇓⇓⇓⇓–24 Most interesting, the Meyer scale agreement score was higher than that of the Raymond scale despite the higher chance for better agreement when grading with fewer responses.8 This outcome is likely because the Meyer score allows a little more flexibility in the assessment of angiographic occlusion. For example, in a case in which an aneurysm is well-occluded with the exception of filling in the interstices, some may mark the status as complete occlusion on the Raymond scale and others may mark the status as aneurysm remnant. However, with the Meyer scale, a visual assessment of the percentage of angiographic filling may be more reproducible. The interstitial opacification criteria have previously been taken into account in the recently introduced modified Raymond scale, with a higher performance in predicting recurrence at follow-up.20
Beyond assessing an adequate or inadequate operator result, grading aneurysm occlusion at the end of the coiling is of interest for predicting the long-term outcome and managing follow-up. Subtotal initial treatment leads to a higher likelihood of potential bleeding risk.10,11 Raymond et al22 reported major recurrence rates of 9% for aneurysms initially completely occluded, 23% for aneurysms with a residual neck, and 47% for aneurysms with residual aneurysm. Thornton et al25 reported a recurrence rate of 1.8% for aneurysms that were 100% occluded and 26% for aneurysms with a residual neck. To our knowledge, this study is the first to evaluate the Meyer scale for predicting the risk of major recurrence at follow-up. Both the Raymond and the Meyer scales, combined with the interstitial opacification criteria, have a fair performance level in predicting the risk of unfavorable outcome at follow-up. This is important because in cases of an inadequate posttreatment angiographic results, given the higher risk for major recurrence, physicians could plan to perform surveillance angiography earlier or more frequently.12,19,26,27 Further studies are needed to determine a cutoff value for adequate-versus-inadequate posttreatment occlusion according to the Meyer scale. In our study, we focused on posttreatment angiographic grading, but this single criterion is not the only one that influences the recurrence outcome. Other factors such as aneurysm size,9,22 smoking status,28,29 and rupture status22 should be taken into account to have a model for predicting recurrence.
Limitations
Our study is limited by its retrospective nature and the use of only selected images for the DSA readers' assessment. Readers did not have access to the complete angiographic run when assessing angiographic occlusion. Another limitation of this study is that DSA was performed at different time points, which can modify the outcomes, depending on the length of follow-up.22
Conclusions
This study highlights the consistency and the reliability of both the Meyer and Raymond occlusion scales for DSA evaluation of coiled aneurysms. Both scales have substantial interobserver agreement and fair performance levels in predicting major recurrence at follow-up. These findings are very important in managing patients at follow-up.
Footnotes
Disclosures: Giuseppe Lanzino—UNRELATED: Consultancy: Covidien.* David F. Kallmes—UNRELATED: Grant: Sequent Medical,* Comments: preclinical research; Support for Travel to Meetings for the Study or Other Purposes: Sequent Medical,* Comments: travel to training for a clinical trial; UNRELATED: Board Membership: GE Healthcare,* Comments: Cost-Effectiveness Board; Consultancy: ev3/Medtronic,* Comments: Steering Committee membership for a clinical trial; Grants/Grants Pending: MicroVention,* Surmodics,* Sequent Medical,* Codman,* NeuroSigma*; Royalties: University of Virginia Patent Foundation (Spine Fusion); Travel/Accommodations/Meeting Expenses Unrelated to Activities Listed: ev3/Medtronic,* Comments: travel to FDA panel meeting. *Money paid to the institution.
References
- Received July 2, 2015.
- Accepted after revision September 18, 2015.
- © 2016 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- No citing articles found.