Abstract
BACKGROUND AND PURPOSE: The WEB Occlusion Score has been proposed to assess angiographic outcomes for intracranial aneurysms treated with the Woven EndoBridge (WEB) device. Using a large series of experimental aneurysms treated with the WEB, we had the following objectives: 1) to compare angiographic outcomes as measured by the WEB Occlusion Scale with histologic results, and 2) to assess interobserver and intraobserver agreement of the WEB Occlusion Scale.
MATERIALS AND METHODS: Intracranial aneurysms were created in 80 rabbits and treated with WEB devices. Animals were sacrificed at last follow-up for histologic evaluation. DSA was performed just after the deployment of the device and at follow-up. Four investigators independently and retrospectively graded the DSA twice according to the WEB Occlusion Scale. One histopathologist blinded to the angiographic results graded the occlusion according to a 4-point scale patterned on the WEB Occlusion Scale. Intra- and interobserver agreement were evaluated for DSA. Follow-up angiographic grading and histologic reference were compared to determine the WEB Occlusion Scale accuracy for complete (with or without recess filling) versus incomplete occlusion and adequate (complete occlusion or neck remnant) versus inadequate occlusion.
RESULTS: Inter- and intraobserver weighted κ for the angiographic WEB Occlusion Scale were, respectively, 0.76 and 0.76, indicating substantial agreement. The sensitivity and specificity of the WEB Occlusion Scale for complete occlusion at follow-up compared with the histologic reference standard were, respectively, 75% and 83.3%, with an overall accuracy of 80%. Similarly, for adequate occlusion at follow-up, sensitivity was 97.7%, specificity was 64.9%, and overall accuracy was 82.5%.
CONCLUSIONS: The WEB Occlusion Scale appears to be consistent, reliable, and accurate compared with a histologic reference standard.
ABBREVIATIONS:
- WEB
- Woven EndoBridge
- WOS
- WEB Occlusion Scale
The Woven EndoBridge (WEB) device (Sequent Medical, Aliso Viejo, California) is a new and promising tool in the treatment of wide-neck and bifurcation intracranial aneurysms.1⇓⇓–4 The endosaccular implant conforms to the aneurysm wall and spans the aneurysm neck to essentially function as a flow diverter at the aneurysm neck–parent artery interface. Initial experimental and clinical studies have demonstrated that the WEB is both safe and effective in the treatment of intracranial aneurysms.1⇓⇓⇓⇓⇓⇓⇓–9
Because the mechanism of action of the device differs from that in flow diverters and endosaccular coils, developing tools to assess angiographic occlusion has become a recent priority among many neurointerventionists, and several grading scales have been proposed.4,7⇓⇓–10 Perhaps the most commonly used scale is the WEB Occlusion Scale (WOS).7 However, none of these scales, including the WOS, has previously been validated with histologic studies, to our knowledge. In a large series of elastase-induced aneurysms in rabbits treated with the WEB device, the objectives of this study were the following: 1) to compare angiographic outcomes as measured by the WOS with histologic results, and 2) to assess interobserver and intraobserver agreement of the WOS.
Materials and Methods
In Vivo Experiments
The Institutional Animal Care and Use Committee at our Institution approved the animal procedures. Elastase-induced aneurysms were created in 80 New Zealand white rabbits. Aneurysm creation procedures were performed as previously described by our study group.11 Aneurysms were treated at least 3 weeks after aneurysm creation.12 No antiplatelet therapy was used before or after treatment. During the device-deployment procedure, animals were anesthetized, the right femoral artery was exposed, and a 5F sheath was inserted, followed by injection of 500 U of heparin through the sheath. A 5F guide catheter (Envoy; Codman & Shurtleff, Raynham, Massachusetts) was advanced into the aortic arch. Digital subtraction angiography was performed with contrast injection through the guide catheter. A microcatheter (Renegade Hi-Flo; Boston Scientific, Natick, Massachusetts) was advanced into the aneurysm lumen over a microguidewire (Transend EX; Stryker, Kalamazoo, Michigan) through the guide catheter. The WEB size was selected according to the aneurysm size. After deployment of the device, the microcatheter was removed and DSA was performed through the guide catheter at the brachiocephalic trunk immediately and 5 minutes following device implantation. Follow-up angiographic evaluation was performed at the sacrifice end point according to the WEB Occlusion Scale as described by Fiorella et al.7 Animals were sacrificed with a lethal injection of pentobarbital. Aneurysm and parent artery tissue were immediately fixed in 10% neutral buffered formalin.
Angiographic Evaluations
Four investigators independently and retrospectively examined selected images of the posttreatment and follow-up DSA to grade the occlusion status according to the WEB Occlusion Scale. These readings were performed twice by each of the investigators at 2-month intervals to analyze the intraobserver correlation in the readings. The WEB Occlusion Scale is a 4-point scale using the following grades: complete aneurysm occlusion, complete occlusion with recess filling, aneurysm neck remnants, and aneurysm remnants. The investigators also evaluated the modification of the aneurysm occlusion status between posttreatment and follow-up DSA as follows: improvement, stable, or recurrence. To compare follow-up DSA readings and histologic findings, we dichotomized DSA results as complete occlusion (with or without proximal recess filling) or incomplete occlusion (neck remnant or aneurysm remnant); similarly, we dichotomized DSA results as adequate occlusion (complete occlusion or neck remnant) or inadequate occlusion (aneurysm remnant), according to previous studies.8,13 In case of disagreement among readers, a fifth reader adjudicated between adequate or inadequate occlusion.
Histopathologic Processing and Analysis
One histopathologist blinded to the angiographic results did the processing and analysis for healing evaluation. Aneurysm samples were processed at 1000-μm intervals in a coronal orientation, permitting long-axis sectioning of the aneurysm neck, with use of an IsoMet Low Speed Saw (Buehler, Lake Bluff, Illinois). After the device segments were carefully removed under a dissecting microscope, the samples were then re-embedded in paraffin, sectioned at 5–6 μm, and stained with hematoxylin-eosin. The sections were evaluated by using our previously reported evaluation criteria.14
The histologic results for each aneurysm were evaluated according to a 4-point histologic scale, patterned on the angiographic WEB Occlusion Scale with the same items: complete aneurysm occlusion, complete occlusion with recess filling, aneurysm neck remnants, and aneurysm remnants.
Statistical Analysis
Statistical analysis was performed by using the statistical software package SAS 9.0 (SAS Institute, Cary, North Carolina). Interobserver and intraobserver agreement was assessed by using the quadratic weighted κ statistic. κ statistics for interobserver agreement were calculated between each observer. Mean κ values were calculated as well. The ANOVA intraclass correlation between readers was calculated. The sensitivity, specificity, and accuracy of DSA evaluations at last angiographic follow-up were calculated by using the histologic findings as a reference standard. These were calculated on a per-reading basis.
Results
Population
Eighty consecutively treated rabbits were included in this study. The length of follow-up varied from 30 to 365 days, with a mean length of follow-up of 101.4 days. Rabbits were sacrificed at day 30 (n = 27), day 50 (n = 5), day 90 (n = 30), day 180 (n = 12), or day 365 (n = 6). A Single-Layer WEB was used in 55% of cases (n = 44), and a Dual-Layer WEB was used in 45% of cases (n = 36).
Angiographic Results
Eight readings were performed for each of the 80 posttreatment and 80 follow-up DSAs (4 readers doing the same reading twice at 2-month intervals). Immediate posttreatment DSA showed 8.3% complete occlusion (n = 53/640 readings), 3.8% (n = 24) complete occlusion with recess filling, 12.7% (n = 81) neck remnants, and 75.3% (n = 482) aneurysm remnants. At last follow-up DSA, we observed 24.6% complete occlusion (n = 157/639 readings), 21.6% (n = 138) complete occlusion with recess filling, 30.8% (n = 197) neck remnants, and 23.0% (n = 147) aneurysm remnants.
The interobserver weighted κ coefficient for agreement of the occlusion grade according to the angiographic WEB occlusion scale was 0.76 (95% CI, 0.76–0.82), indicating substantial agreement among readers (range, 0.68–0.81). The intraobserver κ for the 2 sequential readings of the angiographic results was also substantial at 0.76 (95% CI, 0.72–0.81), ranging from 0.58 to 0.85. The ANOVA intraclass correlation among readers was 0.86 (95% CI, 0.81–0.90) for the first reading and 0.82 (95% CI, 0.76–0.87) for the second reading. The overall intraclass correlation score for both readings was 0.84 (95% CI, 0.79–0.88). Inter- and intraobserver agreement is presented in On-line Tables 1 and 2.
Variation of the aneurysm occlusion status between posttreatment and follow-up DSA showed 73.3% (n = 469/640) improvement, 16.6% (n = 106) stable occlusion, and 10.1% (n = 65) increase of opacification of the aneurysms compared with posttreatment DSA. The intra- and interreader κ was 0.73 (95% CI, 0.70–0.77) and 0.78 (95% CI, 0.71–0.84), respectively. These data are summarized in Table 1.
Inter- and intraobserver κ agreement for follow-up DSA readings
Complete versus Incomplete Occlusion
We dichotomized follow-up DSA results as complete versus incomplete occlusion. The interobserver κ at follow-up was 0.63 (95% CI, 0.56–0.70), ranging from 0.42 to 0.76; the intraobserver κ was 0.71 (95% CI, 0.55–0.88), ranging from 0.47 to 0.85.
To compare with histologic findings, a fifth reader was needed in 31.2% cases (25/80) to adjudicate between complete or incomplete occlusion. The adjudicated dichotomized DSA evaluation yielded 40% (32/80) complete occlusion and 60% (48/80) incomplete occlusion.
Adequate versus Inadequate Occlusion
We dichotomized DSA results as adequate occlusion (complete healing or proximal recess persistence) versus inadequate occlusion (neck or aneurysm remnants). The interobserver κ at follow-up was 0.69 (95% CI, 0.64–0.73), ranging from 0.50 to 0.79; the intraobserver κ was 0.75 (95% CI, 0.62–0.89), ranging from 0.55 to 0.85.
To compare the DSA results with histologic findings, we determined a consensus value to allocate in the adequate versus inadequate occlusion group, according to the most present value without need for a fifth reader to adjudicate. The consensus dichotomized DSA evaluation yielded 68.8% (55/80) adequate occlusion and 31.2% (25/80) inadequate occlusion.
Histologic Results
Histologic evaluation at the time of follow-up DSA depicted 18.8% complete healing (n = 15/80 aneurysms) and 21.2% (n = 17) proximal recess persistence; 13.8% of cases (n = 11) had aneurysm neck remnants, and 46.2% (n = 37) had aneurysm remnants. We observed, respectively, 40% (32/80) complete occlusions, 60% (48/80) incomplete occlusions, 53.8% (43/80) adequate, and 46.2% (37/80) inadequate occlusions.
Correlation between Angiographic and Histologic Results
Illustrative correlations between follow-up DSA and histology are presented in Figs 1 and 2.
Complete aneurysm occlusion. Angiographic-histologic correlation. Pretreatment DSA shows an aneurysm on the right subclavian artery at the origin of the right common carotid artery (A). Follow-up DSA shows complete occlusion of the aneurysm (B). An all-mount histologic sample shows complete occlusion of the aneurysm, filled with conjunctive tissue and with any lack of organized tissue in to the aneurysm cavity or at the level of the neck (C). Photomicrograph (hematoxylin-eosin, original magnification ×100) shows complete coverage of the neck with endothelial tissue with complete healing (D) and organized connective tissue within the aneurysm lumen.
Aneurysm remnant. Angiographic-histologic correlation. Pretreatment DSA shows an aneurysm on the right subclavian artery at the origin of the right common carotid artery (A). Follow-up DSA shows partial filling of the aneurysm sac (blue arrow, B). An all-mount histologic sample shows a lack of filling of the aneurysm by the conjunctive tissue (blue arrow, C). Photomicrograph (hematoxylin-eosin, original magnification ×100) shows a lack of coverage of the neck by endothelial tissue and filling of the aneurysm pouch with a partial thrombosis in the aneurysm sac (D).
Correlation for the Diagnosis of Complete Occlusion
We observed 24 (30%) cases for which DSA and histologic evaluations agreed for complete occlusion (complete occlusion with or without recess filling) and 40 (50%) cases for which DSA and histology agreed for incomplete occlusion. In 8 (10%) cases, DSA suggested a complete occlusion but histology identified incomplete, and similarly, 8 (10%) cases that had been misassessed as incomplete by DSA were found to have complete occlusion on histology (Table 2). Sensitivity and specificity of the DSA for the diagnosis of complete versus incomplete occlusion at follow-up compared with the histologic results as the reference standard were respectively 75% (95% CI, 56%–88%) and 83.3% (95% CI, 69%–92%). The overall accuracy of the DSA evaluation compared with the reference standard histology was 80% (95% CI, 65%–90%).
Correlations between the histologic reference standard and the follow-up DSA WOS evaluation: contingency table for complete versus incomplete occlusion
Correlation for the Diagnosis of Adequate Occlusion
We observed 42 (52.5%) cases for which DSA and histologic evaluations agreed for adequate occlusion (complete occlusion or neck remnant) and 24 (30%) cases for which DSA and histology agreed for inadequate occlusion. In 13 (16.2%) cases, DSA suggested an adequate occlusion but histology identified inadequate occlusion, and similarly 1 (1.3%) case was misassessed as inadequate by DSA but was found to have adequate occlusion on histology (Table 3). Sensitivity and specificity of the DSA for the diagnosis of adequate versus inadequate occlusion at follow-up compared with the histologic results as the reference standard were respectively 97.7% (95% CI, 86.2%–99.9%) and 64.9% (95% CI, 47.4%–79.3%). The overall accuracy of the DSA evaluation compared with the reference standard histology was 82.5% (95% CI, 73.8%–91.9%).
Correlations between the histologic reference standard and the follow-up DSA WOS evaluation: contingency table for adequate versus inadequate occlusion
Discussion
Using a histologic reference standard, our study performed on a large series of experimental aneurysms demonstrates that the angiographic WOS is sensitive, specific, and accurate in assessing aneurysm occlusion following treatment of intracranial aneurysms with the WEB device. In addition, we found a substantial level of inter- and intraobserver agreement for the WEB Occlusion Scale. These findings suggest that the WEB Occlusion Scale is an easily reproducible and accurate tool in assessing aneurysm occlusion following treatment with the WEB device.
Our current results are in accordance with the clinical study published by Fiorella et al,7 which reported a κ value statistic at 0.779 (95% CI, 0.70–0.86). Our study is the first to compare angiographic assessment of aneurysm occlusion after treatment with the intrasaccular WEB device with histologic controls. Assessment of occlusion is challenging following treatment with the WEB because the angiographic appearance differs from that of coiled or intraluminal flow-diverter-treated aneurysms.7 The proximal surface of the WEB is slightly recessed into the body of the device, thus forming a concave “marker recess” at the parent artery–aneurysm interface, to avoid any protrusion of the device in the parent artery. This marker recess opacification can be mistaken for residual filling of the aneurysm neck.
The WEB Occlusion Scale (WOS),15 based on the modified Raymond Scale, has been developed for the standardized reporting of angiographic occlusion assessment achieved with intrasaccular mesh implants, taking into account the distinction between recess opacification and aneurysm neck remnant. In this scale, complete aneurysm occlusion with or without opacification of the proximal recess is considered complete occlusion7; complete occlusion and neck remnant are considered adequate angiographic outcomes according to previous studies.13,15,16 This scale has been used in several previous studies and is also used in currently ongoing clinical trials: WEB Clinical Assessment of IntraSaccular Aneurysm Therapy (www.clinicaltrials.gov, NCT 01778322) and Wide Neck Bifurcation Intracranial Aneurysms; Intracranial Aneurysms (NCT 02191618). No other large-scale study has been published comparing histologic findings with DSA in the setting of the evaluation of aneurysm occlusion either for coiled or flow-diverter-treated aneurysms. However, several studies have reported that interobserver and intraobserver agreement rates for the assessment of aneurysm occlusion following both coiling and flow-diverter treatment are moderate to substantial.13,16⇓⇓–19 The degree of agreement seen in our study for the WOS is similar to that reported in prior studies of coiling and flow-diverter-treated aneurysms. At a time when consensus definitions for reporting angiographic outcomes following endovascular treatment of intracranial aneurysms are required, scales that are both easily reproducible and histologically validated are of the utmost importance.20,21
Limitations
Our study is limited by its retrospective nature and the use of only selected images for the DSA readers' assessment. Readers did not have access to the complete angiographic run when assessing angiographic occlusion. Furthermore, only 1 experienced reader evaluated the histologic samples. However, this reader was blinded to the DSA outcomes. Another limitation of this study is that rabbits were sacrificed at different time points, which can modify the outcomes after WEB implantation, depending on the length of follow-up.
Conclusions
This study confirms the consistency and reliability of the WEB Occlusion Scale for DSA evaluation of WEB-treated aneurysms with substantial interobserver and intraobserver agreement. Furthermore, the WEB Occlusion Scale appears to be accurate compared with a histologic reference standard, which is of great importance to justify its use in clinical studies for the evaluation of the WEB device.
Acknowledgments
We thank Ravi Lingineni, BST, from the Department of Health Sciences Research of Mayo Clinic, for his contribution to statistical analysis.
Footnotes
Disclosures: Waleed Brinjikji—UNRELATED: Grants/Grants Pending: Brain Aneurysm Foundation.* David F. Kallmes—RELATED: Grant: Sequent Medical,* Comments: preclinical research studies; UNRELATED: Board Membership: GE Healthcare (Cost- Effectiveness Board membership); Consultancy: ev3/Covidien/Medtronic,* Comments: planning and implementing clinical trials, Steering Committee membership; Grants/Grants Pending: MicroVention,* Codman,* Surmodics,* NeuroSigma,* ev3/Covidien/Medtronic,* Comments: preclinical research and clinical trials; Royalties: University of Virginia Patent Foundation, Comments: Spinal Fusion; Travel/Accommodations/Meeting Expenses Unrelated to Activities Listed: ev3/Covidien/Medtronic,* Comments: FDA panel presentation, open comments section. Ramanathan Kadirvel—UNRELATED: Grants/Grants Pending: National Institutes of Health.* *Money paid to the Institution.
References
- Received May 26, 2015.
- Accepted after revision July 14, 2015.
- © 2016 by American Journal of Neuroradiology