Abstract
BACKGROUND AND PURPOSE: Independent evaluation of angiographic images is becoming widely applied in the assessment of treatment outcomes of cerebral aneurysms. In the current study, we assessed the agreement between an independent core laboratory and the operators regarding angiographic appearance in a recent randomized, controlled trial.
MATERIALS AND METHODS: Data were derived from the Cerecyte Coil Trial. Angiographic images of each coiled aneurysm, taken immediately after embolization and at 5- to 7-month follow-up, were evaluated by the operator at the treating center and by an independent neuroradiologist at the core laboratory. For the purpose of this study, images were interpreted on a 3-point scale to provide uniformity for analysis; grade 1: complete occlusion, grade 2: neck remnant; and grade 3: sac filling. “Unfavorable angiographic appearance” was defined as grade 3 at follow-up or interval worsening of grade between the 2 time points.
RESULTS: The study included 434 aneurysms. Immediately after embolization, grade 3 was reported by operators in 39 (9%) compared with 52 (12%) by the core laboratory (P = .159). On follow-up, grade 3 was reported by operators in 44 (10%) compared with 81 (19%) by the core laboratory (P < .0001). Overall, operators noted unfavorable angiographic appearance in 78 (18%) compared with 134 (31%) by the core laboratory (P < .0001). At every time point, agreement between the core laboratory and the operators was slight.
CONCLUSIONS: Unfavorable angiographic appearance was noted almost twice as frequently by an independent core laboratory as compared with the operators. Planning of trials and interpretation of published studies should be done with careful attention to the mode of angiographic appearance interpretation.
Angiographic imaging represents the principal metric for evaluating endovascular treatment of cerebral aneurysms and comparing the efficacy of different coil types. The relationship between complete or stable aneurysm occlusion and incomplete occlusion and the risk of delayed aneurysm re-bleeding remains uncertain. Angiographic results are also used to decide on whether further treatment is considered. Treatment success and indications for further management, including the possibility of an additional intervention, are highly dependent on angiographic appearance in the belief that patients with incomplete occlusion are at risk of hemorrhage, though the size of this risk is unknown. Most previous literature included assessments of angiographic appearance on the basis of readings performed in the treating center(s).1⇓–3 However, other investigators,4⇓⇓⇓⇓⇓⇓–11 especially in the setting of randomized, controlled trials,9⇓–11 have used an independent core laboratory for angiographic assessment. Independent core laboratories represent a centralized approach and can provide a standard operating procedure for imaging review. In this respect, they are used to provide a more objective assessment within a trial setting than may be possible at treating centers. This suggested objectivity may also result from the core laboratory being blinded to treatment settings, using the same prespecified systematic reviewing protocols, and usually consisting of more experienced reviewers. On the other hand, operators, being aware of the details of the patient status and the procedure, may have a hands-on sense of “success” when considering the risk-benefit ratio of adding more coils to the treated aneurysm. Also, there is currently no study addressing the degree of objectivity of operators judging their own procedural results at the end of prolonged risk procedures.
Reported angiographic appearance after endovascular treatment has varied widely in previous studies.12⇓⇓–15 Many factors are known to influence angiographic appearance, including aneurysm size, neck diameter, initial rupture status, and follow-up duration.10,16⇓–18 However, the influence of core laboratory interpretation on outcomes for research purposes, compared with operator evaluations at the treating centers, remains poorly studied.19,20 In the current study, we assessed the agreement between an independent core laboratory and the operators at the treating centers regarding angiographic outcomes in a recent randomized, controlled trial.
Materials and Methods
All data were derived from the Cerecyte Coil Trial.9,21 The trial compared patients with either a ruptured or unruptured aneurysm treated with bioactive Cerecyte coils (Micrus Endovascular, San Jose, California) or bare platinum coils. Details about patient enrollment and inclusion/exclusion criteria are described elsewhere.9,21 Briefly, patients with 1 aneurysm deemed suitable for endovascular treatment were enrolled into the trial and randomly assigned to either receiving Cerecyte or bare platinum coils. There were 23 participating centers worldwide. Aneurysms were imaged by DSA immediately after embolization and by either DSA or MR angiography at 5–7 months of follow-up. Each imaging session was evaluated by the operator and by an independent core laboratory. The core laboratory consisted of a single, experienced neuroradiologist who did not have access to the center evaluations and was blinded to the type of coil used. In a similar fashion, the operators at the treating centers were blinded to the core laboratory assessment. Images of the studied aneurysms were sent to the core laboratory reader on compact discs for assessment without further details about the procedure other than vessel location of the target aneurysm. More details about the process of sending images are described elsewhere.9
Angiographic appearance was evaluated at the core laboratory and each of the treating centers by use of predefined scales that are summarized in Table 1. Operators evaluated angiographic images immediately after embolization and at follow-up by use of a 4-point scale, including 1) complete occlusion; 2) subtotal occlusion or neck remnant; 3) incomplete occlusion/aneurysm filling; and 4) failed embolization. The core laboratory used a slightly modified 4-point scale immediately after embolization and another one at follow-up. The core laboratory scale immediately after embolization included 1) apparently complete occlusion; 2) overlapping coils/neck; 3) neck remnant; and 4) inflow/incomplete occlusion (body filling). The core laboratory scale at follow-up included 0, no change compared with immediate postembolization DSA; 1, complete occlusion; 2, stable neck appearance/neck remnant; and 3, residual filling. For the purposes of this study, and after confirming with the core laboratory reader about the definitions of the scale points, we considered “complete occlusion” from operator readings to be equivalent to “apparently complete,” “overlapping coils/neck,” or “complete occlusion” from core laboratory readings (grade 1 for the purposes of the current study); “subtotal occlusion or neck remnant” to be equivalent to “neck remnant” or “stable neck appearance/neck remnant” (grade 2 for the purposes of the current study); and “incomplete occlusion/aneurysm filling” to be equivalent to “inflow/incomplete (body filling)” or “residual filling” (grade 3 for the purposes of the current study). When the core laboratory reported that there was “no change” in the angiographic appearance on follow-up, the aneurysm was given the same grade of the core laboratory readings in the immediate postembolization assessment. “Failed embolization” cases were excluded from all analysis as well as cases with incomplete imaging data. Last, we defined “unfavorable angiographic appearance” as either grade 3 at follow-up or interval worsening of grade between the immediate postembolization angiogram and the follow-up imaging.
Definitions of the scales of assessment used
Statistical Analysis
A simple κ coefficient and a 95% CI were used to assess agreement between the core laboratory reader and the operators in evaluating angiographic appearance. Kappa scores were interpreted as follows22: <0 as “less than chance agreement,” 0.01–0.20 as “slight agreement,” 0.21–0.40 as “fair agreement,” 0.41–0.60 as “moderate agreement,” 0.61–0.80 as “substantial agreement,” and 0.81–0.99 as “almost perfect agreement.”
The McNemar test of symmetry was used to assess the marginal homogeneity of the proportion of grade 3 immediately after embolization and at follow-up as well as the proportion of aneurysms identified as having had an “unfavorable appearance” for both the operators and core laboratory readings. All statistical analysis was performed with the use of SAS software (v9.3, SAS Institute, Cary, North Carolina).
Results
Patients (n = 497) with 1 confirmed aneurysm suitable for coil embolization were enrolled in the trial. Of these 497 target aneurysms, 49 (9.9%) were excluded from the angiographic analysis because of incomplete imaging data and 14 (2.8%) were also excluded as the operators deemed them as “failed embolizations.” Thus, the current analysis included 434 aneurysms (87.3%) of 497 treated aneurysms that were reviewed immediately after embolization and at 5–7 months of follow-up by both the operators and core laboratory reader.
Angiographic appearance of the treated aneurysms immediately after embolization is presented in Table 2. Of the 395 (91%) of 434 aneurysms graded either 1 or 2 by the operators, 49 were graded as 3 by the core laboratory reader. The overall number of aneurysms graded as 3 by the operators was 39 (9%) of 434 aneurysms compared with 52 (12%) graded as 3 by the core laboratory reader (P = .159).
Immediate postembolization angiographic appearance on the 3-point scale
Angiographic appearance of the treated aneurysms at follow-up is presented in Table 3. Of the 390 (90%) of 434 aneurysms that were graded as either 1 or 2 by the operators, 57 (16%) were graded as 3 by the core laboratory reader. The overall number of aneurysms graded as 3 by the operators was 44 (10%) of 434 aneurysms compared with 81 (19%) graded by the core laboratory reader (P < .0001).
Follow-up angiographic appearance on the 3-point scale
Of the 356 aneurysms that had “favorable angiographic appearance” according to the operators, 86 (24%) had “unfavorable angiographic appearance” according to the core laboratory reader. The overall “unfavorable angiographic appearance” according to the operators was 78 (18%) of 434 aneurysms compared with 134 (31%) according to the core laboratory reader (P < .0001).
Agreement between the core laboratory reader and the operators by use of the simple κ coefficient is presented in Table 4. At every time point, agreement between the core laboratory and the operators was slight.
Agreement between operators and the core laboratory reader on different time point assessments
Discussion
In the current study, we have shown that interpretation angiographic appearance for the same treated aneurysms can significantly change when reviewed by an independent core laboratory versus the treating operators. At follow-up, the rates of both unfavorable angiographic appearance, as well as those deemed as grade 3, were nearly twice as high for the core laboratory compared with the operator assessments. Agreement between the core laboratory and operators was only “slight” after embolization and at follow-up. This appears to be far worse than the “good” agreement among readers in a single-center study.23 These findings suggest not only that single-center studies with angiographic appearance assessed by operators are fundamentally different from those assessed by independent core laboratories but also that outcomes assessed at the treating centers are likely to be systematically more favorable than those from core laboratories, even for the same angiograms. The core laboratory recognized that some post-coiling angiograms appear complete but have coils overlapping aneurysm necks due to the view needed to separate parent vessel from coils not being available. An example is some anterior communicating aneurysms best viewed with a basal submental vertex view, but not done at the time of coiling.
The impact of core laboratory evaluation of outcomes has previously been studied in cardiology and its superiority over operator readings suggested.24⇓⇓⇓–28 A recent systematic review and meta-analysis of the endovascular literature for cerebral aneurysms19 found that studies that used an independent core laboratory reported higher numbers of unfavorable angiographic outcomes as compared with treating center evaluations, but the quality of evidence was low for the heterogeneity of the literature studies. The current study, which allowed direct comparison of individual cases from a randomized cohort instead of relying on literature reviews, adds additional evidence for the impact of core lab readings on reported outcomes. Another group of investigators20 has shown that core laboratory evaluation of immediate angiographic images resulted in doubling the number of incompletely occluded aneurysms when compared with treating center evaluation, similar to our own findings.
The current study had some limitations. First, to achieve uniformity of the reviews as much as possible between core laboratory reader and operators at the treating centers, we had to exclude 13% of the aneurysms for the various reasons that were listed earlier. We could not standardize the level of experience and former training of the core laboratory reader with the individual operators and also account for the variations between readers from different centers. Also, the Cerecyte Coil Trial was not set up to answer the specific question of comparing the core laboratory and operator readings. One of the resulting challenges of the differently oriented design was the ambiguity of the boundaries between the grades of each scale that was used. The possible misperception of the scale definitions among readers can, in part, explain some of the resulting differences. The core laboratory reader of the Cerecyte Coil Trial included, for every case, a derivation of “millimeter” measurements and measured neck remnant depth establishing 2 mm as an attempted measured boundary between grade 2 and grade 3 instead of distinguishing them in the eye of the beholder. This strategy was not applied by the operators in the treating centers. The core laboratory reader received a limited amount of information about the individual cases on compact discs, which may affect the overall understanding of the procedure's details and possibly affect the subsequent judgment. Finally, this study assesses 1 core laboratory with a single reader, which reduces the generalizability of the conclusions. A future study would ideally recruit different core laboratory readers to assess for interreader variability.
Conclusions
Unfavorable angiographic appearance was noted almost twice as frequently by an independent core laboratory as compared with the operators at the treating centers for aneurysms treated with coil embolization. The planning of trials and interpretation of published studies should be done with careful attention to the mode of angiographic appearance interpretation. More specific attention to the consistent use of the categories of less-than-complete aneurysm obliteration may reduce the discrepancies.
Footnotes
Disclosures: The chief Investigator for the Cerecyte Coil Trial was Dr Andrew J. Molyneux. The trial was sponsored and funded (unrestricted research grant) by Micrus Endovascular but was independently co-ordinated by the Oxford Neurovascular & Neuroradiology Research Unit (ONNRU), Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK. The Core Laboratory was led by Dr Allan J. Fox at Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada. ONNRU hold all the trial data and the data for this report was provided by ONNRU on request. ISRCTN82461286. Mary Sneade—RELATED: Support for Travel to Meetings for the Study or Other Purposes: Micrus Endovascular,* Comments: Travel and accommodation support for site visits as part of the Cerecyte Coil Trial protocol and meetings for the study set-up and data presentation. Andrew J. Molyneux—RELATED: Grant: Micrus Endovascular,* Comments: Unrestricted research grant; Support for Travel to Meetings for the Study or Other Purposes: Micrus Endovascular,* Comments: Attendance at ASNR Annual Meeting. Allan Fox—RELATED: Grant: Micrus Endovascular. Comments: Angiographic Core Laboratory for the Cerecyte Coil Trial; Support for Travel to Meetings for the Study or Other Purposes: Cerecyte Coil Trial, Comments: Travel expenses provided for participation in trial workshops; OTHER RELATIONSHIPS: Was chair of the Clinical Events Committee for the MAPS trial (randomized, parallel, control trial) organized and run by Boston Scientific. Time for work and meetings was paid on an hourly basis. David Kallmes—RELATED: research support Micrus for enrolling patients in the Cerecyte Clinical Trial. No funding was provided for any work related to the current paper focused on Core Laboratory outcomes. UNRELATED: Board Membership: Deputy Editor for Radiology*; OTHER RELATIONSHIPS: Research support: Microvention (eV3) Inc, Covidien, AG, NFocus Consulting Inc, Sequent Medical Inc, Penumbra Inc, Benvenue Medical, UVA patent (*money paid to institution).
REFERENCES
- Received February 21, 2013.
- Accepted after revision March 28, 2013.
- © 2014 by American Journal of Neuroradiology
Jump to section
Related Articles
Cited By...
- A core-lab adjudicated analysis of single-stent assisted coiling of wide-neck bifurcation aneurysms
- Evaluation of flow diverters for cerebral aneurysm therapy: recommendations for imaging analyses in clinical studies, endorsed by ESMINT, ESNR, OCIN, SILAN, SNIS, and WFITN
- Endovascular treatment for isolated posterior cerebral artery occlusion stroke in the MR CLEAN registry
- Five-year results of randomized bioactive versus bare metal coils in the treatment of intracranial aneurysms: the Matrix and Platinum Science (MAPS) Trial
- Low-Profile Intra-Aneurysmal Flow Disruptor WEB 17 versus WEB Predecessor Systems for Treatment of Small Intracranial Aneurysms: Comparative Analysis of Procedural Safety and Feasibility
- WEB embolization versus stent-assisted coiling: comparison of complication rates and angiographic outcomes
- Safety and efficacy of the Derivo Embolization Device for the treatment of ruptured intracranial aneurysms
- Safety and efficacy of the Derivo Embolization Device for the treatment of unruptured intracranial aneurysms: a multicentric study
- Wide-neck bifurcation aneurysms of the middle cerebral artery and basilar apex treated by endovascular techniques: a multicentre, core lab adjudicated study evaluating safety and durability of occlusion (BRANCH)
- Voxel based analysis of recurrence dynamics in intracranial aneurysms after coiling
- China Intracranial Aneurysm Project (CIAP): protocol for a prospective cohort study of interventional treatment and craniotomy for unruptured aneurysms
- GEL THE NEC: a prospective registry evaluating the safety, ease of use, and efficacy of the HydroSoft coil as a finishing device
- How safe and effective are existing treatments for wide-necked bifurcation aneurysms? Literature-based objective performance criteria for safety and effectiveness
- Inter- and Intrarater Agreement on the Outcome of Endovascular Treatment of Aneurysms Using MRA
- HydroCoils Reduce Recurrence Rates in Recently Ruptured Medium-Sized Intracranial Aneurysms: A Subgroup Analysis of the HELPS Trial
- Is Visual Evaluation of Aneurysm Coiling a Reliable Study End Point?: Systematic Review and Meta-Analysis
- Patients Prone to Recurrence after Endovascular Treatment: Periprocedural Results of the PRET Randomized Trial on Large and Recurrent Aneurysms
- Effects of Circle of Willis Anatomic Variations on Angiographic and Clinical Outcomes of Coiled Anterior Communicating Artery Aneurysms