Observer Agreement in the Assessment of Endovascular Aneurysm Therapy and Aneurysm Recurrence

Materials and Methods

Results

The results are summarized in Tables 1 and 2.

Angiographic Completeness of Aneurysm Therapy

In general, scales with more allowable responses were associated with less agreement than scales with fewer allowable responses. The 4-response scale yields the least interobserver and intraobserver agreement for both the initial angiogram and the follow-up angiogram. The 3-response scale yields more agreement than the 4-response scale. The 2-response scale yields more agreement than the 3-response scale.

Intraobserver agreement, stated as κ, for follow-up angiograms assessed with the 4-response and 3-response scales ranged from 0.75 to 0.78, whereas for interobserver agreement, they ranged from 0.66 to 0.67. This indicates less interobserver than intraobserver agreement in the categorization of aneurysms when more choices are available in the scale.

For initial postprocedure angiogram assessments, the κ values for interobserver and intraobserver comparisons were 0.48 to 0.67 for all scales used. For follow-up angiogram assessments, the κ values for interobserver and intraobserver comparisons were 0.66 to 0.97 for all scales used. Thus, both interobserver and intraobserver agreement was better on follow-up angiograms than initial angiograms for all scales used. Examples of cases with discordant readings are shown in Fig 1.

• Download figure
• Open in new tab
• Download powerpoint

Fig 1.

Examples of observer variability in the assessment of completeness of cerebral aneurysm therapy. A carotid aneurysm before (A) and immediately after (B) endovascular therapy, which was assessed variably as “complete,” “dog ear,” and “incomplete.”

Discussion

Our study elucidates the observer variability inherent in the angiographic assessment of completeness of cerebral aneurysm therapy. With assessments of both initial and follow-up angiograms for completeness of aneurysm therapy, interobserver and intraobserver agreement was progressively worse with increasing response choices in the aneurysm assessment scale. Likewise, for the comparisons of the initial angiogram to the follow-up angiogram, there was worse observer agreement with the 3-response (“better, same, worse”) scale than with the 2-response (“not worse, worse”) scale. These findings are supported by previous work demonstrating that the value of κ tends to increase as the number of categories is decreased, thus indicating better agreement when fine distinctions are eliminated.¹³

There was less agreement on initial angiograms than on 3–6-month follow-up angiograms. This finding can be explained by a high degree of subjectivity in the assessment of areas within an aneurysm that still fill with contrast material immediately after endovascular aneurysm therapy. The interpreter is left to guess whether these areas will thrombose over the next few hours.

Our study used 2 readers of very similar training and practice background. Interobserver variability between readers with more varied backgrounds might be even higher.

The degree of aneurysm occlusion after endovascular therapy is really a continuous variable. Therefore, commonly used scales for grading the degree of aneurysm occlusion apply an ordinal scale on what is really a continuous variable. This system forces a reader to categorize an outcome into a discrete category, which is rather subjective and can contribute to observer variability (Fig 1). This practice evolved because it is quite difficult to accurately quantify the continuous variable of “completeness of aneurysm occlusion” using 2-dimensional images of irregular 3D objects. The creation of ordinal categories forces the image interpreter to subjectively categorize the completeness of aneurysm treatment into one of the categories based on 2-dimensional images. 3D angiographic imaging now available might provide a more quantitative assessment of aneurysm occlusion, but standardized performance of such 3D measurement techniques is not currently developed enough to be practical for use on all, or even most, of the angiographic equipment used at each of the medical centers that might participate in a multicenter study of aneurysm therapy. Thus, the use of ordinal scales for the assessment of completeness of aneurysm occlusion remains as an important tool that potentially allows for the comparison of treatment data from multiple medical centers and multiple studies.

Because cerebral aneurysm therapy is intended to prevent subarachnoid hemorrhage, the future risk of hemorrhage from a cerebral aneurysm is ideally what would be measured to judge the success of therapy. Because endovascular therapy is rather successful at preventing subarachnoid hemorrhage, hemorrhagic events in these patients are quite uncommon. With such a low event rate for subarachnoid hemorrhage after endovascular therapy, studies of very large numbers of patients over a number of years would be necessary for rate of subarachnoid hemorrhage to be a practical outcome measure. The International Subarachnoid Aneurysm Trial (ISAT)¹⁴ is a trial that has enrolled enough patients (1073 treated with coils) and follows them over a long enough period of time (at least 5 years) that it can make a meaningful assessment of the rehemorrhage rate. However, it is not practical for all studies of outcomes of endovascular cerebral aneurysm therapy to be so large. Thus, investigators use completeness of coil therapy and angiographic recurrence as surrogate markers for risk of future hemorrhage, making the assumption that risk of future hemorrhage correlates with degree of persistent filling of the aneurysm with contrast during angiography.

Another factor that makes assessment of degree of aneurysm occlusion important is that the completeness of aneurysm therapy at the time of treatment is somewhat predictive of subsequent risk of recurrence. For aneurysms that were angiographically completely treated, Murayama et al⁴ reported a recurrence rate of 1.1% for small aneurysms with small necks, 12.5% for small aneurysms with wide necks, and 20% in large aneurysms. For aneurysms with a neck remnant, the recurrence rates were 7.7% for small aneurysms with small necks, 29.4% for small aneurysms with wide necks, and 44.4% for large aneurysms. Raymond et al³ reported major recurrence rates of 9% for aneurysms initially completely occluded, 23% for aneurysms with residual neck, and 47% for aneurysms with residual aneurysm. Thornton et al¹¹ reported a recurrence rate of 1.8% for aneurysms that were “100%” occluded, and 26% for aneurysms with “residual neck.” As an aneurysm progressively recanalizes after endovascular therapy, the degree of protection of the patient from rehemorrhage is expected to decrease. Because the presence of a residual neck is predictive of future recanalization, it is a worthwhile distinction to make.

Scales with fewer observer responses may not identify aneurysm subgroups that have differing risks of recurrence and/or rehemorrhage. Without identification of lesser degrees of aneurysm remnant and recurrence, it would not be possible to ascertain such issues as the extent to which a minor recurrence predicts a higher risk of future hemorrhage and the risk that a minor recurrence will turn into a major recurrence. In choosing a scale for use in a trial of endovascular aneurysm therapy, a balance must be struck between reducing observer variability by offering fewer responses and improving identification of important subgroups by offering more responses.

Change in degree of aneurysm occlusion on follow-up angiography is a critical variable to record because it specifically assesses the major weakness of endovascular therapy relative to surgery (ie, aneurysm recurrence). Scales of degree of aneurysm occlusion can be deceptive with regard to recurrence, because an aneurysm may have a worsening degree of occlusion at follow-up angiography, yet it may not change categories on a grading scale. For example, an aneurysm treated and initially classified as “residual aneurysm” may show interval increase in the size of the residual aneurysm cavity at follow-up angiography and yet remain classified simply as “residual aneurysm”; ie, the degree of occlusion of the aneurysm changes, but the classification does not. Raymond et al³ defined a recurrence as “any increase in size of the remnant” and defined a recurrence as “major” if “it was saccular and its size would theoretically permit retreatment with coils.” Murayama et al⁴ defined “recanalization” as a “more than 10% increase in contrast filling of the aneurysm.” We chose to make the assessments as objective as possible by making the scales qualitative rather than quantitative.

One might suspect that an important marker of failure of aneurysm therapy is the need for retreatment of the aneurysm. The necessity of aneurysm retreatment, however, is not something that can be objectively measured as an end point for scientific research. The decision to retreat is quite subjective in many cases. The risk of treating relative to not treating a recurrent aneurysm can be quite difficult to ascertain in many cases, because the risk of future rupture of a given partially treated aneurysm is largely unknown. In our experience, recurrences are often more technically challenging to treat than the original, untreated aneurysm. Physicians vary greatly in terms of how aggressively they would treat an aneurysm recurrence with balloon remodeling, an adjunctive stent, or perhaps referral for surgery. Patients may refuse retreatment for a variety of reasons. The treating physician might have subjective reservations about further therapy based on experiences at the time of treatment (“you had to be there”) or based on intimate knowledge of the patient (“you have to know this patient”). The operating physician may be biased against retreating an aneurysm that he or she originally treated because of an unwillingness to admit failure of treatment. Conversely, a physician may be biased toward retreating an aneurysm that he or she originally treated because of a perception of the recurrence as a personal failure that he or she would like to confront. Ideally, such personal biases should be left out of patient care decisions, but they undoubtedly have some effect on management decisions. With so much uncertainty, bias, and subjectivity, simply tallying whether an aneurysm is retreated has little value in scientifically assessing the success of aneurysm treatment.

Scales to assess angiographic completeness of aneurysm therapy and recurrence will continue to be necessary for the development of new endovascular therapies. Such scales are of critical importance in assessing the efficacy of new aneurysm therapies in clinical trials. To minimize bias, assessment of angiogram studies of aneurysm treatment should be made by a central reader rather than the treating physician. Because of the potential for interobserver variability, especially with assessment scales with more than 2 responses, it may be useful to have more than 1 reader make the assessments and to have readers try to review difficult cases together to reach consensus.

Conclusions

Both interobserver and intraobserver variability are inherent to assessment scales of completeness of cerebral aneurysm therapy. Observer variability is substantially lower in scales that offer fewer observer responses. However, scales with fewer observer responses may not identify aneurysm subgroups that have differing risks of recurrence and/or rehemorrhage. These limitations must be considered when using such scales in clinical trials that assess the efficacy of endovascular cerebral aneurysm treatment.