Changing the Rules of the Game: The Problem of Surrogate Angiographic Outcomes in the Evaluation of Aneurysm Treatments

Surrogate outcomes (indicators or signs used in place of the true clinical outcome measure) are often used in the evaluation of neurovascular treatments, but there are many pitfalls.¹ The various ways surrogate angiographic outcomes are handled in our literature can be sources of confusion, particularly when new endovascular devices are introduced or when attempts are made to compare the results of different treatments.

One danger in focusing on a surrogate measure is losing sight of the goal of therapy: a good clinical outcome. In the case of aneurysm treatments, this means minimizing treatment-related morbidity and efficacy in the prevention of rupture. Angiographic outcomes are still important for at least 2 reasons. First, in clinical care, we cannot wait for ruptures to occur before we decide whether treatment was a success or a failure. Doctors need some indicator of the results of their actions so that they can, for example, offer a second treatment if need be. Second, and this is particularly true for unruptured aneurysms, the number of ruptures that occur during follow-up are few. An inordinately large number of patients followed for many years would be required for a randomized trial to show clinical outcomes are improved. A surrogate angiographic outcome measure can be obtained much sooner and can then be used to show superiority of a treatment with a smaller number of patients, provided the surrogate outcome truly predicts patients’ future clinical outcomes. Surrogate angiographic outcomes are thus often used in randomized controlled trials (RCTs) that compare different endovascular devices.^2⇓⇓-5

A famous example of the classic pitfall comes from cardiology: arrhythmias were known to cause death after myocardial infarction, and antiarrhythmic agents were found to decrease the incidence of arrhythmias,⁶ but they were then shown to cause more deaths!⁷ To avoid drawing the wrong conclusion about a treatment, clinical trials must always include hard clinical end points, and the “primary outcome” of a trial should be simple, clinical, and meaningful.⁸ For neurovascular treatments, the problem is that these larger trials are infrequently done.

One exception is the International Subarachnoid Aneurysm Trial (ISAT),⁹ which compared surgical clipping and coiling of ruptured aneurysms. The primary outcome was appropriately clinical (mRs ≥3 at 1 year), but it required the enrollment of 2143 patients. Angiographic outcomes were not recorded in ISAT, but it was known that the angiographic results of coiling were not as good or as durable, as advocates of surgical clipping complained: clipping achieved complete occlusion in 90% of cases compared with 45%–50% for endovascular treatment at best.¹⁰ The appropriate response to that concern is that the clinical results are what count; they were better with coiling in ISAT.⁹ In the meantime, because complete occlusions were infrequent with coiling, we learned to accept a residual neck as “acceptable” because rebleeding was rare compared with patients with residual aneurysms.¹⁰

Now flow diverters are introduced; how can the case be made that they should be used instead of coiling? A randomized trial with a clinical outcome measure, say, to ensure the complication rate of flow diversion was not double that of coiling (ie, 5% to 10%), would require at least 865 patients. So, we use a surrogate angiographic outcome. But because we now consider residual necks as “satisfactory” and we can obtain this result in 80%–90% of cases,¹¹ even this strategy won’t work: a trial of 1000–1500 patients would be necessary to show that flow diversion can improve already good results. What’s left to do? Well, what was done is that the threshold for what constitutes a good angiographic outcome was changed to no longer accept residual necks; in other words, the rules of the game were changed.

With this altered definition, flow diversion can be made to look good. Consider how simple it would have been to do an RCT with success defined as “complete aneurysm occlusion”: a trial of approximately 110 patients or so could have been sufficient to show that flow diversion improved complete occlusions rates from 50% to 75%. With a trial this size, of course, no one would be able to properly evaluate clinical outcomes: whether the improvement in the surrogate imaging finding (from near-complete to complete occlusion) was worth the added risk of flow diversion or whether it translated into better clinical outcomes in the long run. But was this simple trial even done? The unfortunate reality is that an RCT was not required to introduce flow diverters; flow diverter complete occlusion rates were simply “compared” with historical controls.¹²

The latest iteration of the changing-of-the-rules-as-you-go-along problem is to adapt the classification of angiographic results to the needs of the new device, as was recently done for the WEB device. If by design the device regularly leaves a residuum at its base, it is now proposed to consider that result akin to “complete occlusion.” “Adequate occlusion” is now a broad class that subsumes complete occlusions, including “complete occlusions with opacification of the recess, in addition to neck remnants need to close.^13,14 In this scheme, aneurysm necks are “acceptable” (post-ISAT), then “not acceptable” (to justify flow diversion), and then “acceptable” again (to justify WEB) (Figure). Advocates of surgical clipping especially should decry that the rules of the game keep on changing. Had complete occlusion consistently stayed the reason to choose one treatment over the other (without ever properly checking impacts on clinical outcomes as was done in ISAT), clipping would still be the treatment of choice for most aneurysm patients!

• Download figure
• Open in new tab
• Download powerpoint

FIGURE.

Changing the rules of the game.

One final related pitfall should be mentioned: attempts to compare treatments are further encumbered now because device-specific scales are used to evaluate angiographic results for each treatment.^13,15⇓-17 Using a different scale for every different treatment only ensures that the results of aneurysm treatments can no longer be compared.

Pitfalls notwithstanding, angiographic outcomes will continue to play an important role in the evaluation of neurovascular treatments. However, clinicians must remain wary of how they can be manipulated to show treatment results in a good light.

• © 2020 by American Journal of Neuroradiology