METHODS

Patient selection

Patients with symptomatic ischaemic cerebrovascular disease who were suitable for carotid endarterectomy were seen prospectively in our neurovascular clinic by a stroke physician or neurologist and assessed clinically. They were examined with Doppler ultrasound (DUS), the ultrasonographer being blinded to the side and nature of the symptoms. A detailed report was completed at the time of examination, documenting the degree of stenosis from velocity criteria and arterial appearance.⁶ Those with a tight stenosis on DUS (70–99% reduction in lumen) in the symptomatic carotid artery were admitted for confirmation of this with DSA. CTA and MRA were also performed during the admission if the patient gave consent. Thus the study population consisted of all patients known to have at least one symptomatic tight carotid stenosis as demonstrated by ultrasound during a one year period.

Imaging techniques

CTA and MRA were performed within about 24 hours of DSA. The time between DUS and the other investigations varied depending on bed availability. All imaging equipment was less than three years old.

Doppler ultrasound

DUS was performed by one of three operators (two consultant neuroradiologists and one neuroradiographer) on an Acuson 128XP10v with a 7.5 MHz probe (Acuson Inc, Mountainview, California, USA). The peak systolic and end diastolic velocities were recorded routinely from the internal and common carotid arteries and the vertebral arteries, and the peak systolic velocity from the external carotid arteries. The degree of stenosis was determined by comparison with standard velocity tables⁶ in conjunction with the images of the arteries, all documented and reported at the time of examination.

Digital subtraction intra-arterial arteriography

DSA was done with a GE Advantex system (GE Medical Systems, Milwaukee, Illinois, USA). Both common carotid arteries were selectively catheterised using the Seldinger technique by the femoral approach. Three standard views (lateral and bilateral 45° oblique) of the carotid bifurcation were obtained, with additional views in ambiguous cases. For DSA, stenosis measurements were made on the projectional image showing the maximum stenosis.

CT angiography

CTA was performed on one of two spiral CT scanners (Somatom Plus and GE CTi scanners, Siemens, Erlangen, Germany). Following a test bolus to determine the time to optimal contrast enhancement, 100 ml of non-ionic contrast (Niopam 300 mg/ml) was injected at 3 ml/s, and the neck scanned from C2 to C7 using 3 mm collimation (pitch 1.0, small FOV (field of view), 120 kV, 260 mA). Axial images were reconstructed at 1 mm increments throughout the scan volume. All axial images were hard copied, with additional maximum intensity projection (MIP) and surface shaded display (SSD) images obtained in the majority of cases. For CTA, measurements were made on magnified axial images at the level of maximum stenosis.

MR angiography

MRA was performed on one of two systems (Siemens Magnetom 63SP 1.5T and GE Impact 1.0T). Both two dimensional time of flight (TOF) and three dimensional TOF sequences were obtained at the level of the carotid bifurcations. Twelve 15° MIP images of each carotid bifurcation were generated for each sequence and printed out to hard copy film. For MRA, measurements were made on the projectional MIP image showing the maximum stenosis. If a flow gap was present at the point of maximum stenosis, then “flow gap” was recorded.

Stenosis measurement

The DSA, CTA, and MRA imaging were each read by two of a group of six radiologists experienced in the assessment of the carotid bifurcation (four consultant neuroradiologists, one consultant radiologist in cross sectional imaging, and one neuroradiology fellow). Each mode was read by the two observers, blind to all other clinical and imaging data and to each other's results. The presence of any carotid disease was noted and the maximum percentage stenosis measured, using the common carotid artery (CCA) diameter at a non-atheromatous level as the denominator and the narrowest stenosis as the numerator to calculate the percentage stenosis (this has been shown previously to be the most reliable method for determining the degree of stenosis on DSA).⁷ All measurements were made on hard copy images using a watchmaker's high resolution measuring eye piece.

Data analysis

All data were entered into a database (Microsoft Access) and used to compute the interobserver variability of CTA, MRA, and DSA. Flow gaps on MRA were coded as 90% stenosis. We could not determine interobserver variability prospectively for DUS in this series as there was only one operator per patient. However, this was determined in a separate study.⁸ For the comparison of CTA, MRA, and DSA, where there was more than a 10% difference between the observers' readings of a particular scan the imaging was reassessed by the two observers and a consensus value for the percentage stenosis obtained. For other less discrepant values, the average of the readings by the two observers was calculated. The values from DSA were taken to be the reference standard.

In patients who underwent all three non-invasive proceedings as well as DSA, we compared the MRA, CTA, and DUS readings with the DSA readings by two methods. First, the data were dichotomised into operable carotid disease (80% to 99% stenosis)¹ and non-operable carotid disease (0–79% stenosis, or occlusion) and analysed using the κ statistic, which assesses agreement beyond that expected by chance alone. A κ of 0.8 to 1.0 indicates excellent agreement, 0.6 to 0.79 good, 0.46 to 0.59 moderate, 0.2 to 0.39 poor, and 0 to 0.19 no agreement. Symptomatic arteries were analysed separately from the contralateral asymptomatic arteries, as we were interested in the accuracy per patient, not per artery. Second, the stenosis values for CTA, MRA, and DUS were compared as continuous variables with DSA using scatterplots. The interquartile ranges of the differences between the two observers for each procedure were also calculated.

To determine the effect of substituting non-invasive tests for DSA on outcome, we then used the European carotid surgery trial (ECST) graph¹ of expected change in life expectancy following endarterectomy (which is split by age, sex, and percentage stenosis) to calculate the effect on life expectancy of performing CTA, MRA, and DUS instead of DSA. If the non-invasive procedure could have changed the decision on whether a patient underwent endarterectomy (according to DSA), then the change in life expectancy resulting from that decision was noted.

We also assessed the accuracy of a strategy of using two non-invasive tests combined compared with only one in the patients who had all four procedures.

Patient preferences

Following discharge after DSA, the patients were sent a postal questionnaire (appendix) asking about which clinical procedure (CTA, MRA, or DSA) they most and least preferred. Their opinion of DUS was not sought as it was considered to be a standard part of their carotid assessment. Also, DUS was performed some weeks before CTA/MRA/DSA so it would have been more difficult to remember.

RESULTS

Seventy three patients were entered into the study after Doppler examination, but only 67 (92%) actually had DSA. Fifty nine (81%) had CTA and 47 (64%) had MRA (fig 1). Thirty four patients (47%) had all four imaging procedures. Reasons for not performing the procedures are shown in table 1. The most likely procedure to be missed because of patient refusal or contraindications was MRA (10%), followed by DSA (5%); CTA (3%) was the least likely not to be performed. Adverse reactions occurred during 10% of DSA procedures, but not during any other procedure (table 1). These accounted indirectly for a further 6–10% of patients not undergoing MRA or CTA.

The median time lapse between DUS and the other three imaging techniques was 33 days (range 27 to 185 days), whereas CTA and MRA were all performed within 24 hours of DSA.

Interobserver variability

In the symptomatic arteries, the two observers for DSA agreed with each other's categorisation of operable/non-operable in 79% of patients (κ = 0.6). For MRA it was 79% (κ = 0.52) and for CTA, 76% (κ = 0.48) (table 2). The interquartile ranges for the differences between the two observers were: −0.08 to +4.78 for DSA, −4.08 to +10.57 for MRA, and −0.60 to +9.70 for CTA (fig 2A). In the asymptomatic arteries, there were only three with stenosis of 80% to 99% (that is, operable had they been symptomatic) and therefore the analysis using the κ statistic was very unreliable (table 3 and fig 2B). We did not test intraobserver agreement for CTA, MRA, or DSA in the present study for several reasons—it is unlikely to be worse than interobserver agreement, it would have doubled the amount of work, and it would have been difficult to blind adequately.

The interobserver agreement for DUS was assessed in a separate substudy and is reported in full elsewhere.⁸ In this substudy, the κ for operable v non-operable stenosis for the two observers who examined the most patients with Doppler ultrasound (n = 206) was 0.62—comparable with the agreement between two observers for the operability of symptomatic stenosis, and better than that for asymptomatic stenosis on DSA in the present study (see below). On DUS, the median difference in per cent stenosis between the two observers was 0%, the interquartile range was also zero, and the minimum and maximum differences were −55% and +65%, respectively. The population in the DUS observer study,⁸ however, included many fewer patients with operable stenosis (11/206, 5%) than in the present study (20/35 on the symptomatic side (57%)), and more similar to our findings on the asymptomatic side (3/34, 9%). We have not routinely assessed intraobserver reliability on DUS owing to the problem of blinding the observer to the result of their first examination.

Imaging accuracy – individual tests

In the symptomatic arteries, DUS agreed with the DSA categorisation of operable/non-operable in 79% of patients (table 4), MRA agreed in 82%, and CTA agreed in 79%. Most discrepancies in the assessment of eligibility for endarterectomy occurred in patients with 75–80% stenosis on DSA (fig 3A). Compared with DSA, CTA underestimated the degree of stenosis (sensitivity 0.65, specificity 1.0), MRA tended to overestimate the degree of stenosis (sensitivity 1.0, specificity 0.57), and DUS fell between the two (table 4 and fig 3A).

In the asymptomatic arteries, there was a greater proportion of lesser degree stenosis, and the few operable stenoses (had they been symptomatic) rendered the sensitivity calculations unreliable (table 5 and fig 3B).

If the decision to proceed to endarterectomy for symptomatic carotid stenosis were based purely on the degree of stenosis without taking any clinical features into consideration (which would of course not happen in clinical practice), then using CTA instead of DSA would have resulted in no unnecessary operations but seven missed operations (21% inappropriate management). For DUS, four unnecessary operations would have been performed, and three operations missed (21% inappropriate management), and for MRA, six unnecessary operations would have been performed and no operations missed (18% inappropriate management).

However, if risk factors for stroke after TIA other than just the degree of stenosis are taken into account (that is, the patient factors of sex and age), as defined by the ECST graph,¹ 15% of patients having DUS, 12% of patients having MRA, and 21% of patients having CTA would have received inappropriate treatment. This translates into a change in outcome (life expectancy) of one month lost with DUS and MRA, and two months lost for CTA.

Imaging accuracy – combinations of tests

Finally, the accuracy of using two non-invasive tests in combination was assessed in the 34 patients who had all four imaging tests. CTA disagreed with MRA in the categorisation of operable/inoperable in 13 patients (38%) and with Doppler in 14 patients (41%); MRA disagreed with Doppler in nine patients (26%). However, if any two of the three non-invasive procedures were performed on every patient, with a third non-invasive procedure used if the first two disagreed, then there would only be a disagreement with DSA in two patients (6%).

Patient preferences

Patients most preferred CTA, with DSA and MRA being equally less popular (table 6). Of the 44 patients who underwent more than one imaging procedure, 45 (76%) preferred CTA, 15 (22%) preferred DSA, and 10 (21%) preferred MRA. Opinion on DUS preference was not sought because of the time lapse from DUS examination to hospital admission for angiography and because it was an integral and unavoidable part of patient screening.

DISCUSSION

In this prospective study of a typical population of sequentially referred patients with symptomatic cerebrovascular disease and tight carotid stenosis, who were being considered for endarterectomy, DUS, CTA, and MRA all performed similarly in the measurement of carotid stenosis. CTA underestimated the stenosis, MRA overestimated it, and Doppler lay somewhere in between the two. None of the techniques was highly accurate, particularly around the 70–80% stenosis range where greatest discrimination is needed. We do not know whether the loss of accuracy with non-invasive tests resulting in a few inappropriate endarterectomy decisions may be offset by avoidance of the risk of angiography itself. However, based on the findings in this study, CTA and MRA are inappropriate replacements for DSA so far. Further technical improvements in CTA (for example, multislice CTA⁹) and MRA (contrast enhanced three dimensional MRA¹⁰) have been described, but these still need rigorous assessment against the reference standard, DSA.

The complications of DSA—particularly cardiovascular and peripheral vascular complications, some of which were serious—were in line with previous observations in similar populations.² This reflects the high prevalence of co-morbidity in a typical patient population encountered in clinical practice, and emphasises the importance of continuing to seek an alternative non-invasive diagnostic test.

The ECST tables for estimating risk and benefit of carotid endarterectomy are based on measurement of stenosis from DSA and were derived from patients who had survived DSA and were randomised into the trial—that is, the risk of DSA was not included in the risk to benefit equation for endarterectomy. The greater safety of non-invasive imaging will result in an overestimation of the change in life expectancy (outcome) when using the non-invasive test instead of DSA. However, this difference is likely to be small, and in any case in the present study the inaccuracy of the non-invasive tests would result in loss of life expectancy. Their greater safety might result in net gain in life expectancy, but this could only be determined in a large randomised trial.

The interobserver variability between paired observers for each imaging mode was similar for CTA, MRA, and DSA and for DUS in our previous study.⁸ This variability has significant implications with regard to the reliability of individual measurements of stenosis when using any mode, and puts into question the use of a single measurement of stenosis on DSA as the reference standard. Our results are similar to those obtained in two previous studies of the interobserver variability of measurement of stenosis from DSA, which found κ values indicating moderate to good agreement.^11,12 We did not measure intraobserver variability, for the reasons given in Results, but it is likely to be no worse than the interobserver variability.

Our analysis does not really compare like with like: CTA and MRA were both assessed by two observers and discrepancies discussed to produce a final agreed value, whereas DUS was performed once only with no second observer, as well as there being a time lapse to DSA during which the carotid disease could truly have changed. It was not practical to repeat the DUS owing to lack of time in a busy clinic. This might account for some of the apparent discrepancies in the degree of stenosis compared with DSA. Where there was a significant delay between DUS and DSA, there was potential for true stenosis progression or occlusion. However, despite the study design allowing several reasons for DUS to appear less accurate than CTA or MRA, it performed equally well.

A sizeable proportion (53%) of patients admitted to the study on an “intention to image” basis did not complete all imaging. However, this reflects the reality of investigating patients with multiorgan disease with complex neuroradiological techniques in clinical practice. In addition to their carotid disease, many of the patients had ischaemic heart disease, peripheral vascular disease, or chronic obstructive airways disease. Indeed six of the 73 patients admitted to hospital for the key test of DSA did not have it, and there was a significant number of complications following DSA which contributed to the number of patients who did not undergo CTA or MRA.

Numerous previous studies of MRA,¹⁴ and increasingly of CTA,¹⁵ report high sensitivities and specificities for assessing carotid stenosis. However, many studies had methodological problems that may have led to overestimates of accuracy.^3–5 We have documented all our cases, rather than excluding from mention those who did not complete all tests. We focused on patients with known symptomatic tight carotid stenosis, the population in whom a confirmatory non-invasive test needs to be most accurate. Many other larger studies have included substantial proportions of subjects with a lower incidence of disease,^3,5 which may have led to an overestimation of accuracy.

Conclusions

Our study shows that in a population of patients with a high prevalence of tight symptomatic carotid stenosis, the use of DUS, CTA, or MRA alone instead of DSA may result in clinically important differences in patient treatment. Further studies are required to assess whether a combination of tests, or the new technical advances in MRA and CTA, are sufficient to replace DSA.