Abstract
BACKGROUND AND PURPOSE: Recanalization is the important outcome measure for acute stroke therapy. Several methods of recanalization assessment are used in clinical practice, but few studies have addressed their reliability. We, therefore, sought to assess interobserver reliability of the diagnosis of intracranial artery recanalization following intervention by using TIMI criteria.
MATERIALS AND METHODS: The digital angiography scans of all patients with acute ischemic stroke during 2009 undergoing DSA and endovascular procedures at Ostrava University Hospital were assessed in the study. Images were retrospectively evaluated for intracranial artery recanalization on the TIMI scale by 2 experienced neuroradiologists who were blinded to clinical findings and to each other.
RESULTS: The angiography scans of 43 patients (16 females; age, 70.5 ± 14 years; median baseline NIHSS score, 15 [IQR, 11–18]) were retrospectively evaluated in our study. At 3 months, 27% of patients had mRS scores ≤ 2 and mortality was 18%. Two radiologists diagnosed TIMI grades as follows: TIMI 0, 16%, and 16%; TIMI 1, 21%, and 8%; TIMI 2a, 32% and 29%; TIMI 2b, 13% and 16%; TIMI 3, 18, and 31%. Interobserver agreement for recanalization was weighted κ = 0.4 (95% CI, 0.2–0.6).
CONCLUSIONS: The diagnosis of recanalization after interventional procedures was found to have poor interobserver agreement between 2 experienced neuroradiologists. TIMI criteria, therefore, do not permit reliable comparison of the efficacy of recanalization therapy among different studies.
ABBREVIATIONS:
- CI
- confidence interval
- IQR
- interquartile range
- IVT
- intravenous thrombolysis
- mRS
- modified Rankin Scale
- TICI
- thrombolysis in cerebral infarction
- TIMI
- thrombolysis in myocardial infarction
Occlusion of cervical or intracranial arteries is the most common cause of acute ischemic stroke.1 Recanalization improves outcome after stroke and is, therefore, considered an important surrogate outcome measure for the efficacy of therapeutic intervention in patients with acute ischemic stroke.1⇓⇓–4 However, there is, at present, no consensus on how intracranial artery recanalization should be evaluated. Various methods of assessment used in clinical practice include TICI and TIMI scales and their variants, but only limited data are available concerning their reliability.5⇓–7 The aim of the present study was to evaluate interobserver reliability of assessment by using TIMI criteria for intracranial artery recanalization following intervention.
Materials and Methods
All patients with acute ischemic stroke during 2009 undergoing DSA and endovascular procedures at Ostrava University Hospital were enrolled in the study. Eligibility criteria for the endovascular procedure were as follows: time of ≤8 hours from stroke onset, moderate or severe neurologic deficit (NIHSS score, >8), occlusion of a major intracranial artery on CT angiography, and failure of intravenous thrombolysis (nonrecanalization on transcranial Doppler or no clinical improvement after 30 minutes of intravenous thrombolysis infusion).
We recorded and evaluated demographic and clinical data: age, sex, time from symptom onset to the endovascular procedure, baseline NIHSS score, type of recanalization procedure, results of nonenhanced CT and CT angiography scans before the procedure and nonenhanced CT scans after the procedure, and mRS at 3 months after stroke.
DSA images were recorded according to a standard protocol. The baseline and final DSA images were retrospectively evaluated by 2 neuroradiologists for the site of intracranial artery occlusion and for the presence or absence of arterial recanalization. Both neuroradiologists were experienced in performing mechanical thrombectomy procedures, but they did not pass a certified training in TIMI (TICI) classification before the study.
The 2 neuroradiologists were blinded to the clinical findings and to each other. Recanalization of the intracranial artery was assessed by using modified TIMI classification3 (Table 1).
Modified TIMI classification3
Data Analyses and Statistical Analyses
Interobserver agreement was assessed by using linear weighted κ statistics. Linear weighting here denotes that the same weights were assigned to the difference between adjacent TIMI categories (eg, TIMI 0 versus 1 or TIMI 2 versus 3). Agreement was considered poor with κ <0.4, good with 0.41–0.75, and excellent with >0.75.
Results
A total of 43 patients were considered candidates for intervention. Of these, 1 patient was excluded because no recanalization therapy was performed after the initial angiography. A further 4 patients were excluded as a result of errors in transferring images to the radiologist located in another hospital via the PACS. The scans of the remaining 38 patients were analyzed. Detailed patient characteristics are shown in Table 2. Eleven (27%) patients had mRS ≤ 2 at 3 months.
Patient characteristics
Two radiologists diagnosed TIMI grades as follows: TIMI 0 16% and 16%; TIMI 1, 21% and 8%; TIMI 2a, 32% and 29%; TIMI 2b, 13% and 16%; and TIMI 3, 18% and 31%. Interobserver agreement for recanalization was weighted κ = 0.4 (95% CI, 0.2–0.6) (Table 3).
Interobserver results of recanalization on a modified TIMI scale: cross-tabulation
For patients with MCA versus non-MCA stroke, the weighted κ was 0.39 (95% CI, 0.2–0.6) and 0.45 (95% CI, 0.2–0.9), respectively.
Discussion
In this study, we report that there was poor agreement in the assessment of recanalization on the TIMI scale by 2 experienced radiologists. Our results, therefore, challenge the reproducibility of recanalization assessment after interventional procedures. As a consequence, it is likely to be difficult or even impossible to compare results based on TIMI scores between different clinical trials, recanalization devices, or case series. Furthermore, our results could explain, in part, why the association between recanalization and outcome differs substantially among different studies.2⇓⇓⇓–6
Few studies have addressed the reproducibility of recanalization assessment following intervention. The Interventional Management of Stroke II (IMS II) study3 reported that 41% of evaluations of recanalization by TIMI were discordant, mostly due to over-rating of the TIMI score by the clinical site and under-rating by the core laboratory. In the IMS I study, TIMI and arterial occlusive lesion scores had only modest agreement. It has become common practice to compare revascularization trials with historical controls. For these comparisons to be valid, we must use the same revascularization end points.8
Several factors could explain the low reproducibility of TIMI scores between different observers. First, the TIMI scale was initially developed for the assessment of reperfusion in patients with myocardial infarction; it is possible that the scale is not well-suited for the assessment of recanalization in stroke. Second, neuroradiologists are insufficiently trained in TIMI grading. Third, radiologists who do not themselves perform the procedure might be less confident in interpreting the images—this feature could explain the most discrepant readings (eg, TIMI 3 or TIMI 2a versus 0, which occurred in 2 cases). Finally, the utility of TIMI could differ between different intracranial territories, though our data suggest that the inconsistency of interobserver agreement did not differ for MCA and non-MCA territories.
A major limitation of this study is its retrospective design; potentially this could have affected the quality and standardization of the images. Our study was also performed at a single center with a limited number of patients.
Conclusions
Interobserver agreement in the evaluation of intracranial artery recanalization following endovascular treatment in our study was poor. A more reliable tool for the assessment of recanalization in stroke is needed. The process of evaluation of recanalization must be unified, and discrepancies in the recanalization rate should be decreased by assessment images in a core laboratory. Training programs for improving the evaluation of recanalization should be established.
Acknowledgments
We thank Dana Salounova for assistance with statistics.
Footnotes
-
Disclosures: Michal Bar—RELATED: Support for Travel to Meetings for the Study or Other Purposes: Pfizer, Comments: registration fee to European Stroke Conference in Hamburg, 2011. Antonin Krajina—UNRELATED: Grants/Grants Pending: Czech Healthcare Ministry Grant Agency.
-
Paper previously presented at: European Stroke Conference, May 24–27, 2011; Hamburg, Germany.
Indicates open access to non-subscribers at www.ajnr.org
References
- Received June 23, 2011.
- Accepted after revision August 19, 2011.
- © 2012 by American Journal of Neuroradiology
In this issue
Jump to section
Related Articles
Cited By...
- Reliability of the Modified TICI Score among Endovascular Neurosurgeons
- The Revascularization Scales Dilemma: Is It Right to Apply the Treatment in Cerebral Ischemia Scale in Posterior Circulation Stroke?
- Inter- and Intraobserver Agreement in Scoring Angiographic Results of Intra-Arterial Stroke Therapy
- Recommendations on Angiographic Revascularization Grading Standards for Acute Ischemic Stroke: A Consensus Statement
- Clarifying Differences Among Thrombolysis in Cerebral Infarction Scale Variants: Is the Artery Half Open or Half Closed?