Reliability of the Modified TICI Score among Endovascular Neurosurgeons

DISCUSSION

In the era of endovascular mechanical thrombectomy for acute ischemic stroke, a number of revascularization scores have been developed and some further modified. These scores aim to quantify the success of revascularization or reperfusion and have been important reporting and prognostic metrics.

In 1992, Mori et al¹³ were the first to re-purpose the Thrombolysis in Myocardial Infarction (TIMI) scale from the cardiac literature for cerebral revascularization. These investigators broke down the partial filling grade 2 of TIMI into grades 2 ( < 50% filling) and 3 ( > 50% filling).¹³ Subsequently, the TICI scale was proposed by Higashida et al,¹⁴ in 2003, which focused on the revascularization assessment of territory reperfusion compared with arterial recanalization and changed the partial reperfusion grades 2 and 3 into grades 2a (less than two-thirds territory filling) and 2b (slowed-but-complete territory filling), respectively. Higashida et al reserved grade 3 for complete reperfusion.¹⁵ The Interventional Management of Stroke (IMS) II investigators established the mTICI scale, which continued to focus on reperfusion, but simplified the scale to define 2a as < 50% territory filling and 2b as > 50% filling.¹⁰ The mTICI scale has been widely used in the most recent major mechanical thrombectomy trials.^{1⇓⇓⇓⇓⇓⇓⇓-9}

A further gradation of the partial perfusion grade was suggested by Noser et al,¹⁶ with 2c representing near-complete territory filling with delayed contrast runoff, which is used at many centers but has not been universally adopted. Other revascularization scales have been proposed but have not found widespread use, including the Thrombolysis in Brain Ischemia (TIBI) scale that is based on transcranial duplex measurements, the Arterial Occlusion Lesion (AOL) scale that assesses recanalization, and the Qureshi scale that factors in the site of occlusion.^14,15,17,18

Our study found that raters had fair overall interrater agreement when analyzing the entire mTICI scale, which improved to moderate agreement when the responses were dichotomized to either ≤2a or ≥2b. Five previous studies have researched interrater reliability of the cerebral conventional angiography revascularization score. Bar et al,¹⁹ in 2012, published the reliability of the TIMI scale applied to cerebral revascularization across 2 reviewers assessing 43 cases. They found a weighted κ = 0.4, which is nearly equivalent to our findings (κ = 0.374). Gaha et al,²⁰ in 2014, published their reliability assessment of the original TICI scale, finding an overall κ = 0.45 across 9 observers, and when dichotomized, agreement increased, with κ = 0.62.

In 2013, Suh et al²¹ looked specifically at the effect of changing from a two-thirds territory to one-half territory threshold between 2a and 2b grades of the TICI and mTICI scales, respectively. Interobserver variability was assessed as good for the TICI and mTICI scales (intraclass correlation coefficient = 0.73 and 0.67, respectively). The TICI 2a–2b threshold variability led to different grading in ≈20% of cases, and the mTICI score (using the one-half territory threshold) was better at predicting clinical outcome. Volny et al,²² in 2017, assessed reliability using the addition of the 2c “near-complete reperfusion” to the mTICI scale. Sixty-one patients were assessed by 3 reviewers of different specialties and levels of training, who also compared different combinations of consensus grading with those of different specialties and levels of training. When compared against a criterion standard of a consensus grading between a neurointerventional fellow and attending, they found fair reliability for a stroke physician (κ = 0.36), moderate reliability for a neuroradiologist (κ = 0.48), and moderate reliability for a neurointerventional fellow (κ = 0.56). They also found that different combinations of reviewer consensus grading increased to almost perfect agreement and concluded that mTICI 2c is a feasible adjunct.²²

The most recent mTICI reliability study by Fahed et al,²³ in 2018, assessed 305 patients in the Contact Aspiration vs Stent Retriever for Successful Revascularization (ASTER) trial by 2 blinded neurointerventional radiologists, and these scores were also compared with those of the proceduralist who performed the procedure. Scores were analyzed both on an ordinal scale as well as dichotomized to ≤2a or ≥2b. They found moderate agreement for the nondichotomized mTICI score and substantial agreement with the dichotomized mTICI score.²³ These findings largely mirror our findings.

We also found a trend of proceduralists’ scores being higher than independent raters’ scores, which was also found by multiple prior studies.^23⇓⇓-26 For cases not in agreement, the proceduralists’ score was higher than the raters’ scores in 77% of cases. This speaks to a consistent bias that interventionalists must be aware of and attempt to overcome by rigorous objective assessment or internal blinded review.

Finally, we found that cases of M1 occlusions had the highest agreement, followed by ICA occlusions, while M2 and other vessel occlusions had poor agreement. These findings may be explained by an effort to score only the final cerebral reperfusion result, disregarding the initial perfusion findings. In fact, a posterior cerebral artery occlusion was the 1 case to be scored differently by all 4 raters. Although the scales explicitly use the initial area of the brain not receiving antegrade perfusion as the denominator to calculate the reperfusion result, it is a simpler process to always use an M1 occlusion area as the denominator by removing a step of interpretation (the determination of the initial nonperfusing brain area). Because M1 occlusions are the most common occlusion location requiring mechanical thrombectomy, many interventionalists may gravitate toward this standard consciously or subconsciously. Another explanation is that many descriptions of the mTICI scale use an M1 occlusion as an example, describing a 2b reperfusion result “eg, when greater than 50% of the MCA territory is filling.” Also, the best predictor of good outcome is a lower final infarct volume (regardless of how much brain was potentially at risk at the beginning of the case). Other explanations of this finding include branching variability of the MCA for M2 occlusion assessment, general variability of the posterior circulation, and difficulty in incorporating the anterior cerebral arteries into the scoring for ICA occlusion assessment (ie, an ICA terminus occlusion may still perfuse the ipsilateral anterior cerebral artery through the anterior communicating artery). Any of these may affect both intra- and interrater agreement. Nonocclusive thrombus may also be a challenge and account for some disagreement. Additionally, the evolution of grading scales and nomenclature may have had an effect on reliability because raters may have developed inherent biases based on timing and the institution of training. We chose to study the standard mTICI, given its widespread use throughout the large mechanical thrombectomy trials.

Our study is the first in the literature, to our knowledge, to assess intrarater reliability of the mTICI score. When raters were compared against themselves across the 2 time points, they had substantial agreement, higher than when raters were compared against each other, demonstrating a personal consistency in assessment. When answers were dichotomized to either ≤2a or ≥2b, intrarater reliability rose to substantial-to-almost perfect agreement. The interrater reliability also improved on the second survey for both the ordinal and dichotomized analyses. This outcome may be due to a learning curve for the survey platform or may be evidence of a further familiarity with the mTICI system. This possibility could suggest that standardized training for mTICI, similar to what is done for clinical stroke assessment, will improve reporting consistency. However, the importance of angiogram interpretation standardization is less than that of clinical stroke assessment because an angiogram is a static record that can always be adjudicated at a later date and a patient’s clinical state is fluid with the only record being that of the clinician’s assessment on the day of examination (assuming these are not video-recorded).

Limitations from our study include heterogenous occlusion locations with unequal representation of certain occlusion locations. Although the locations are heterogeneous and unequal, this situation occurs in practice. More education for raters by way of case examples, explanation of rating scales from the literature and trials, and tutorials on how to deal with nuances before their formal scoring on the Web-based platform would have set the stage for more standard results. However, a simple 1-page definition of scores does allow a better real-world assessment of interventionalists’ ratings, whereas heavy training would bias results away from the current state of practice and would be more of an assessment of how our training package standardizes results. We did not want to influence the scoring with our hypothesized biases.

Of the 67 individuals reviewed, reperfusion in 70.15%–83% of them was rated as 2b or 3, depending on the rater. The lack of heterogeneity in categories produces a high estimate of chance agreement, which produces a lower κ score. This is concordant with the HERMES collaboration reporting of the 5 thrombectomy trials published in 2015, which reported mTICI scores of 2b or 3 in 71% of cases. κ statistics could be artificially low, given that there was not enough representation of 0, 1, or 2a mTICI cases. Last, we compared the proceduralists’ mTICI assessment with that of our raters, but we have no knowledge of what the original proceduralists’ cutoffs for TICI grading were due to the retrospective nature of our study.