Article Text
Abstract
Doctors managing acute stroke are expected to recognise signs of early infarction on CT before choosing thrombolytic treatment, according to recent trials and guidelines. The ability of 13 physicians and two neuroradiologists to recognise early infarct signs and decide whether patients should be randomised in a hypothetical stroke treatment trial was tested. Only 65% of the CT scans from 14 stroke patients were correctly identified as normal or abnormal (95% CI 60–69%). Neither observer experience nor knowledge of symptoms significantly improved recognition of abnormality, although experience did significantly improve the observers’ ability to reproduce their results. Parenchymal hypodensity was the least well recognised sign. Only 45% (95% CI 40%–50%) of patients were identified correctly for the hypothetical acute stroke treatment trial. Early infarction on CT is not well recognised even by experienced doctors. Part of the problem may be in understanding the definitions of the extent of infarction. These difficulties should be considered in the design of acute stroke treatment trials and in the introduction of any new acute stroke treatments.
- computed tomography
- stroke
- cerebral infarction
- observer reliability
Statistics from Altmetric.com
Signs of early infarction on CT are subtle. Several recent trials and guidelines for the clinical use of thrombolysis in acute ischaemic stroke specify that patients with a visible infarct in more than one third of the middle cerebral artery (MCA) territory should not be given thrombolytic therapy.1 2 Thus patients may be denied entry to a trial (or treatment in ordinary clinical practice) if the attending physician thinks that there is a visible infarct, yet it is not clear that these signs can be detected reliably.3-6
Our aim was to determine how good doctors are at recognising signs of early infarction on CT, whether doctors could correctly decide which patients to include in a stroke treatment trial, and precisely where problems were occurring in the decision process.
Methods
Two consultant neuroradiologists, three consultant neurologists with an interest in stroke, one consultant stroke physician (constituting “experienced” observers), and five consultant neurologists, one general practitioner, and three trainee physicians (constituting “inexperienced” observers) took part in the study.
The CT scans were taken from the ECASS trial book of examples of patients scanned within 6 hours of stroke,7 which details symptoms, CT findings, and whether the patient had been randomised appropriately according to their CT appearance. Examples of 12 scans were made into slides without loss of image quality. Slides of identical quality and layout were also made of two normal CT brain scans from our neuroradiology department. Each slide showed four adjacent CT images, with no patient, scan, or hospital identifying data or date.
The slides were viewed three times in different, randomly determined orders (twice without and once with knowledge of symptoms), in a room with good projection facilities, allowing about 2 minutes for each slide. The observers were asked to record (a) whether the scans were normal or abnormal, and if abnormal, (b) the side of the brain involved, and (c) the features of early infarction which were visible (choices were: dense MCA,8 loss of basal ganglia outline, loss of insular ribbon,9 hypodensity,10 effacement of sulci or ventricles, and one invented sign, dense cortical sulci, the last to test knowledge of early infarct signs). Thus if the scan was correctly coded as abnormal, the observer could still incorrectly code the side of the brain or the abnormalities present. A fresh coding sheet was used for each viewing. The observers were not told that some scans were definitely normal. On the third viewing, with knowledge of symptoms, the two definitely normal CT scans were omitted. Finally, the observer had to decide whether the patient was suitable for an acute stroke treatment trial (eligible only if the scan was normal or showed infarction in less than one third of the MCA territory7). The data were entered into a database and analysed in SAS using logit modelling.
Results
ABNORMAL VERSUS NORMAL SCANS
Overall, 65% of the scans (read without knowledge of symptoms) were correctly coded as “normal” or “abnormal” (95% CI 60%–69%), but note that this does not take into consideration whether or not the side or nature of the abnormality was correctly identified. The “experienced” clinicians identified 68% of scans correctly, the “inexperienced”, 63% (p=0.3) indicating no benefit from experience in recognising normal versus abnormal CT scans of early infarction.
REPEATABILITY
Seventy nine per cent (95% CI 73%-84%) of scans were coded as “normal” or “abnormal” identically on the first and second readings. Both readings were blind to symptoms. Experienced observers reproduced their results significantly better than inexperienced observers (experienced observers coded 85% of scans identically on the two occasions, inexperienced observers 81%, p=0.049). The overall accuracy of the “normal"/"abnormal” coding did not differ significantly between readings 1 and 2 (69% of scans were correctly identified on reading 1 and 61% on reading 2, p=0.09).
EFFECT OF KNOWLEDGE OF SYMPTOMS
Overall, 69%, 60%, and 61% of the 12 abnormal scans were correctly identified as “abnormal” on readings 1, 2, and 3 respectively (figure). Sixty six per cent of “normal” scans were correctly identified as normal on reading 1 and 63% on reading 2 (the definitely normal CT scans were omitted from reading 3 as those subjects had no stroke symptoms).
RECOGNISING THE SIDE OF THE LESION
On average, the side of lesion (which could only be assessed for scans correctly coded as “abnormal”) was correctly identified in 78% of scans that were thought to be abnormal on reading 1, 86% on reading 2, and 90% on reading 3. Therefore the percentage of abnormal scans correctly identified as being abnormal and with the side of the abnormality also correctly identified was 54% on reading 1, 52% on reading 2, and 55% on reading 3.
KNOWLEDGE OF EARLY INFARCT SIGNS
The invented sign, dense cortical sulci, was said to be present by five doctors on at least one scan (four inexperienced and one experienced).
RECOGNITION OF EARLY INFARCT SIGNS
The data for the “inexperienced” doctors were so poor, with numerous missing responses, that only data for the “experienced” doctors were analysed for recognition of early infarct signs. The “experienced” observers missed many features that were present, but rarely noted a feature that was not present (hence the high specificities and low sensitivities: table). Dense MCA sign was seen most reliably. Hypodensity was missed in most cases.
RANDOMISATION INTO AN ACUTE STROKE TRIAL
On average, both experienced and inexperienced observers identified only 45% (95% CI 40%–50%) of all patients correctly (whether their scan had been coded normal or abnormal) for randomisation, almost significantly worse than guessing (p=0.07). Experienced observers did not perform significantly better than inexperienced observers in deciding whether to randomise (49%v 46% respectively, p=0.2). The three readings were not significantly different (49%, 45%, 41% p=0.4). However, doctors were consistent, coding reading 1 and 2 identically in 80% of scans (95% CI 72%–87%) and readings 2 and 3 in 71% of scans (95% CI 61%–79%), with inexperienced observers repeating their results as well as experienced observers (p>0.9).
Discussion
This study shows that a group of hospital doctors, many of whom were experienced in interpreting CT on stroke patients, are not very good at recognising cerebral infarction on CT within six hours of the stroke, and even worse at judging when an infarct occupies “a third or more of the MCA territory”.
The experience of the doctor and knowledge of the patient’s symptoms did not improve this significantly. However, the high repeatability of their results between readings (both for coding “normal”v “abnormal” and whether the patient should be randomised in a stroke treatment trial) suggests consistency of observer coding (despite repeatedly failing to recognise the true early infarct signs). Although the observers failed to recognise the early infarct signs that were present, they rarely said that a sign was present when it wasnot. Worryingly, hypodensity, the sign that was particularly used in trials and guidelines to identify the extent of the infarct (to determine whether one third of the MCA territory was affected), was the least well recognised sign.
Regarding randomisation to a hypothetical acute stroke treatment trial, the observers did no better than if they had just guessed. However, the participants repeated their findings between readings, indicating consistent coding, but not meeting the trial entry criteria. It is uncertain whether this was due predominantly to the poor recognition of “hypodensity” in a third of the MCA territory, or the participants being unable to decide what constituted “a third” of the MCA territory. However, we suspect that the problem is twofold. Firstly, doctors may fail to recognise “hypodensity” and secondly they may fail to understand what constitutes an infarct in a third of the MCA territory as the boundaries of the MCA territory vary from patient to patient and it is hard to see where an ill defined hypodensity, swelling, or loss of grey-white matter differentiation stops. However, the ECASS radiologists, against whose opinions our observers were tested, may have over-interpreted the early infarct signs on their CT, as they were not blind to the follow up scans.1 7
Problems in the present study include possible loss of image quality by projection of copied films (whether from book or original scan), viewing only four images per patient (although they were the most abnormal), the short time taken to examine the images (though more time was available if needed), and the possibility of fatigue (not supported by the improving recognition of the side of abnormality). Observers may do better with the entire image set, (although in emergencies, the images may be viewed only one at a time from a CT console), and if motivated by a real life clinical situation.
Previous studies of detection of early infarct signs on CT also found difficulties in recognising early infarction with or without training, but did not try to pinpoint the main cause or how the problem might be remedied. For example, physicians trained on CT had difficulty recognising any early infarct without and with symptoms (κ=0.22 and 0.27 respectively), and the extent of the MCA territory affected (κ=0.37).5 In another study, stroke physicians correctly recognised only 44% of “difficult” early infarcts.6In a study by von Kummer et al,3 six neuroradiologists recognised focal brain swelling, hypodensity, and the extent of the infarct (κ=0.56, 0.55, and 0.57 respectively), but this did not improve with knowledge of symptoms. Although there is some evidence of improvement with specific training,8 the figures among experienced observers only represent at best moderate observer agreement, and are not sufficiently reliable for making decisions about thrombolytic treatment on the CT appearance.
Our study suggests that increased training—that is, experience—may not be the whole answer, but that two factors might improve recognition of early infarction. Firstly, simplification of which signs are most important to recognise, clarification of their definition and focusing training of doctors on recognition of those signs, and secondly a clearer description of the infarct extent. It is uncertain whether early infarct visibility is truly a contraindication to thrombolytic (or other acute) stroke treatment, but determination of this will be very difficult unless a more reliable criterion is found.
Acknowledgments
The Medical Research Council funded PJD, JMW, SCL, and PAGS. We thank Dr R Gibson, Dr D Collie, Dr M Dennis, Professor C Warlow, Dr C Mumford, Dr R Grant, Dr A Zeman, Dr R Knight, Dr D Schultz, Dr S Johnson, Dr U Nath, Dr F Skinner, Dr M Liu, and Dr M McLeod for participating in the study.