Abstract
BACKGROUND AND PURPOSE: Diagnostic errors in radiology are classified as perception or interpretation errors. This study determined whether specific conditions differed when perception or interpretation errors occurred during neuroradiology image interpretation.
MATERIALS AND METHODS: In a sample of 254 clinical error cases in diagnostic neuroradiology, we classified errors as perception or interpretation errors, then characterized imaging technique, interpreting radiologist's experience, anatomic location of the abnormality, disease etiology, time of day, and day of the week. Interpretation and perception errors were compared with hours worked per shift, cases read per shift, average cases read per shift hour, and the order of case during the shift when the error occurred.
RESULTS: Perception and interpretation errors were 74.8% (n = 190) and 25.2% (n = 64) of errors, respectively. Logistic regression analyses showed that the odds of an interpretation error were 2 times greater (OR, 2.09; 95% CI, 1.05–4.15; P = .04) for neuroradiology attending physicians with ≤5 years of experience. Interpretation errors were more likely with MR imaging compared with CT (OR, 2.10; 95% CI, 1.09–4.01; P = .03). Infectious/inflammatory/autoimmune diseases were more frequently associated with interpretation errors (P = .04). Perception errors were associated with faster reading rates (6.01 versus 5.03 cases read per hour; P = .004) and occurred later during the shift (24th-versus-18th case; P = .04).
CONCLUSIONS: Among diagnostic neuroradiology error cases, interpretation-versus-perception errors are affected by the neuroradiologist's experience, technique, and the volume and rate of cases read. Recognition of these risk factors may help guide programs for error reduction in clinical neuroradiology services.
Medical errors are a substantial cause source of patient morbidity and mortality.1 Given the important role that diagnostic imaging plays in patient management, errors in diagnostic imaging interpretation likely contribute to this burden of medical error.2 To improve patient care, radiologists should analyze errors to identify conditions that may alter the risk of future errors. Such analysis may uncover contributory factors that can inform appropriate systematic corrective steps and improve the overall quality of patient care.3⇓–5
Errors in diagnostic radiology take 2 broad forms: errors of perception and errors of interpretation. A perception error occurs when a radiologist fails to identify an abnormality that is clearly present to most readers on a diagnostic image (false-negative errors). An interpretation error occurs when a radiologist identifies a finding on a diagnostic image but misinterprets its significance or meaning. Interpretation errors may take several forms such as attributing the wrong diagnosis to a finding, misjudging the severity of a finding, interpreting a normal finding as an abnormality, or interpreting an abnormal finding as a normal finding.
Prior studies of errors in radiology demonstrated that errors of perception compose most radiologic errors (60%–80%).6⇓–8 Little is known about potential risk factors that might predispose radiologists to perceptual-versus-interpretive errors. We hypothesized that the imaging technique, specific imaging finding, underlying diagnosis, radiologist's experience, and environmental conditions would influence the type of error made by neuroradiologists. Our study characterized whether case-specific metrics differed between perception and interpretation errors among a cohort of recognized neuroradiology error cases from a large academic medical center.
Materials and Methods
Case Selection and Categorization
This study was Health Insurance Portability and Accountability Act–compliant and was approved by the local institutional review board.
Cases of diagnostic errors were collected by staff and trainees in the neuroradiology division at a large tertiary care medical center as part of an ongoing practice-quality-improvement initiative. Each attending physician, fellow, and rotating resident in the neuroradiology division was instructed to place any error case they identified during clinical practice into a specified digital folder in our PACS. This requires a single-click drag-and-drop function facilitating addition of relevant cases to the cohort. A monthly divisional “M&M” conference was held where error cases were presented in a confidential manner, and the error was adjudicated by consensus among board-certified staff neuroradiologists.
Cases for this study accrued from July 2014 to January 2016 (19 months) during which time 75,804 CT and MR imaging neuroradiology examinations were interpreted at our institution. The cases and radiologist reports were retrospectively reviewed by a neuroradiology fellow and 2 board-certified attending neuroradiologists (with 5 and 7 years of experience, respectively) confirming that an error had occurred by consensus, with incorporation of any relevant clinical information available in the electronic medical record or PACS before and subsequent to the case in question. Errors were defined as findings or interpretations that both neuroradiologists thought should have been in the impression of the radiology report and that both neuroradiologists agreed would potentially change clinical management (eg, missed aneurysm, but not inflammatory maxillary sinus changes in an immunocompetent patient). Because not all cases have criterion standard final diagnoses (such as surgical pathology distinguishing meningioma from schwannoma), for interpretation errors, we emphasized missing differential diagnoses that both neuroradiologists thought should have been included in the radiology report. A total of 254 errors were discovered. Each error was categorized as either a perception error (the relevant finding was not identified) or an interpretation error (the relevant finding was identified, but not interpreted correctly) (Figs 1 and 2).
Perception error. A 55-year-old woman undergoing brain MR imaging for multiple sclerosis. A, Contrast-enhanced T1WI (using a 3D magnetization prepared rapid acquisition of gradient echo sequence) reveals an incidental aneurysm of the right internal carotid artery (arrow), which was not identified on this examination. It was detected on subsequent MR imaging. B, A cerebral angiogram confirms the aneurysm (arrow).
Interpretation error. A 17-year-old adolescent boy undergoing brain MR imaging for headache. A round mass (arrows) was identified in the right parietal lobe, demonstrating peripheral hypointense “blooming” on the T2*WI (A) and both central and peripheral contrast enhancement on the contrast-enhanced T1WI (B). The MR imaging finding was detected but incorrectly interpreted as representing a cavernous malformation. C, A subsequent angiogram revealed that this was, in fact, a partially thrombosed aneurysm arising from the right middle cerebral artery (arrow). (In A and B, there are artifacts from the patient's braces).
Each error case was categorized by the imaging technique (CT, MR imaging, radiograph), anatomic location of the relevant finding (cerebral gray matter, cerebral white matter, infratentorial brain, vasculature, meninges/ventricles, calvaria/scalp, face/skull base, neck or spine/paraspinal), and etiology (neoplastic, vascular/ischemic, congenital, trauma, degenerative, infection/inflammation/autoimmune, or other). The diagnoses were tabulated for each case. The experience level of the radiologist who made the error was noted (trainee; non-neuroradiology attending physician; attending neuroradiologist with ≤5 years, 6–10 years, 11–20 years, or >20 years of experience). There were 12 attending neuroradiologists who made at least 5 errors included in this study, and each attending neuroradiologist experience level comprised exactly 3 such attending neuroradiologists. The remaining attending neuroradiologists who made <5 errors included in our study were part-time or per diem neuroradiologists, neuroradiologists who previously worked at our institution but have since left, and community neuroradiologists affiliated with our institution. Trainee errors were only included if the trainee created and signed a “preliminary report” that was available in the electronic medical record for viewing by the referring clinicians. The cases were classified as those reported during regular weekday hours (Monday through Friday, 8 am to 5 pm) and those reported at all other times (ie, on-call situations). Characteristics of the specific radiologist's work shift when the error occurred were available in a subset (n = 217) of the cohort and included the number of cases read during the total shift, the shift length (in hours), and the average number of cases read per hour during that shift. Finally, the sequence in which the error case occurred during the shift was recorded (ie, was it the fifth case of the day or the 30th?).
Statistical Analysis
Perception errors and interpretation errors were summarized by frequencies and percentages. Logistic regression was used to examine whether interpretation-versus-perception errors were systematically related to the neuroradiologist's experience, imaging technique, anatomic location, disease etiology, day of the week, and time of day. The logistic regression outcome variables were binary variables that were assigned the value 1 if the diagnostic error was classified as an interpretation error and value 0 if the diagnostic error was classified as a perception error. We tested the null hypothesis that there was no systematic relationship between the categories of the predictor variable and the log-odds of cognitive error. The Wald statistic served as the pivotal quantity of the null hypothesis test, with P < .05 defined a priori as a statistically significant difference. The confidence interval was based on the Wald large-sample approximation method. Additionally, 2-tailed t tests at the P < .05 significance level were performed to compare the hours worked per shift, cases read per shift, cases read per shift hour, and the order of the error case during the shift, between perception and interpretation errors.
Results
A total of 254 recognized diagnostic neuroradiology errors were included. During the same time as error collection, approximately 75,000 diagnostic neuroradiology CT and MR imaging studies were performed at our institution. There were a total of 190 perception errors (74.8%) and 64 interpretation errors (25.2%). The On-line Table lists the errors by various categories. The most common diagnoses among the errors were the following: metastasis (n = 23), infarction (n = 20), vascular thrombus (n = 14), intracranial aneurysm (n = 14), and malformation of cortical development (n = 11).
There was no association among the 3 radiologist subgroups, trainees, non-neuroradiology attending physicians, and attending neuroradiologists for interpretation-versus-perception errors (P = .57). We note that during the shifts when errors occurred, the trainees (versus attending physicians) worked slightly longer hours per shift (9.0; 95% CI, 7.14–10.86 versus 7.3; 95% CI, 6.92–7.68; P = .04) and read fewer cases per shift hour (3.83; 95% CI, 2.91–4.74 versus 5.88; 95% CI, 5.53–6.24; P < .001). However, we found no significant difference in the average total cases read per shift (33.5 versus 41.3; P = .22) and sequential error occurrence during the shift (14th case versus 23rd case; P = .08) between trainees and attending physicians.
Examining only attending neuroradiologists did not reveal a statistical association between attending physician experience and interpretation-versus-perception error, when attending physician experience was categorized as 1–5 years, 6–10 years, 11–20 years, and >20 years of experience (P = .18). However, a post hoc comparison did reveal that the errors made by those with ≤5 years of experience were more likely to be interpretation errors (OR, 2.09; 95% CI, 1.05–4.15; P = .04) compared with the errors made by those with >5 years of experience.
There were 161 MR imaging cases (including 20 MRA/MRV), 89 CT cases (including 9 CTA/CTV), and 4 plain film cases. Errors associated with MRI examinations were more likely to be interpretation errors (OR, 2.10 [1.09, 4.01], P = 0.03) compared to errors associated with CT examinations. There was a significant association between interpretation-versus-perception error and disease etiology (P = .04), driven by the infectious/inflammatory/autoimmune category of which interpretation errors represented 52% of the errors (for all other disease categories, interpretation errors represented <30% of total errors).
There was no significant association between interpretation-versus-perception errors and the anatomic location of the relevant imaging findings (P = .48). There was no association between the type of error and the time of the week when the case was read (comparing routine weekday hours with all other times, P = .92).
Perception errors were associated with a 16% higher mean number of cases read per shift (42.5; 95% CI, 39.0–46.0 versus 36.7; 95% CI, 32.0–41.3; P = .05) and a 19% higher mean number of cases read per shift hour (6.01; 95% CI, 5.6–6.4 versus 5.03; 95% CI, 4.5–5.5; P = .004). There was no difference in the mean total hours per shift between the occurrence of perception errors or interpretation errors (7.3; 95% CI, 6.9–7.7 versus 7.6; 95% CI, 6.8–8.5; P = .45); however, perception errors did occur sequentially later during the shift (23.5; 95% CI, 20.8–26.2 versus 17.9; 95% CI, 13.7–22.0 previous cases read; P = .04).
Discussion
We characterized the relative proportion of perceptual and interpretive errors and the underlying conditions associated with these errors among a cohort of known, recognized error cases interpreted predominantly by academic subspecialty-trained neuroradiologists. To our knowledge, this is a novel approach to studying diagnostic error. Prior studies of diagnostic errors in neuroradiology have focused on determining the rate and significance of errors when interpreting emergency imaging studies9⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓⇓–21 or discrepancy rates between general practice radiologists and academic subspecialists.22⇓⇓–25 Discrepancy rates can be misleading because the incidence can be confounded by discovery or recall bias and were not the focus of this study.
Our results identified certain specific case, neuroradiologist, and reading situation features that were associated with interpretative or perception errors in neuroradiology practice. Less experienced neuroradiologists made a higher proportion of interpretation errors, while more experienced neuroradiologists made a higher proportion of perception errors. Errors were more likely to be perceptual with CT studies and interpretative with MR imaging studies. Errors were more likely to be interpretative during evaluation of patients with infectious, inflammatory, or autoimmune diseases. Perception errors were associated with a higher case volume, faster reading rates, and later parts of a work shift compared with interpretation errors.
Our results regarding imaging technique should be intuitive to most radiologists. Compared with CT, MR imaging is known to produce far greater sensitivity for many common pathologic processes seen on neuroimaging studies (eg, acute infarcts,26 cerebral neoplasms,27 demyelinating disease28), which should decrease the number of perception errors among MR imaging cases. Conversely, MR imaging has more artifacts, there are more MR imaging sequences per study to evaluate, and the greater complexity inherent to MR imaging findings relative to CT all might have contributed to the relatively higher proportion of interpretation errors among MR imaging-versus-CT cases.
Prior research in medical imaging perception might help to explain our results concerning the radiologist's experience level. Studies using eye-tracking technology indicate that more experienced radiologists have shorter image-viewing times, fewer visual fixations, fewer visual fixations on nondiagnostic areas of the image, and less dwell time per visual fixation when evaluating diagnostic imaging studies.29 Although such tendencies correlate with greater efficiency in image search, they might also explain the higher proportion of perceptual-type errors (ie, not seeing a pertinent imaging finding) among more experienced neuroradiologists. We stress, however, that while the proportion of errors among more experienced neuroradiologists compared with less experienced neuroradiologists was more heavily weighted toward perceptual errors, our results have not established that more experienced radiologists have a higher overall rate of perceptual errors.
Gaining an appreciation for predisposing conditions toward error type (interpretive-versus-perceptual) might aid future efforts towards error reduction in diagnostic neuroimaging. Perceptual errors (ie, nonidentification of a relevant imaging finding) might be reduced when radiologists use consistent search patterns or checklists during image evaluation,30 have improved accessibility to an accurate clinical indication,31 and read studies from state-of-the-art imaging systems.32 Clinical coverage schedules can be tailored to avoid overly long work shifts that may predispose to fatigue-related perceptual errors.33,34 Specific practice measures can also be implemented to reduce interpretation errors (ie, misinterpretation of a detected imaging finding), though such measures are less well-established in the literature. In theory, these practices may include multidisciplinary case discussion,35,36 selective second reading of studies associated with a higher risk for error,37,38 and application of a “slow-thinking” metacognition approach to avoid heuristic pitfalls when interpreting imaging abnormalities.39
This study has limitations. We collected 254 errors during a time when 75,000 CT and MR imaging studies were performed at our institution. This discovered error rate (<0.5%) is lower than previous clinically significant error rates reported from similar academic radiology practices.40 This finding likely reflects differences in the discovery mechanism and/or differences in what was considered a clinically significant error and does not indicate differences in interpretation accuracy between institutions. The subsequent study analysis implicitly assumes that error discovery was random and the subset used was a true representation of all errors that occurred. Error detection depended on a second radiologist's review of the images or a follow-up study; diagnostic errors that occur in outpatients without follow-up or image review during multidisciplinary conferences may be under-represented in our sample. Yet, our error-collection mechanism appeared to provide an externally valid representative cohort of cases; the percentage of perceptual errors and interpretive errors (74.8% and 25.2%, respectively) here is similar to that in previous reports.6⇓–8 Moreover, prior studies have shown that neoplastic and vascular etiologies are the most common neuroradiology error cases,25,40 concordant with our dataset (On-line Table). It would also be helpful in a future study to match error cases to similar cases that were interpreted correctly (ie, controls) to better resolve specific factors that contribute to errors. Such a measure may allow determination of differences in the error rate based on case-specific, work shift–specific, or radiologist-specific metrics and would be important for future study of this topic. The 2 neuroradiologists who adjudicated errors for this study would be considered less experienced in the above analytic framework, but these readers each had 5+ years reading only neuroradiology studies at an academic institution, adjudicated the studies by consensus (after prior group discussion), and had access to the full medical record (including subsequent clinical data).
Finally, we chose a simple, binary, and mutually exclusive scheme for classifying the error cases (ie, each case was either a perception or an interpretation error). Although this form of radiologic error categorization is commonly accepted in the literature,30 errors can take far more specific or overlapping forms than simply errors of perception or interpretation. For instance, research on medical imaging perception using eye-tracking technology indicates that perceptual errors broadly take 3 forms: 1) search error: the lesion is never fixated on; 2) recognition error: the lesion is fixated on, but below the threshold necessary for recognition; and 3) decision error: the lesion is fixated on for long periods, but its features are not fully appreciated and/or are dismissed.41 Interpretation errors also may vary, for instance, depending on whether a specific heuristic bias was involved in a given misinterpretation or a whether lack of sufficient knowledge contributed to the error.42,43 Understanding error trends at these higher levels of nuance is an important challenge for future research efforts.
Conclusions
Among diagnostic neuroradiology error cases, interpretation-versus-perception errors are associated with neuroradiology attending physician experience, imaging technique, disease etiology, case volume, reading rate, and case order. Our results highlight differences between error types that may inform future situation-specific effort toward error reduction in clinical neuroradiology services.
Footnotes
Disclosures: Sohil H. Patel—UNRELATED: Expert Testimony: medicolegal work*; Grants/Grants Pending: Radiological Society of North America Research Scholar Grant (RSCH1819)*. Jason N. Itri—UNRELATED: Expert Testimony: medicolegal work; Grants/Grants Pending: Coverys Community Healthcare Foundation, Comments: grant entitled, “Strategies to reduce diagnostic errors in imaging.”* Timothy M. Shepherd—UNRELATED: Expert Testimony: medicolegal work; Grants/Grants Pending: National Institutes of Health, National Institute on Aging K23-AG048622; Stock/Stock Options: MICroStructure Imaging (MICSI), Inc, Comments: start-up focused on advanced MRI processing and applications. *Money paid to the institution.
Paper previously presented, in part, at: Annual Meeting of the American Society of Neuroradiology and the Foundation of the ASNR Symposium, April 22–27, 2017; Long Beach, California.
REFERENCES
- Received March 21, 2019.
- Accepted after revision June 9, 2019.
- © 2019 by American Journal of Neuroradiology
