Accuracy of Preliminary Interpretation of Neurologic CT Examinations by On-Call Radiology Residents and Assessment of Patient Outcomes at a Level I Trauma Center

doi:10.1016/j.jacr.2009.07.021

Journal of the American College of Radiology

Volume 6, Issue 12, December 2009, Pages 864-870

https://doi.org/10.1016/j.jacr.2009.07.021 Get rights and content

Purpose

The aims of this study were to evaluate the accuracy of preliminary interpretations of emergency neurologic CT scans after hours by on-call radiology residents and to assess the clinical impact of residents' errors at a level I trauma center.

Methods

A quality assurance database of neurologic CT examinations was reviewed to compare preliminary interpretations by on-call residents with final analyses by attending neuroradiologists during a 12-month period. All disagreements were reviewed for confirmation of the findings and categorized as significant or nonsignificant. Significant errors were further classified as acute intracranial, acute extracranial, and nonacute. Medical records for scans with significant errors were reviewed to evaluate any negative impact on the patient for each significant case. Residents' postgraduate years were also recorded.

Results

There were 252 cases (3.7%) with disagreements among 6,852 total cases. Of those, 226 (3.3%) were confirmed as resident errors, which included 171 (2.5%) that were significant. There were 73 (1.1%) acute intracranial, 77 (1.1%) acute extracranial, and 21 (0.3%) nonacute misinterpretations. Among the 171 significant cases, 105 (1.5%) had no changes in clinical management, and 55 (0.8%) required some changes.

Conclusion

The rate of significant errors by on-call radiology residents was low. These errors had a minimal impact on clinical outcomes. Continued monitoring of residents' performance is important to maintain or improve patient safety.

Section snippets

Study Design

We reviewed the QA data from April 10, 2006, to April 8, 2007. Preliminary interpretations of 6,859 neurologic CT examinations were provided by on-call radiology residents on-site in the ED at Harborview Medical Center. Our institution is the only level I trauma center for 5 northwestern states (Washington, Wyoming, Alaska, Montana, and Idaho), covering 25% of the US land mass. The neurologic CT examinations during off hours consisted of head CT (80%), maxillofacial CT (9%), head CT angiography

Results

Of the 6,859 cases imaged during the study period, 5 were excluded because of incomplete data, 1 because of a lack of clarification of the discrepant interpretation in the final report, and 1 because of the attending neuroradiologist's revision of his conclusion from “disagree” to “agree.”

Discussion

In our study, significant (class I) resident errors were found in 2.5% of all after-hours neurologic CT examinations during the 1-year study period. Patients' clinical management and courses were altered in about one-third of these class I errors (0.8% of all patients). We found no data in the literature to compare with this finding. Several studies have reported similar radiology resident error rates [1, 2, 3, 4, 5, 6, 7] (Table 2), but the data are not comparable between studies for several

Conclusion

The rate of resident misses and misinterpretations was 2.5% for significant misses at this high-volume level I trauma center with a high rate of CT examinations with positive results. The rate was comparable with the rate of disagreement by nighthawks using teleradiology for emergency head CT. Changes in patient management or outcomes were observed in <1% of cases. The most frequent misinterpretation was of subtle intracranial hemorrhage. The continual monitoring of resident error rates as a

Acknowledgment

We would like to acknowledge Martin Gunn, MD, and Marie Leonard for their contributions to the completion of this article and editorial assistance.

References (10)

B.F. Branstetter et al.
Preliminary reports in the emergency department: is a subspecialist radiologist more accurate than a radiology resident?
Acad Radiol
(2007)
N.R. Lal et al.
Clinical consequences of misinterpretations of neuroradiologic CT scans by on-call radiology residents
AJNR Am J Neuroradiol
(2000)
W.M. Strub et al.
Overnight preliminary head CT interpretations provided by residents: locations of misidentified intracranial hemorrhage
AJNR Am J Neuroradiol
(2007)
W.K. Erly et al.
Radiology resident evaluation of head CT scan orders in the emergency department
AJNR Am J Neuroradiol
(2002)
M.G. Wysoki et al.
Head trauma: CT scan interpretation by radiology residents versus staff radiologists
Radiology
(1998)

There are more references available in the full text version of this article.

Cited by (33)

Utility of Artificial Intelligence Tool as a Prospective Radiology Peer Reviewer — Detection of Unreported Intracranial Hemorrhage
2021, Academic Radiology
Misdiagnosis of intracranial hemorrhage (ICH) can adversely impact patient outcomes. The increasing workload on the radiologists may increase the chance of error and compromise the quality of care provided by the radiologists.
We used an FDA approved artificial intelligence (AI) solution based on a convolutional neural network to assess the prevalence of ICH in scans, which were reported as negative for ICH. We retrospectively applied the AI solution to all consecutive noncontrast computed tomography (CT) head scans performed at eight imaging sites affiliated to our institution.
In the 6565 noncontrast CT head scans, which met the inclusion criteria, 5585 scans were reported to have no ICH (“negative-by-report” cases). We applied AI solution to these “negative-by-report” cases. AI solution suggested there were ICH in 28 of these scans (“negative-by-report” and “positive-by-AI solution”). After consensus review by three neuroradiologists, 16 of these scans were found to have ICH, which was not reported (missed diagnosis by radiologists), with a false-negative rate of radiologists for ICH detection at 1.6%. Most commonly missed ICH was overlying the cerebral convexity and in the parafalcine regions.
Our study demonstrates that an AI solution can help radiologists to diagnose ICH and thus decrease the error rate. AI solution can serve as a prospective peer review tool for non-contrast head CT scans to identify ICH and thus minimize false negatives.
Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 1: Fixed Effects Identify Influences of Exam, Specialty, Fatigue, and Training on Resident Performance
2020, Academic Radiology
Citation Excerpt :
Neuroradiology cases were the most difficult to interpret correctly (p<0.0001) with effect size of 2.3 (95% CI 1.9–3.2) points lower than body and musculoskeletal oriented cases, which were not different from each other (p = 0.36). Multiple analyses by other investigators of resident-attending discrepancy reflect this trend with neuroradiology overnight readings having significantly higher rates compared to other sub-specialties (16–21). Case acuity had a significant effect (p<0.0001) on residence performance.
To describe our full-resolution simulation of critical care imaging coupled with posthoc grading of resident's interpretations and present results from the fixed effects terms in a comprehensive mixed regression model of the resulting scores.
The system delivered full resolution DICOM studies via clinical-grade viewing software integrated with a custom built web-based workflow and reporting system. The interpretations submitted by participating residents from 47 different programs were graded (scores of 0–10) on a case by case basis by a cadre of faculty members from our department. The data from 5 yearly (2014–2018) cycles consisting of 992 separate 65 case, 8 hour simulation sessions were collated from the transaction records. We used a mixed (hierarchical) statistical model with nine fixed and four random independent variables. In this paper, we present the results from the nine fixed effects.
There were 19,916/63,839 (27.0%, CI 26.7%–27.4%) scores in the 0–2 range (i.e., clinically significant miss). Neurological cases were more difficult with adjusted scores 2.3 (CI 1.9–3.2) lower than body/musculoskeletal cases. There was a small (0.3, CI 0.20–0.38 points) but highly significant (p<0.0001) decrease in score for the final 13/65 cases (fifth quintile) as evidence of fatigue during the last hour of an 8 hour shift. By comparing adjusted scores from mid-R1 (quarter 3) to late-R3 (quarter 12) we estimate the training effect as an increase of 2.2 (CI 1.90–2.50) points.
Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. Analysis of the resulting scores yields multiple insights into the interpretative process.
Interpretation of emergency CT scans in polytrauma: trauma surgeon vs radiologist
2020, African Journal of Emergency Medicine
Time is critical in the trauma setting. Emergency computed tomography (CT) scans are usually interpreted by the attending doctor and plans to manage the patient are implemented before the formal radiological report is available. This study aims to investigate the discrepancy in interpretation of emergency whole body CT scans in trauma patients by the trauma surgeon and radiologist and to determine if the difference in trauma surgeon and radiologist interpretation of emergency trauma CT scans has an impact on patient management.
This prospective observational comparative study was conducted over a 6 month period (01 April–30 September 2016) at the Inkosi Albert Luthuli Central Hospital which has a level 1 trauma department. The study population comprised 62 polytrauma patients who underwent a multiphase whole body CT scans as per the trauma imaging protocol. The trauma surgeons' initial interpretation of the CT scan and radiological report were compared. All CT scans reported by the radiology registrar were reviewed by a consultant radiologist. The time from completion of the CT scan and completion of the radiological report was analysed.
Since the trauma surgeon accompanied the patient to radiology and reviewed the images as soon as the scan was complete, the initial interpretation of the CT was performed within 15–30 min. The median time between the CT scan completion and reporting turnaround time was 75 (16–218) min. Critical findings were missed by the trauma surgeon in 4.8% of patients (bronchial transection, abdominal aortic intimal tear and cervical spine fracture) and non-critical/incidental findings in 41.94%. The trauma surgeon correctly detected and graded visceral injury in all cases.
There was no significant discrepancy in the critical findings on interpretation of whole body CT scans in polytrauma patients by the trauma surgeon and radiologist and therefore no negative impact on patient management from missed injury or misdiagnosis.
The turnaround time for the radiology report does not allow for timeous management of the trauma patient.
Reducing interruptions during duty radiology shifts, assessment of its benefits and review of factors affecting the radiology working environment
2018, Clinical Radiology
To investigate and reduce the number of inappropriate interruptions to the duty radiology registrar, as well as subjectively assess the effect of reducing interruptions and identify other methods of improving the duty working environment.
A pre- and post-intervention prospective quantitative study and post-intervention retrospective qualitative study of duty radiology registrars was performed at a tertiary referral centre. The first cycle of the quantitative study was performed prior to implementation of a telephone triage system. The second cycle and qualitative study were performed afterwards.
The average number of interruptions per day dropped by 43.7% after the intervention. Moreover, inappropriate interruptions dropped from one in three to one in five interruptions. Improvement was demonstrated following the intervention for perceived patient safety, workload, reporting efficiency, reporting accuracy, work satisfaction, and stress. The most common hindrance to a good working environment was interruptions (36%). The most common suggestion for improvement was improved comfort (33%).
The present study demonstrates that a telephone triage system can substantially reduce the number of interruptions to the duty radiologist. It also demonstrates that reducing interruptions in radiology has the potential to improve the working environment in many ways.
The accuracy of interpretation of emergency abdominal CT in adult patients who present with non-traumatic abdominal pain: results of a UK national audit
2017, Clinical Radiology
To evaluate major/minor discrepancy rates for provisional (initial) and addendum (supplementary senior review) emergency computed tomography (CT) reports in patients presenting with non-traumatic abdominal pain.
Ethical approval for this type of study is not required in the UK. All radiology departments with an approved lead for audit registered with the Royal College of Radiologists were invited to participate in this retrospective audit. The first 50 consecutive patients (25 surgical, 25 non-surgical) who underwent emergency abdominal CT for non-traumatic abdominal pain in 2013 were included. Statistical analyses were performed to identify organisational and report/patient-related variables that might be associated with major discrepancy.
One hundred and nine (58%) of 188 departments supplied data to the study with a total of 4,931 patients (2,568 surgical, 2,363 non-surgical). The audit standard for provisional report major discrepancy was achieved for registrars (target <10%, achieved 4.6%), for on-site consultants (target <5%, achieved 3.1%) and consultant addendum (target <5%, achieved 2.9%). Off-site reporters failed to meet the standard target (<5%, achieved 8.7% overall and 12.7% in surgical patients). The standard for patients coming to harm was not met in the surgical group (target <1%, achieved 1.5%) and was narrowly missed overall (target <1%, achieved 1%).
This study should be used to provide impetus to improve aspects of out-of-hours CT reporting. Clear benefits of CT interpretation/review by on-site and more senior (consultant) radiologists have been demonstrated.
Do telephone call interruptions havean impact on radiology resident diagnostic accuracy?
2014, Academic Radiology
Citation Excerpt :
Other minor discrepancies not felt to negatively impact patient management were not recorded. There have been several prior studies investigating potential factors associated with resident discrepancies, including imaging volume, imaging modalities, and on-call resident training level (21–28), but to our knowledge, there have been no studies directly measuring telephone calls as a root cause for diagnostic errors. Telephone call interruptions are one of the most frequent radiologist workflow interrupters (15), and thus, we focused on telephone calls as a potential root cause for radiology resident diagnostic errors.
The purpose of this study was to measure the effect of distractions, in the form of telephone call interruptions, on radiology resident diagnostic accuracy.
Radiology resident discrepancy reports and reading room telephone logs at an academic tertiary care pediatric hospital were collected over a 13-month period. Phone call times and durations were recorded. Major discrepancy shifts (defined as a call shift where at least one major discrepancy was discovered the following morning by the attending radiologist between the resident preliminary and attending final reports), and dictation time stamps for each discrepant preliminary dictation were also recorded. Telephone call volume and preliminary report time stamps were compared between “discrepancy shifts” and “no discrepancy shifts.”
Each call shift spanned 14 hours, during which one radiology resident was responsible for the generation of preliminary interpretations. Review of the discrepancy log data revealed 51 major discrepancies in 41 shifts, of which 39 discrepancies had documented error details and resident preliminary report time stamps. The average number of telephone calls for the “discrepancy shifts” was slightly greater than the “no discrepancy shifts” (48.59 vs. 44.02) but was not statistically significant (P = .0575). However, there was a statistically significant increase in the average number of phone calls in the 1 hour preceding the generation of a discrepant preliminary report versus the “no discrepancy shifts” (4.23 vs. 3.24 calls, P = .027). One additional phone call during the hour preceding the generation of a discrepant preliminary report resulted in a 12% increased likelihood of a resident error (P = .017).
Distractions in the form of telephone call interruptions may negatively impact on-call radiology resident diagnostic accuracy. Efforts should be made to limit distractions in the reading room.

View all citing articles on Scopus

View full text

Original articleAccuracy of Preliminary Interpretation of Neurologic CT Examinations by On-Call Radiology Residents and Assessment of Patient Outcomes at a Level I Trauma Center

Purpose

Methods

Results

Conclusion

Section snippets

Study Design

Results

Discussion

Conclusion

Acknowledgment

Acad Radiol

Clinical consequences of misinterpretations of neuroradiologic CT scans by on-call radiology residents

AJNR Am J Neuroradiol

Overnight preliminary head CT interpretations provided by residents: locations of misidentified intracranial hemorrhage

AJNR Am J Neuroradiol

Radiology resident evaluation of head CT scan orders in the emergency department

AJNR Am J Neuroradiol

Head trauma: CT scan interpretation by radiology residents versus staff radiologists

Radiology

Original article
Accuracy of Preliminary Interpretation of Neurologic CT Examinations by On-Call Radiology Residents and Assessment of Patient Outcomes at a Level I Trauma Center