Radiology Resident Preliminary Reporting in an Independent Call Environment: Multiyear Assessment of Volume, Timeliness, and Accuracy

doi:10.1016/j.jacr.2014.08.005

Journal of the American College of Radiology

Volume 12, Issue 1, January 2015, Pages 95-100

https://doi.org/10.1016/j.jacr.2014.08.005 Get rights and content

Abstract

Purpose

The objective of this paper is to assess the volume, accuracy, and timeliness of radiology resident preliminary reports as part of an independent call system. This study seeks to understand the relationship between resident year in training, study modality, and discrepancy rate.

Methods

Resident preliminary interpretations on radiographs, ultrasound, CT, and MRI from October 2009 through December 2013 were prospectively scored by faculty on a modified RADPEER scoring system. Discrepancy rates were evaluated based on postgraduate year of the resident and the study modality. Turnaround times for reports were also reviewed. Differences between groups were compared with a chi-square test with a significance level of 0.05. Institutional review board approval was waived as only deidentified data were used in the study.

Results

A total of 416,413 studies were reported by 93 residents, yielding 135,902 resident scores. The rate of major resident–faculty assessment discrepancies was 1.7%. Discrepancy rates improved with increasing experience, both overall (PGY-3: 1.8%, PGY-4: 1.7%, PGY-5: 1.5%) and for each individual modality. Discrepancy rates were highest for MR (3.7%), followed by CT (2.4%), radiographs (1.4%), and ultrasound (0.6%). Emergency department report turnaround time averaged 31.7 min. The average graduating resident has been scored on 2,746 ± 267 reports during residency.

Conclusions

Resident preliminary reports have a low rate of major discrepancies, which improves over 3 years of call-taking experience. Although more complex cross-sectional studies have slightly higher discrepancy rates, discrepancies were still within the range of faculty report variation.

Introduction

A great deal of attention has been paid to the rate of errors in the medical system. Training programs have undergone considerable change over the past decade, including stringent work-hour regulations, increased supervision, and decreased trainee independence. Radiology residencies have not been exempt from this culture shift, with the experience of independently taking call being delayed into the 3rd postgraduate year (PGY-3) and increased pressure for faculty interpretation of studies 24 hours a day [1].

Large studies of resident preliminary reports have been in general agreement that resident discrepancy rates (i.e., the frequency of discrepancies between a resident’s versus a faculty member’s assessment of a patient) are low, with major discrepancy rates ranging from 0.1% to 2.6% 2, 3, 4, 5, 6, 7, 8. These values are comparable to published results of major discrepancy rates from internal peer review, which range from 0.8% to 2.1% 8, 9, and rates of 0.7%-3.6% when assessments by nonsubspecialists are reviewed by subspecialists 2, 10, 11, 12.

However, several papers examining smaller subsets of resident preliminary reports, such as abdominal or trauma CT, have shown higher discrepancy rates, of 5%-10%, and have been used to advocate for 24-hour faculty coverage 13, 14, 15. One report published a very low major discrepancy rate (0.6%) but still concluded that 24-hour faculty coverage is required to combat even the perception of a higher discrepancy rate [16]. Further complicating this issue, most of the studies were done prior to the ACGME requirement that residents complete 12 months of radiology training prior to taking call independently, and they did not address the increasing volume of MR performed in many emergency departments.

The objective of this paper is to assess the accuracy and timeliness of resident preliminary reports as part of an independent call system with remote faculty supervision. To achieve this goal, all resident quality data obtained prospectively over a 4-year period at a large tertiary hospital were evaluated. Additionally, the relationships of report discrepancy rates to the PGY of the resident and the study modality were investigated.

Section snippets

Resident Call Structure

Resident preliminary reporting was evaluated at a county hospital (approximately 800 beds) with a large emergency department, from October 2009 to December 2013. The facility offers 24-hour technologist coverage for radiography, ultrasound, CT, nuclear medicine, and MRI. Subspecialty faculty interpret exams during regular business hours (weekdays from 7:30 am to 5:00 pm), and after hours, on-call residents provide reports with the assistance of subspecialty faculty providing indirect

Study Modality and Volume

Studies were collected over a period of 4 years and 3 months, from October 2009 to December 2013. Over this period, a total of 416,413 studies were dictated by residents on call, 390,759 from the emergency department, and 25,654 stat-priority inpatients (Table 1). During this time period, 93 residents rendered preliminary reports, with a total of 135,902 review scores assigned by 95 faculty members (distribution shown in Table 2). The majority of these scores were collected on studies performed

Discussion

Although radiology resident on-call performance has been previously evaluated, this study provides several advantages. First, it uses electronically entered data collected prospectively as part of the resident education process over a multiyear period, without the errors that can be introduced by manual tabulation, or bias due to retrospective evaluation. Second, the study is a comprehensive representation of resident call duties, rather than an evaluation focused on one area, such as abdomen

Take-Home Points

•
Resident preliminary reports have a low rate of major discrepancies compared with the final subspecialist interpretation while still meeting hospital timeliness standards.
•
Radiology residents taking independent call steadily improve over their 3 years of call-taking experience.
•
Resident discrepancies are highest on MR, followed by CT, radiographs, and then ultrasound.
•
Residents interpreting MR on call benefit from structured education in MR prior to beginning their call experience and a higher

References (26)

A. Bansal
Twenty-four-hour attending physician coverage and its impact on resident training
J Am Coll Radiol
(2005)
B.F. Branstetter et al.
Preliminary reports in the emergency department: is a subspecialist radiologist more accurate than a radiology resident?
Acad Radiol
(2007)
V.F. Cooper et al.
Radiology resident interpretations of on-call imaging studies: the incidence of major discrepancies
Acad Radiol
(2008)
M.S. Davenport et al.
Effect of work hours, caseload, shift type, and experience on resident call performance
Acad Radiol
(2010)
J. Ruma et al.
Cross-sectional examination interpretation discrepancies between on-call diagnostic radiology residents and subspecialty faculty radiologists: analysis by imaging modality and subspecialty
J Am Coll Radiol
(2011)
A.T. Ruutiainen et al.
Identifying benchmarks for discrepancy rates in preliminary interpretations provided by radiology trainees at an academic institution
J Am Coll Radiology
(2011)
J.P. Borgstede et al.
RADPEER quality assurance program: a multifacility study of interpretive disagreement rates
J Am Coll Radiol
(2004)
J.C. Hillier et al.
Trainee reporting of computed tomography examinations: do they make mistakes and does it matter?
Clin Radiol
(2004)
N. Tieng et al.
Discrepancies in interpretation of ED body computed tomographic scans by radiology residents
Am J Emerg Med
(2007)
C.E. Blane et al.
Academic radiology and the emergency department: does it need changing?
Acad Radiol
(2007)

V.P. Jackson et al.

RADPEER scoring white paper

J Am Coll Radiol

(2009)

R.H. Briggs et al.

Provisional reporting of polytrauma CT by on-call radiology registrars. Is it safe?

Clin Radiol

(2010)

N.A. Shah et al.

Correlation among on-call resident study volume, discrepancy rate, and turnaround time

Acad Radiol

(2010)

Cited by (44)

Discrepancy rate and clinical impact of preliminary reports from radiology residents
2023, Heliyon
Residents usually cover night and weekend shifts issuing the preliminary reading of radiological studies in university hospitals. This is essential to strengthening decision-making skills when facing complex cases independently. However, there should be a balance between patient safety and academic experience since some concern has been expressed about the accuracy of the interpretations generated by trainees. This work aims to evaluate and characterize the discrepancies in preliminary reports issued by radiology residents.
Radiologists filled out a questionnaire to evaluate preliminary reports of trainees considering diagnosis, findings description, clinical approach changes, and critical findings. Analysis was performed considering modality, imaging type, body part, and resident academic year. A Chi-square test with a significance level α of 0.05 was used to make group comparisons.
A total of 9072 studies were reviewed. Major and minor overall discrepancy rates were 1.7% and 8.3%, respectively. Minor discrepancy rate, findings description, and critical findings identification improved with increasing academic year, both overall and by modality. Discrepancy rates were lower for CT than MR and neuroimaging than for body-imaging studies. The highest major and minor discrepancy rates as abdomen/pelvis CT and lumbar-spine MR, respectively. Two percent of reports presented discrepancies that could generate a medical approach change.
Discrepancy rates are low and comparable with those reported in the literature. These rates tend to improve as the resident's academic year increases. Our results suggest that radiology residents' coverage of night shifts and weekends is a practice that benefits the educational process without negatively impacting patient safety.
Effect of Independent Resident Night Call Versus 24-7 Attending Radiologist Coverage on Subsequent Practice Performance
2021, Journal of the American College of Radiology
Citation Excerpt :
Radiology residents receive delayed supervision by attending radiologists, most commonly the following morning, at which time any errors uncovered are remediated [1,2]. This standard practice has been frequently evaluated and is generally accepted as being extremely safe for patients, with a very low rate of significant resident errors and creating no appreciable increased risk of patient harm resulting from the residents’ role [3-6]. In recent years, nearly half of radiology residency programs have moved to providing continuous attending radiologist supervision to residents after hours, a training model that we will refer to as the 24-7 attending radiologist model.
Assessing the Training Costs and Work of Diagnostic Radiology Residents Using Key Performance Indicators – An Observational Study
2020, Academic Radiology
To quantify the costs and work of diagnostic radiology (DR) residents using the radiology key performance indicator turn-around time (TAT) as the outcome measure.
In an Institutional Review Board-approved study, the annual cost of a DR resident was determined using salary, benefits, and a cost allocation of faculty effort. The volume of cases reported in the 2015–16 academic year and median and interquartile range (IQR) TAT for a trainee preliminary (Complete to Prelim, C-P) or an attending final (Complete to Final, C-F) radiology report were measured and stratified by time of day and patient location. Wilcoxon rank-sum tests were used (significance, p values < 0.05).
The annual cost of a DR resident was $99,109, 34% greater than direct salary/benefits and 27% of the direct salary/benefits cost of an attending. The total per minute cost of rendering care was $4.36 with both trainee ($0.70/minute) and faculty ($3.66/minute). Residents participated in 139,084/235,417 (59%) imaging studies. The C-P TAT was 74 (IQR, 27–180) minutes compared to 51 (IQR, 18–129) minutes C-F TAT of faculty working alone and C-F TAT of 213 (IQR, 71–469) minutes with a resident (p < 0.001). The C-P TAT vs C-F TAT between 4 pm–9 am and weekends with residents is 44 (IQR, 18–119) minutes vs 60 (IQR, 18–179) minutes without.
The cost of training DR residents exceeds the salary and benefits allocated to their training. Residents increase the absolute professional labor cost of caring for a patient. Overall TAT is slower with residents but the care delivered by residents after-hours is faster.
The Resident Preliminary Report
2019, Journal of the American College of Radiology
Resident Case Volume Correlates with Clinical Performance: Finding the Sweet Spot
2019, Academic Radiology
Citation Excerpt :
Several authors have noted that clinical performance improves with training. As expected, more senior radiology residents have lower discrepancy rates (8,19). It is logical that even within a given class year, residents who have interpreted more examinations over the preceding years will, in general perform better than residents who have interpreted fewer examinations.
To determine whether the total number of studies interpreted during radiology residency correlates with clinical performance as measured by objective criteria.
We performed a retrospective cohort study of three graduating classes of radiology residents from a single residency program between the years 2015–2017. The total number of studies interpreted by each resident during residency was tracked. Clinical performance was determined by tracking an individual resident's major discordance rate. A major discordance was recorded when there was a difference between the preliminary resident interpretation and final attending interpretation that could immediately impact patient care. Accreditation council for graduate medical education milestones at the completion of residency, Diagnostic radiology in-training scores in the third year, and score from the American board of radiology core exam were also tabulated. Pearson correlation coefficients and polynomial regression analysis were used to identify correlations between the total number of interpreted films and clinical, test, and milestone performance.
Thirty-seven residents interpreted a mean of 12,709 studies (range 8898–19,818; standard deviation [SD] 2351.9) in residency with a mean major discordance rate of 1.1% (range 0.34%–2.54%; stand dev 0.49%). There was a nonlinear correlation between total number of interpreted films and performance. As the number of interpreted films increased to approximately 16,000, clinical performance (p = 0.004) and test performance (p = 0.01) improved, but volumes over 16,000 correlated with worse performance.
The total number of studies interpreted during radiology training correlates with performance. Residencies should endeavor to find the “sweet spot”: the amount of work that maximizes clinical exposure and knowledge without overburdening trainees.
Common Resident Errors When Interpreting Computed Tomography of the Abdomen and Pelvis: A Review of Types, Pitfalls, and Strategies for Improvement
2019, Current Problems in Diagnostic Radiology
The purpose of this study was to identify common errors that radiology residents make when interpreting abdominopelvic (AP) computed tomography (CT) while on call, to review the typical imaging findings of these cases, and to discuss strategies for improvement.
AP (or chest, abdomen, pelvis) CTs from 518 weekend senior call shifts (R3 or R4) were retrospectively reviewed. Discrepancies between preliminary and final reports were identified and then rated by whether the miss could impact short-term management. The imaging findings from the cases were reviewed.
A total of 4695 CTs were reviewed, revealing a total of 145 discrepancies that could affect short-term clinical management (miss rate 3.1%). The most common misses were related to blood clots (13.8%), colitis (8.3%), misplaced lines or tubes (6.9%), or pyelonephritis (5.5%). Common pitfalls and strategies from improved detection are discussed using image examples.
Through increased attention to the vasculature, colon, devices, and kidneys, trainees may improve their discrepancy rates and improve on-call reporting.

View all citing articles on Scopus

: JC is a member of the Clinical Advisory Board for PeerVue, San Francisco CA.

: TB is a Clinical Advisor for McKesson Corporation, San Francisco CA.

View full text

Original articleRadiology Resident Preliminary Reporting in an Independent Call Environment: Multiyear Assessment of Volume, Timeliness, and Accuracy

Abstract

Purpose

Methods

Results

Conclusions

Introduction

Section snippets

Resident Call Structure

Study Modality and Volume

Discussion

Take-Home Points

J Am Coll Radiol

Acad Radiol

Acad Radiol

Acad Radiol

J Am Coll Radiol

J Am Coll Radiology

J Am Coll Radiol

Clin Radiol

Am J Emerg Med

Acad Radiol

J Am Coll Radiol

Clin Radiol

Acad Radiol

Original article
Radiology Resident Preliminary Reporting in an Independent Call Environment: Multiyear Assessment of Volume, Timeliness, and Accuracy