Quality Measurements in Radiology: A Systematic Review of the Literature and Survey of Radiology Benefit Management Groups

doi:10.1016/j.jacr.2015.06.038

Journal of the American College of Radiology

Volume 12, Issue 11, November 2015, Pages 1173-1181.e23

https://doi.org/10.1016/j.jacr.2015.06.038 Get rights and content

Abstract

Purpose

As the US health care system transitions toward value-based reimbursement, there is an increasing need for metrics to quantify health care quality. Within radiology, many quality metrics are in use, and still more have been proposed, but there have been limited attempts to systematically inventory these measures and classify them using a standard framework. The purpose of this study was to develop an exhaustive inventory of public and private sector imaging quality metrics classified according to the classic Donabedian framework (structure, process, and outcome).

Methods

A systematic review was performed in which eligibility criteria included published articles (from 2000 onward) from multiple databases. Studies were double-read, with discrepancies resolved by consensus. For the radiology benefit management group (RBM) survey, the six known companies nationally were surveyed. Outcome measures were organized on the basis of standard categories (structure, process, and outcome) and reported using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

Results

The search strategy yielded 1,816 citations; review yielded 110 reports (29 included for final analysis). Three of six RBMs (50%) responded to the survey; the websites of the other RBMs were searched for additional metrics. Seventy-five unique metrics were reported: 35 structure (46%), 20 outcome (27%), and 20 process (27%) metrics. For RBMs, 35 metrics were reported: 27 structure (77%), 4 process (11%), and 4 outcome (11%) metrics. The most commonly cited structure, process, and outcome metrics included ACR accreditation (37%), ACR Appropriateness Criteria (85%), and peer review (95%), respectively.

Conclusions

Imaging quality metrics are more likely to be structural (46%) than process (27%) or outcome (27%) based (P < .05). As national value-based reimbursement programs increasingly emphasize outcome-based metrics, radiologists must keep pace by developing the data infrastructure required to collect outcome-based quality metrics.

Introduction

As health care costs continue to escalate, policymakers have pursued ways to transition away from a purely volume-oriented, fee-for-service system toward one in which providers are also rewarded for providing higher quality care. To determine the meaning of quality in health care settings, numerous public, private, and nonprofit entities have developed measures to evaluate and quantify health care quality. These metrics are increasingly used by patients, payers, and other entities to compare providers and, in some cases, determine reimbursement.

Many of the early metrics were focused on aspects of the health care system that were easy to measure (eg, process metrics) but were only indirectly connected to population health outcomes [1]. Within the field of radiology, many quality metrics are in use, and many more have been proposed, but there have been limited attempts to systematically review, collect, and describe these metrics, particularly from private practice settings.

The purpose of our study was to develop a comprehensive inventory of academic and private sector radiology quality metrics, organized and classified using a formal framework widely used by other specialties (structure, process, and outcome metrics) [1]. Because the vast majority of imaging is diagnostic in nature and thus depends on many downstream factors in order to influence patient outcomes, we hypothesized that the majority of imaging metrics would be either structural or process oriented and that only a minority would be directly linked to patient or case outcomes.

Section snippets

Methods

To ascertain quality metrics used in academic and private practice settings, a systematic review of the literature was conducted as well as a survey of radiology benefit management groups (RBMs). Because the study involved preexisting, publically available databases and did not involve patients, institutional review board approval was not obtained. The study was based at a large academic medical center.

Results

Our initial search strategy yielded 1,816 unique citations (Fig. 1). These citations were reviewed by two individuals (C.C. and A.N.), with agreement noted in 1,649 of the citations (90.8%). Differences were resolved by consensus, yielding a total of 110 articles for detailed review. Independent detailed review of the 110 articles was performed, and disagreements were resolved by consensus, yielding 25 articles. Four additional articles were derived from searching reference lists, yielding a

Discussion

Historically, providers have been reimbursed per procedure, which is thought to have contributed to the overutilization of health care services without concomitant improvements in population health outcomes [42]. With rapidly increasing health care costs, public and private entities have begun to move away from a pure fee-for-service system toward value-based reimbursement. Although many definitions of “value” are in use, all generally have some form of aggregate quality metric in the numerator

Take-Home Points

■
Numerous metrics have been developed to evaluate the quality of medical imaging services delivered in both academic and private settings.
■
When classified according to the framework of Donabedian, imaging quality metrics are significantly more likely to be structural (46%), as opposed to process based (27%) or outcomes based (27%).
■
The metrics in use by RBMs are even more likely to be structural (77%), with comparatively less use of process (11%) or outcome (11%) metrics by those entities.

References (28)

D.W. Lee et al.
Radiology benefit managers: cost saving or cost shifting?
J Am Coll Radiol
(2011)
J.D. Schuur et al.
Assessment of Medicare’s imaging efficiency measure for emergency department patients with atraumatic headache
Ann Emerg Med
(2012)
S. Patel
Value management program: performance, quantification, and presentation of imaging value-added actions
J Am Coll Radiol
(2015)
J. You-Ling Shyu et al.
Performance measures in radiology
J Am Coll Radiol
(2014)
D.J. Durand et al.
The health care value transparency movement and its implications for radiology
J Am Coll Radiol
(2015)
A. Donabedian
Evaluating the quality of medical care 1966
Milbank Q
(2005)
National Imaging Associates. Home page. Available at: http://www.niahealthcare.com. Accessed October...
AIM Specialty Health. Home page. Available at: http://www.aimspecialtyhealth.com. Accessed October...
HealthHelp. Home page. Available at: http://www.healthhelp.com. Accessed October...

Care to Care. Home page. Available at: http://www.caretocare.com. Accessed October...

CareCore National. Home page. Available at: https://www.carecorenational.com. Accessed October...

MedSolutions. Home page. Available at: http://www.medsolutions.com. Accessed October...

N. Anumula et al.

Physician Quality Reporting System

AJNR Am J Neuroradiol

(2011)

Cited by (21)

Disparities in Ultrasound Use for Diagnosing Pediatric Appendicitis Across United States Emergency Departments
2024, Journal of Surgical Research
An ultrasound (US)-first approach for evaluating appendicitis is recommended by the American College of Radiology. We sought to assess the access to and utilization of an US-first approach for children with acute appendicitis in United States Emergency Departments.
Utilizing the 2019 Nationwide Emergency Department Sample, we performed a retrospective cohort study of patients <18 y with a primary diagnosis of acute appendicitis based on International Classification of Disease 10th Edition Diagnosis codes. Our primary outcome was the presentation to a hospital that does not perform US for children with acute appendicitis. Our secondary outcome was the receipt of a US at US-capable hospital. We developed generalized linear models with inverse-probability weighting to determine the association between patient characteristics and outcomes.
Of 49,703 total children, 24,102 (48%) received a US evaluation. The odds of presenting at a hospital with no US use were significantly higher for patients aged 11-17 compared to patients <6 y (adjusted odds ratio [aOR] [95% confidence interval (CI)]: 1.59, [1.19- 2.13], P = 0.002); lowest median household income quartile compared to highest (aOR [95% CI]: 2.50, [1.52-4.10], P < 0.001); rural locations compared to metropolitan (aOR [95% CI]: 8.36 [5.54-12.6], P < 0.001), and Hispanic compared to non-Hispanic White (aOR [95% CI]: 0.63 [0.45-0.90], P = 0.01). The odds of receiving a US at US-capable hospitals were significantly lower for patients >6 y, lowest median household income quartiles, and rural locations (P < 0.05).
Rural, older, and poorer children are more likely to present to hospitals that do not utilize US in the diagnosis of acute appendicitis and are less likely to undergo US at US-capable hospitals.
Key performance indicators for value-based reimbursement in radiology: A review
2023, Procedia Computer Science
Value-based reimbursement is increasingly discussed in association with efforts to develop modern healthcare systems value-based. Relevant information (clinical, economic) should support the reimbursement systems that mirror the actual cost and benefits of a healthcare service. The reimbursement model should be understandable by decision makers and commensurate with the hospital information system (HIS). HIS should be aligned with the clinical and administrative data collection required to support the characterization of the care episode and allow a calculation of the benefits and costs incurred. The significant amount of information that HIS can gather results in an information overload that healthcare managers and professionals hardly can simplify. KPIs are crucial measurement tools capable of compiling and aggregating such information's. Radiology departments are highly technology-related service, operator-dependent, involving multiple stakeholders and considered as an adjunct to quality care, but not a primary contributor to value. Current radiology reimbursement levels are based on the same over the past 20 years criteria. Given the complexity and the inadequate criteria the first stage for new reimbursement models development, the aim of this study, is the identification of appropriate metrics to be used to monitor and evaluate the quality of services provided. The identified 160 metrics were aggregated and synthetized according to the radiology value chain in seven categories in a total of 56 value KPI's.
Predictors of Clinical Outcomes in Pediatric Appendicitis: Role of the Individual Sonographer and Radiologist When Using a First-Line Ultrasound Approach
2021, Journal of the American College of Radiology
Citation Excerpt :
Peer review, patient and provider satisfaction, and diagnostic accuracy are commonly used as quality measures in radiology departments. However, such quality and safety metrics are intermediary process metrics with unclear linkage to patient outcome [1,2]. As the paradigm of health care delivery and reimbursement shifts toward outcome measures and value-based care, leaders and researchers in quality and safety have become interested in identifying and validating novel outcome measures for use in radiology departments [1,3-5].
To determine if differences between individual sonographers and radiologists performing and interpreting appendix ultrasound affect clinically important outcomes in children with suspected acute appendicitis.
Patients <18 years of age who presented to our emergency department (ED) with suspected acute appendicitis and underwent an appendix ultrasound were identified. Sonographers who performed fewer than 100 examinations and radiologists who interpreted fewer than 100 examinations during the study period were excluded. Multivariable logistic regression was performed to assess the effect of sonographer, radiologist, clinical variables, and system factors on key clinical outcomes, including hospital admission and appendectomy.
In all, 9,283 appendix ultrasounds (mean age, 9.9 ± 4.2 years; 5,400 [58.2%] boys) performed by 31 sonographers (mean number of examinations, 299 ± 140 [range, 115-610]) and interpreted by 31 radiologists (mean number of examinations, 299 ± 157 [range, 101-845]) were included. Mean admission frequency per sonographer was 34.0% ± 3.3% (range, 27.8%-42.6%) and per radiologist was 33.5% ± 3.9% (range, 23.7%-41.6%). Mean appendectomy frequency per sonographer was 20.3% ± 2.6% (range, 14.9%-27.0%) and per radiologist was 20.3% ± 3.1% (range, 15.2%-28.7%). Significant multivariable predictors of hospital admission included temperature (P < .0001), white blood cell count (P < .0001), male sex (P = .002), imaging performed at the main hospital (versus satellite hospital) (P = .001), abdominal tenderness with ultrasound transducer compression (P < .0001), presence of rebound tenderness (P = .001), and presence of acute appendicitis by ultrasound (P < .0001), but not sonographer or radiologist. Predictors of appendectomy included weight (P < .0001), white blood cell count (P < .0001), male sex (P = .0004), abdominal tenderness with ultrasound transducer compression (P < .0001), and the presence of acute appendicitis by ultrasound (P < .0001), but not sonographer or radiologist.
Differences in individual sonographers and radiologists did not predict clinically important outcomes in children undergoing ultrasound in the ED for suspected acute appendicitis.
Radiology Patient Outcome Measures: Impact of a Departmental Pay-for-Performance Initiative on Key Quality and Safety Measures
2021, Journal of the American College of Radiology
Citation Excerpt :
Such metrics specific to radiology could relate to a wide range of activities including improving imaging safety, enhancing report communication, implementing effective peer review systems, and engaging in hospital policy-setting activities [2,13,14]. A systematic review of radiology quality measures identified as many as 75 unique measures spanning structural, process, and outcome-related data, including patient access times, report communication timeliness, procedure complication rates, and patient and referring physician satisfaction [14,15]. However, there is a relative paucity of data on which measures should be implemented in radiology and how to effectively assess their impact for sustainable improvement efforts [10,16].
Assess impact of a multifaceted pay-for-performance (PFP) initiative on radiologists’ behavior regarding key quality and safety measures.
This institutional review board–approved prospective study was performed at a large, 12-division urban academic radiology department. Radiology patient outcome measures were implemented October 1, 2017, measuring report signature timeliness, critical results communication, and generation of peer-learning communications between radiologists. Subspecialty division-wide and individual radiologist targets were specified, performance was transparently communicated on an intranet dashboard updated daily, and performance was financially incentivized (5% of salary) quarterly. We compared outcomes 12 months pre- versus 12 months post-PFP implementation. Primary outcome was monthly 90th percentile time from scan completion to final report signature (CtoF). Secondary outcomes were percentage timely closed-loop communication of critical results and number of division-wide peer-learning communications. Statistical process control analysis and parallel coordinates charts were used to assess for temporal trends.
In all, 144 radiologists generated 1,255,771 reports (613,273 pre-PFP) during the study period. Monthly 90th percentile CtoF exhibited an absolute decrease of 4.4 hours (from 21.1 to 16.7 hours) and a 20.9% relative decrease post-PFP. Statistical process control analysis demonstrated significant decreases in 90th percentile CtoF post-PFP, sustained throughout the study period (P < .003). Between 95% (119 of 125, July 1, 2018, to September 30, 2018) and 98.4% (126 of 128, October 1, 2017, to December 31, 2017) of radiologists achieved >90% timely closure of critical alerts; all divisions exceeded the target of 90 peer-learning communications each quarter (range: 97-472) after January 1, 2018.
Implementation of a multifaceted PFP initiative using well-defined radiology patient outcome measures correlated with measurable improvements in radiologist behavior regarding key quality and safety parameters.
Radiology Reports: What YOU Think You're Saying and What THEY Think You're Saying
2017, Current Problems in Diagnostic Radiology
Citation Excerpt :
Lexicon refinement is also beneficial and will become increasingly important for quality measurement and compliance reporting.52 Moving forward, the radiology lexicon requires adaptation and agreement regarding terms that explain images in a reproducible manner to enhance delivery of actionable information, improve data mining, and build quality metrics.6,15,27,44,53-56 The radiologic lexicon also affects the patient in a more obvious manner.
Image interpretation and translation into written language is an imperfect process. Yet, the radiology report represents the link between radiologist's opinion and patient's images. Increased access to images through remote viewing stations has made direct communication between radiologists and clinicians less commonplace. We are interested in how accurately the descriptive contents within radiology reports convey the feelings of the radiologist to the referring clinician. We hypothesize that certain words and phrases hold different connotations for radiologists and clinicians.
A two part survey was designed. Medical specialty, level of training, and number of radiology reports read/week was contained in part I. Part II concerned the quantification of radiologists' diagnostic confidence in range percentages based on specific words and phrases. These voluntary surveys were emailed to all faculty at a single university medical center. Additional paper surveys were randomly distributed to medical students, residents, and physicians. A total of 100 completed surveys were collected (33 radiologists and 67 non-radiologists). Data was exported to EXCEL for statistical analysis. Direct comparisons were made between the survey answers from radiologists and nonradiologists.
Percentile ranges for most radiologists and non-radiologists were in agreement in 25/36 questions. However, the absolute percentage value was somewhat variable. 11/36 questions generated discrepancy between radiologists and non-radiologists. The following words and phrases were in disagreement: “diagnostic of”, “consistent with”, “compatible with”, “evidence of”, “may represent”, “normal”, “degraded by artifact”, “obscured detail”, “mildly limited”, “moderately limited”, and “nondiagnostic”.
Sound physician communication is a critical component of quality healthcare delivery. Certain words and phrases carry different meanings for radiologists and clinicians. With structured reporting becoming more prevalent, the radiology lexicon should be defined in a more concrete manner. Ambiguous terms should be eliminated all together.
Radiology Research in Quality and Safety: Current Trends and Future Needs
2017, Academic Radiology
Citation Excerpt :
In radiology, process metrics include adherence to appropriateness criteria, patient wait times, time from order to examination, repeat rates, and report turnaround time (TAT). A meta-analysis and survey of quality metrics required by six major radiology benefit management groups (RBMs) (56) demonstrated that the most common process metric was use of appropriateness criteria (85.0%), followed by report finalization time (60.0%) and patient wait time (50.0%). Of the appropriateness criteria used, the ACR Appropriateness Criteria were most commonly cited.
Promoting quality and safety research is now essential for radiology as reimbursement is increasingly tied to measures of quality, patient safety, efficiency, and appropriateness of imaging. This article provides an overview of key features necessary to promote successful quality improvement efforts in radiology. Emphasis is given to current trends and future opportunities for directing research. Establishing and maintaining a culture of safety is paramount to organizations wishing to improve patient care. The correct culture must be in place to support quality initiatives and create accountability for patient care. Focused educational curricula are necessary to teach quality and safety-related skills and behaviors to trainees, staff members, and physicians. The increasingly complex healthcare landscape requires that organizations build effective data infrastructures to support quality and safety research. Incident reporting systems designed specifically for medical imaging will benefit quality improvement initiatives by identifying and learning from system errors, enhancing knowledge about safety, and creating safer systems through the implementation of standardized practices and standards. Finally, validated performance measures must be developed to accurately reflect the value of the care we provide for our patients and referring providers. Common metrics used in radiology are reviewed with focus on current and future opportunities for investigation.

View all citing articles on Scopus

: Dr Carrino has received personal fees from Pfizer, personal fees from BioClinica, and nonfinancial support from GE, outside the submitted work. Dr Durand has received other support from Evolent Health, personal fees and other support from Gerson Lehrman Group Consulting, other support from RadiologyResponse.com, other support from National Decision Support Company, outside the submitted work.

View full text

Original articleClinical practice managementQuality Measurements in Radiology: A Systematic Review of the Literature and Survey of Radiology Benefit Management Groups

Abstract

Purpose

Methods

Results

Conclusions

Introduction

Section snippets

Methods

Results

Discussion

Take-Home Points

J Am Coll Radiol

Ann Emerg Med

J Am Coll Radiol

J Am Coll Radiol

J Am Coll Radiol

Evaluating the quality of medical care 1966

Milbank Q

Physician Quality Reporting System

AJNR Am J Neuroradiol

Original article
Clinical practice management
Quality Measurements in Radiology: A Systematic Review of the Literature and Survey of Radiology Benefit Management Groups