Original Article
Empirical Bayes estimates generated in a hierarchical summary ROC analysis agreed closely with those of a full Bayesian analysis

https://doi.org/10.1016/j.jclinepi.2003.12.019Get rights and content

Abstract

Background and objective

A range of fixed-effect and random-effects meta-analytic methods are available to obtain summary estimates of measures of diagnostic test accuracy. The hierarchical summary receiver operating characteristic (HSROC) model proposed by Rutter and Gatsonis in 2001 represents a general framework for the meta-analysis of diagnostic test studies that allows different parameters to be defined as a fixed effect or random effects within the same model. The Bayesian method used for fitting the model is complex, however, and the model is not widely used. The objective of this report is to show how the model may be fitted using the SAS procedure NLMIXED and to compare the results to the fully Bayesian analysis using an example.

Methods

The HSROC model, its assumptions, and its interpretation are described. The advantages of this model over the usual summary ROC (SROC) regression model are outlined. A complex example is used to compare the estimated SROC curves, expected operating points, and confidence intervals using the alternative approaches to fitting the model.

Results

The empirical Bayes estimates obtained using NLMIXED agree closely with those obtained using the fully Bayesian analysis.

Conclusion

This alternative and more straightforward method for fitting the HSROC model makes the model more accessible to meta-analysts.

Introduction

The systematic review of the results of primary studies is becoming increasingly important for assessing and summarizing evidence about the accuracy of diagnostic tests. Guidelines for the conduct of such systematic reviews [1], [2] include defining the objective of the review, retrieval of the relevant literature, data extraction, meta-analytic methods for obtaining summary estimates of test accuracy, and investigating reasons for variation in test accuracy across studies.

The aim of the present report is to (i) provide a brief outline of commonly used approaches for the meta-analysis of diagnostic studies and their limitations, (ii) outline the hierarchical summary receiver operating characteristic (HSROC) model and its advantages, and (iii) describe how maximum likelihood estimation can be used to fit the HSROC model using the NLMIXED procedure in SAS [3]. An example is used to illustrate the method and a sample program is provided. The results are compared with the estimates published for the same data using Bayesian Markov chain Monte Carlo (MCMC) methods [4].

Section snippets

Commonly used methods for meta-analysis of diagnostic tests

Most methods used for the meta-analysis of diagnostic studies use a single estimate of sensitivity and specificity derived from each study. Commonly used approaches are here outlined briefly.

HSROC: hierarchical SROC model

An alternative approach for fitting SROC curves has been proposed by Rutter and Gatsonis [19]. It is based on the ordinal logistic regression model of McCullagh [20], [21] that has been used by Tosteson and Begg [22] to fit an ROC curve when data are available at multiple thresholds in a single study. The model allows for asymmetry in the ROC through inclusion of a scale parameter that determines the shape of the ROC. Rutter and Gatsonis [4], [19] have applied this model to the estimation of a

Example: methods for fitting SROC curves applied to comparison of imaging methods for the detection of lymph node metastases in women with invasive cancer of the cervix

The data are taken from a meta-analysis conducted by Scheidler et al. [31] to compare the global accuracy of three types of diagnostic imaging—lymphangiography (LAG), computed tomography (CT), and magnetic resonance (MR)—to detect lymph node metastases in women diagnosed with invasive cancer of the cervix. A total of 36 studies are included in the analysis and provide 46 estimates of test sensitivity and specificity: 17 for LAG, 19 for CT, and 10 for MR. The observed sensitivity and specificity

Conclusions

The HSROC model provides a general framework for the meta-analysis of diagnostic studies. It allows the meta-analyst to investigate heterogeneity between studies while taking into account both within- and between-study variability. The example considered here compares the accuracy of different imaging methods; however, the same approach can also be applied to assess whether the SROC, and the expected operating point on the SROC, differs according to patient or study characteristics (or both).

References (35)

  • L. Irwig et al.

    Meta-analytic methods for diagnostic test accuracy

    J Clin Epidemiol

    (1995)
  • M.D. Mitchell

    Validation of the summary ROC for diagnostic test meta-analysis: a Monte Carlo simulation

    Acad Radiol

    (2003)
  • E.A. Engels et al.

    Meta-analysis of diagnostic tests for acute sinusitis

    J Clin Epidemiol

    (2000)
  • E.C. Vamvakas

    Meta-analyses of studies of the diagnostic accuracy of laboratory tests: a review of the concepts and methods

    Arch Pathol Lab Med

    (1998)
  • W.L. Deville et al.

    Guidelines for conducting systematic reviews of studies evaluating the accuracy of diagnostic tests

  • SAS Institute

    SAS/STAT user's guide. Version 8

    (1999)
  • C.M. Rutter et al.

    A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations

    Stat Med

    (2001)
  • J.J. Deeks

    Systematic reviews of evaluations of diagnostic and screening tests

  • L. Irwig et al.

    Designing studies to ensure that estimates of test accuracy are transferable

    BMJ

    (2002)
  • J.W.P.F. Kardaun et al.

    Comparative diagnostic performance of three radiological procedures for the detection of lumbar disk herniation

    Methods Inf Med

    (1990)
  • L.E. Moses et al.

    Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations

    Stat Med

    (1993)
  • B. Littenberg et al.

    Estimating diagnostic accuracy from multiple conflicting reports: a new meta-analytic method

    Med Decis Making

    (1993)
  • M.T. Fahey et al.

    Meta-analysis of Pap test accuracy

    Am J Epidemiol

    (1995)
  • D.F. Ransohoff et al.

    Problems of spectrum and bias in evaluating the efficacy of diagnostic tests

    N Engl J Med

    (1978)
  • J.G. Lijmer et al.

    Exploring sources of heterogeneity in systematic reviews of diagnostic tests

    Stat Med

    (2002)
  • J.B. Carlin

    Meta-analysis for 2 × 2 tables: a Bayesian approach

    Stat Med

    (1992)
  • A.S. Midgette et al.

    A meta-analytic method for summarizing diagnostic test performances: receiver-operating-characteristic-summary point estimates

    Med Decis Making

    (1993)
  • Cited by (165)

    View all citing articles on Scopus
    View full text