Elsevier

Clinical Oncology

Volume 32, Issue 1, January 2020, Pages 13-25
Clinical Oncology

Original Article
Inter-Observer Variability in Target Volume Delineations of Benign and Metastatic Brain Tumours for Stereotactic Radiosurgery: Results of a National Quality Assurance Programme

https://doi.org/10.1016/j.clon.2019.06.015Get rights and content

Highlights

  • •

    National, multicentre study of all stereotactic radiosurgery centres in England.

  • •

    Evidence of inter-observer variability across a national population.

  • •

    Variability most evident with the smallest volumes and geometrically complex targets.

  • •

    Peer review highlighted inconsistencies in identifying acceptable and outlying contours.

  • •

    AV50, CCI and DCI metrics can be used to identify outliers of benchmarking cases.

Abstract

Aims

To quantify inter-observer variation between all the intracranial stereotactic radiosurgery (SRS) providers in England in delineating the target volumes of four brain tumour cases.

Materials and methods

Twenty-two, cross-platform SRS providers in England were instructed during a national commissioning assessment to contour the gross tumour volume (GTV) of six brain metastases, one cavernous sinus meningioma, one vestibular schwannoma and one pituitary adenoma. An expert reference group provided feedback if submitted contours were considered to be outliers and those centres were instructed to resubmit their contours. All contours were analysed in Python. The target volume contour (observed volume; VOBS), encompassing volume, 50% agreement volume (AV50), 100% agreement volume (AV100), concordance index (CCI) and discordance index (DCI) were calculated.

Results

Twenty-one centres participated using five different treatment platforms (CyberKnife, Gamma Knife, Varian Linac, Elekta Linac, Tomotherapy) and seven different treatment planning systems (GammaPlan, iPlan, Multiplan, Pinnacle, Eclipse, CMS Focal). The greatest variability was observed in the smallest brain metastases (GTV5 AV50 0.0 cm3, CCI 0.28–0.84, DCI 0.00–0.70) and pituitary case (AV50 1.1 cm3, CCI 0.42–0.82, DCI 0.01–0.40). The greatest agreement was observed with the vestibular schwannoma (AV50 2.8 cm3, CCI 0.77–0.94, DCI 0.00–0.17). There were four resubmissions for the cavernous sinus meningioma and three resubmissions for the pituitary adenoma.

Conclusions

Inter-observer variability was most evident with the smallest brain metastases and pituitary case. Several additional outliers and one acceptable contour were suggested using the metric-based analysis of AV50, CCI and DCI. Comparing contours using these metrics is an effective way to identify whether individual contours are similar to the ‘true’ target and to flag potentially significant deviations.

Introduction

With its innate observer bias, target volume delineation has been identified as the weakest point [1] in the radiotherapy planning process. Previous studies analysing target volume delineations of brain tumour lesions for stereotactic radiosurgery (SRS) have identified significant inter-observer variations [2], [3], [4], [5]. However, these publications have reviewed contouring by a limited number of specialised centres and treatment platforms that may not reflect current practice across all centres delivering SRS in a wide region.

Quality assurance bodies have proposed benchmarking of standard cases for contouring and planning in order to minimise inter-observer variation within clinical trials [6], [7]. In 2016, as part of the National Health Service (NHS) England SRS procurement process, all 22 SRS centres in England were instructed to participate in a quality assurance process in order to benchmark practice. The national trials quality assurance group (RTTQA) collaborated with an expert reference group (ERG) of radiation oncologists, neurosurgeons and medical physicists determined by NHS England to facilitate this process. All centres were instructed to contour four SRS cases according to local practice. The ERG prospectively peer reviewed the target volumes to determine if they were an acceptable standard. Outliers were identified, provided with feedback and asked to resubmit contours. Retrospectively, 50% agreement volume (AV50), concordance index (CCI) and discordance index (DCI) were calculated to identify acceptable and outlying contours. Planning benchmark cases were reported separately [8], [9].

The primary aim of this study was to assess the inter-observer variation in target volume delineation across all centres in England providing intracranial SRS. This was a unique study analysing current practice of all SRS providers across England who were using a variety of treatment planning systems (TPS) and platforms. Our secondary aim was to develop a process to increase contouring accuracy by creating a comparator tool.

Section snippets

Clinical Cases and Images

All 22 providers of intracranial SRS in England were instructed to delineate the target volumes of four benchmark SRS cases. No specific guidance was provided, except for a recommendation to collaborate with a specialist neuroradiologist.

The four cases had been selected by the ERG as representing routine SRS practice. For each case, participants were given a brief clinical history, patient demographics and the appropriate computed tomography and magnetic resonance imaging (MRI) scans in Digital

Participating Centres

Twenty-one of the 22 SRS centres submitted contours. No submissions were received from Centre 15 as its clinicians also worked at Centre 5. Centres 5 and 15 are hereafter a joint centre. All GTV outlines from Centre 4, except Case 2, were corrupted; therefore, these submissions could not be analysed. Table 1 provides an overview of the TPS, contourers and cases submitted per centre. After a prospective analysis by the ERG, four centres were instructed to resubmit a total of seven contours.

Analysis

Table

Discussion

Inter-observer variability of target volume delineations for radiotherapy has been widely discussed in the literature [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], yet variation in brain tumour target volumes for SRS has received comparatively little attention [2], [3], [4], [5], despite the explicit need for accuracy with this treatment. These published studies showing significant inter-observer variability in target volume delineation highlight the need for routine peer

Conclusions

This national, cross-platform study observed minimal variability in delineating the target volume of the vestibular schwannoma, but a high degree of variability with the smallest metastatic tumours and a pituitary case. We therefore recommend that centres focus resources on improving contouring in these areas.

Peer review has been shown to be effective; however, this method has also highlighted discrepancies in the identification of acceptable and outlying contours compared with the method of

Conflicts of Interest

R. Patel and D. Eaton received grants from NHS England during the conduct of this study. A. Cameron is a member of the expert reference group who evaluated the original submissions. No payment was received for this work.

References (23)

  • D. Genovesi et al.

    Interobserver variability of clinical target volume delineation in soft-tissue sarcomas

    Cancer Radiother

    (2014)

    • Cited by (0)

    View full text
    © 2019 The Royal College of Radiologists. Published by Elsevier Ltd. All rights reserved.