Rationale and objectives: In assessing diagnostic accuracy it is often essential to determine the reader's ability both to detect and to correctly locate multiple abnormalities per patient. The authors developed a new approach for the detection and localization of multiple abnormalities and compared it with other approaches.
Materials and methods: The new approach involves partitioning the image into multiple regions of interest (ROIs). The reader assigns a confidence score to each ROI. Statistical methods for clustered data are used to assess and compare reader accuracy. The authors applied this new method to a reader-performance study of conventional film images and digitized images used to detect and locate malignant breast cancer lesions.
Results: The ROI-based approach, the free-response receiver operating characteristic (FROC) curve, and the patient-based approach handle the estimation of the false-positive rate (FPR) quite differently. These differences affect the measures of the respective areas under the curves. In the ROI-based approach the denominator is the number of ROIs without a malignant lesion. In the FROC approach the average number of false-positive findings per patient is plotted on the x axis of the curve. In contrast, the patient-based approach mishandles the FPR by ignoring multiple detection and/or localization errors in the same patient. The FROC approach does not lend itself easily to statistical evaluations.
Conclusion: The ROI-based approach appropriately captures both the detection and localization tasks. The interpretation of the ROI-based accuracy measures is simple and clinically relevant. There are statistical methods for estimating and comparing ROI-based estimates of accuracy.