Skip to main content
Advertisement

Main menu

  • Home
  • Content
    • Current Issue
    • Publication Preview--Ahead of Print
    • Past Issue Archive
    • Case of the Week Archive
    • Classic Case Archive
    • Case of the Month Archive
  • For Authors
  • About Us
    • About AJNR
    • Editors
    • American Society of Neuroradiology
  • Submit a Manuscript
  • Podcasts
    • Subscribe on iTunes
    • Subscribe on Stitcher
  • More
    • Subscribers
    • Permissions
    • Advertisers
    • Alerts
    • Feedback
  • Other Publications
    • ajnr

User menu

  • Subscribe
  • Alerts
  • Log in

Search

  • Advanced search
American Journal of Neuroradiology
American Journal of Neuroradiology

American Journal of Neuroradiology

  • Subscribe
  • Alerts
  • Log in

Advanced Search

  • Home
  • Content
    • Current Issue
    • Publication Preview--Ahead of Print
    • Past Issue Archive
    • Case of the Week Archive
    • Classic Case Archive
    • Case of the Month Archive
  • For Authors
  • About Us
    • About AJNR
    • Editors
    • American Society of Neuroradiology
  • Submit a Manuscript
  • Podcasts
    • Subscribe on iTunes
    • Subscribe on Stitcher
  • More
    • Subscribers
    • Permissions
    • Advertisers
    • Alerts
    • Feedback
  • Follow AJNR on Twitter
  • Visit AJNR on Facebook
  • Follow AJNR on Instagram
  • Join AJNR on LinkedIn
  • RSS Feeds
LetterLETTER

Further Examination of Diagnostic Performance in the Context of a Fellows' Journal Club Article

R.E. Carter and V.T. Lehman
American Journal of Neuroradiology June 2012, 33 (6) E96-E97; DOI: https://doi.org/10.3174/ajnr.A3187
R.E. Carter
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
V.T. Lehman
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • References
  • PDF
Loading

We reviewed the article “The Predictive Value of 3D Time-of-Flight MR Angiography in Assessment of Brain Arteriovenous Malformation Obliteration after Radiosurgery” by Buis et al1 at a recent neuroradiology journal club meeting. In this context, the article was thoroughly reviewed with each element carefully critiqued. While other aspects of the study were questioned, in particular the specific MR images used for AVM assessment after radiosurgery and the consistency of use of the abbreviation “PO” to designate “probable obliteration” as a measure of degree of confidence, we have focused our attention in this letter on the reported measures of diagnostic performance. This letter was not written to critique the aforementioned article but rather to highlight teaching points that are made possible by the article.

In the article, the authors define “sensitivity” as “the probability of finding obliteration on MRI2 among those images demonstrating complete obliteration of DSA2c” and “specificity” as “the probability of finding a patent nidus among those whose images demonstrated no obliterations on DSA2c.” Based on these definitions, the reference standard is the DSA diagnosis and the index test is the MR imaging findings. The definitions are considered atypical because sensitivity, as defined, represents the condition of absence instead of the more traditional presence of a diseased condition. Nevertheless, these definitions are mathematically sound.

The estimation of sensitivity and specificity requires binary decisions: The reference standard is positive or negative; the index test is positive or negative. In the usual way, sensitivity is TP/(TP + FN), where TP is the number of true-positive cases (positive index test and positive reference standard) and FN is the number of false-negatives (negative index test and positive reference standard). Likewise, specificity is TN/(TN + FP), where TN is the number of true-negative cases (negative index test and negative reference standard) and FP is the number of false-positives (positive index test and negative reference standard).

In the context of the article, the application of these standard definitions is not straightforward because Table 3 in the article does not use binary decisions. The MR imaging findings are presented as a trichotomous variable with Patent, PO (partial or probable obliteration), and DO (definitive obliteration) categories. To form a binary classification, these 3 distinct categories need to be combined into 2 values: absent or present. The methods do not provide this decision rule, but by using the authors' definition for sensitivity, 1 grouping of the MR imaging findings would be to treat DO as synonymous with “obliteration” so that sensitivity for reader 1 would equal 61.5% (48/78). This calculation does not match the results reported in their Table 4. Sensitivity for reader 1 is reported as 52%. One possible explanation for the difference is that data were combined in a different manner. The only other option is combining PO with DO values to represent obliteration. This yields a sensitivity of 80.8% (63/78) for reader 1. This calculation still does not agree with the results presented in their Table 4. Using a similar strategy, one could continue to perform calculations for reader 2 and other measures of diagnostic performance reported in their Table 4 and reach the conclusion that the numbers presented are not supported by the data in their Table 3. Did the authors make calculation mistakes or is there another explanation?

The explanation for the discrepancy is that the authors have calculated the diagnostic performance summaries by using the MR imaging findings as the reference standard and the DSA results as the index test with a reversed definition of disease present. Accordingly, one is able to reproduce all numbers in their Table 4 if one combines the Patent and PO categories into reference standard positive (ie, nidus present with certainty or probable certainty) and considers DSA patent as test positive. For example, the “sensitivity” and “specificity” by using these amended definitions for reader 1 would be 52.4% (33/63) and 88.9% (48/54), respectively. Thus while the numbers reported in their Table 4 are reproducible, the meaning of the indices has been altered, with the reversal of the disease-positive and -negative classification and the switching of the reference standard and index test.

This raises the second teaching point: How is 52.4% interpreted if it is not “sensitivity” as defined by the authors? The value actually represents a sample estimate of the positive predictive value (PPV) of MR imaging findings (ie, DSA as the reference standard). Specifically, there would be 33 TP cases and 30 FP cases, so that the PPV would be 33/(33 + 30) or 52.4%. This estimate is only valid in a simple random sample design that measures both MR imaging and DSA results on all cases. This is to ensure that the disease prevalence is not altered experimentally. Approximately half of the cases were not included in the analysis, so it would be unreasonable to assume that the disease prevalence was not altered. Practically speaking, the study may be “enriched” with reference standard–positive cases because the observed prevalence is reported as 67% (78/117). When this occurs, the PPV should be estimated by using estimates of sensitivity, specificity, and the disease prevalence in the general screening population by using this formula: PPV = [Sensitivity × Pr(Disease)]/[Sensitivity × Pr(Disease) + (1 − Specificity) × (1 − Pr(Disease)], where Pr(Disease) represents the disease prevalence and Sensitivity and Specificity are in their decimal (probability) forms.

A similar formula exists for NPV.2 The Table presents PPV and NPV values for various disease-prevalence values. For these calculations, the sensitivity and specificity are estimated as 61.5% (48/78) and 84.6% (33/39) on the basis of the performance of reader 1. The numbers would be different on the basis of the performance of reader 2. The Table illustrates that the disease prevalence has a profound impact on both PPV and NPV.

View this table:
  • View inline
  • View popup

PPV and NPV for time-of-flight MR imaging as a screening test for complete obliteration for various disease-prevalence valuesa

In summary, the article by Buis et al1 emphasizes the need for specific reporting of the decision rules for combining multicategory ratings into the dichotomous ratings required for diagnostic performance calculations. Careful attention to the reference standard and its adjudication is required to interpret sensitivity and specificity correctly. Finally, one must be cautious when interpreting PPV and NPV by ensuring that the disease prevalence is representative of the general screening population and has not been altered through the inclusion/exclusion criteria of the study or by being aware that the disease prevalence may be affected by artifacts of missing data.

References

  1. 1.↵
    1. Buis DR,
    2. Bot JC,
    3. Barkhof F,
    4. et al
    . The predictive value of 3D time-of-flight MR angiography in assessment of brain arteriovenous malformation obliteration after radiosurgery. AJNR Am J Neuroradiol 2012; 33: 232– 38
    Abstract/FREE Full Text
  2. 2.↵
    1. Zhou XH,
    2. Obuchowski NA,
    3. McClish DK
    . Statistical Methods in Diagnostic Medicine. 2nd ed. Hoboken, New Jersey: John Wiley and Sons; 2011
  • © 2012 by American Journal of Neuroradiology
PreviousNext
Back to top

In this issue

American Journal of Neuroradiology: 33 (6)
American Journal of Neuroradiology
Vol. 33, Issue 6
1 Jun 2012
  • Table of Contents
  • Index by author
Advertisement
Print
Download PDF
Email Article

Thank you for your interest in spreading the word on American Journal of Neuroradiology.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Further Examination of Diagnostic Performance in the Context of a Fellows' Journal Club Article
(Your Name) has sent you a message from American Journal of Neuroradiology
(Your Name) thought you would like to see the American Journal of Neuroradiology web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Further Examination of Diagnostic Performance in the Context of a Fellows' Journal Club Article
R.E. Carter, V.T. Lehman
American Journal of Neuroradiology Jun 2012, 33 (6) E96-E97; DOI: 10.3174/ajnr.A3187

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Further Examination of Diagnostic Performance in the Context of a Fellows' Journal Club Article
R.E. Carter, V.T. Lehman
American Journal of Neuroradiology Jun 2012, 33 (6) E96-E97; DOI: 10.3174/ajnr.A3187
del.icio.us logo Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One
Purchase

Jump to section

  • Article
    • References
  • Figures & Data
  • Info & Metrics
  • References
  • PDF

Related Articles

  • Reply:
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Crossref
  • Google Scholar

This article has not yet been cited by articles in journals that are participating in Crossref Cited-by Linking.

More in this TOC Section

  • Pattern Recognition in Mitochondrial Leukodystrophies is Hampered by the Peculiarities of Mitochondrial Genetics
  • Callosal Angle Narrowing in Research Data Bases of the Cognitively Impaired
  • Reply:
Show more Letters

Similar Articles

Advertisement

News and Updates

  • Lucien Levy Best Research Article Award
  • Thanks to our 2021 Distinguished Reviewers
  • Press Releases

Resources

  • Evidence-Based Medicine Level Guide
  • How to Participate in a Tweet Chat
  • AJNR Podcast Archive
  • Ideas for Publicizing Your Research
  • Librarian Resources
  • Terms and Conditions

Opportunities

  • Share Your Art in Perspectives
  • Get Peer Review Credit from Publons
  • Moderate a Tweet Chat

American Society of Neuroradiology

  • Neurographics
  • ASNR Annual Meeting
  • Fellowship Portal
  • Position Statements

© 2022 by the American Society of Neuroradiology | Print ISSN: 0195-6108 Online ISSN: 1936-959X

Powered by HighWire