Longitudinal multiple sclerosis lesion segmentation: Resource and challenge

Aaron Carass; Snehashis Roy; Amod Jog; Jennifer L Cuzzocreo; Elizabeth Magrath; Adrian Gherman; Julia Button; James Nguyen; Ferran Prados; Carole H Sudre; Manuel Jorge Cardoso; Niamh Cawley; Olga Ciccarelli; Claudia A M Wheeler-Kingshott; Sébastien Ourselin; Laurence Catanese; Hrishikesh Deshpande; Pierre Maurel; Olivier Commowick; Christian Barillot; Xavier Tomas-Fernandez; Simon K Warfield; Suthirth Vaidya; Abhijith Chunduru; Ramanathan Muthuganapathy; Ganapathy Krishnamurthi; Andrew Jesson; Tal Arbel; Oskar Maier; Heinz Handels; Leonardo O Iheme; Devrim Unay; Saurabh Jain; Diana M Sima; Dirk Smeets; Mohsen Ghafoorian; Bram Platel; Ariel Birenbaum; Hayit Greenspan; Pierre-Louis Bazin; Peter A Calabresi; Ciprian M Crainiceanu; Lotta M Ellingsen; Daniel S Reich; Jerry L Prince; Dzung L Pham

doi:10.1016/j.neuroimage.2016.12.064

Longitudinal multiple sclerosis lesion segmentation: Resource and challenge

Neuroimage. 2017 Mar 1:148:77-102. doi: 10.1016/j.neuroimage.2016.12.064. Epub 2017 Jan 11.

Authors

Aaron Carass¹, Snehashis Roy², Amod Jog³, Jennifer L Cuzzocreo⁴, Elizabeth Magrath², Adrian Gherman⁵, Julia Button⁴, James Nguyen⁴, Ferran Prados⁶, Carole H Sudre⁷, Manuel Jorge Cardoso⁸, Niamh Cawley⁹, Olga Ciccarelli⁹, Claudia A M Wheeler-Kingshott⁹, Sébastien Ourselin⁸, Laurence Catanese¹⁰, Hrishikesh Deshpande¹⁰, Pierre Maurel¹⁰, Olivier Commowick¹⁰, Christian Barillot¹⁰, Xavier Tomas-Fernandez¹¹, Simon K Warfield¹¹, Suthirth Vaidya¹², Abhijith Chunduru¹², Ramanathan Muthuganapathy¹², Ganapathy Krishnamurthi¹², Andrew Jesson¹³, Tal Arbel¹³, Oskar Maier¹⁴, Heinz Handels¹⁴, Leonardo O Iheme¹⁵, Devrim Unay¹⁵, Saurabh Jain¹⁶, Diana M Sima¹⁶, Dirk Smeets¹⁶, Mohsen Ghafoorian¹⁷, Bram Platel¹⁸, Ariel Birenbaum¹⁹, Hayit Greenspan²⁰, Pierre-Louis Bazin²¹, Peter A Calabresi⁴, Ciprian M Crainiceanu⁵, Lotta M Ellingsen²², Daniel S Reich²³, Jerry L Prince²⁴, Dzung L Pham²

Affiliations

¹ Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA. Electronic address: aaron_carass@jhu.edu.
² CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20892, USA.
³ Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA.
⁴ Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
⁵ Department of Biostatistics, The Johns Hopkins University, Baltimore, MD 21205, USA.
⁶ Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK; NMR Research Unit, UCL Institute of Neurology, WC1N 3BG London, UK.
⁷ Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK.
⁸ Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK; Dementia Research Centre, UCL Institute of Neurology, WC1N 3BG London, UK.
⁹ NMR Research Unit, UCL Institute of Neurology, WC1N 3BG London, UK.
¹⁰ VisAGeS: INSERM U746, CNRS UMR6074, INRIA, University of Rennes I, France.
¹¹ Computational Radiology Laboratory, Boston Childrens Hospital, Boston, MA 02115, USA; Harvard Medical School, Boston, MA 02115, USA.
¹² Biomedical Imaging Lab, Department of Engineering Design, Indian Institute of Technology, Chennai 600036, India.
¹³ Centre For Intelligent Machines, McGill University, Montréal, QC H3A 0E9, Canada.
¹⁴ Institute of Medical Informatics, University of Lübeck, 23538 Lübeck, Germany.
¹⁵ Bahçeşehir University, Faculty of Engineering and Natural Sciences, 34349 Beşiktaş, Turkey.
¹⁶ icometrix, 3012 Leuven, Belgium.
¹⁷ Institute for Computing and Information Sciences, Radboud University, 6525 HP Nijmegen, Netherlands.
¹⁸ Diagnostic Image Analysis Group, Radboud University Medical Center, 6525 GA Nijmegen, Netherlands.
¹⁹ Department of Electrical Engineering, Tel-Aviv University, Tel-Aviv 69978, Israel.
²⁰ Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv 69978, Israel.
²¹ Department of Neurophysics, Max Planck Institute, 04103 Leipzig, Germany.
²² Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Electrical and Computer Engineering, University of Iceland, 107 Reykjavík, Iceland.
²³ Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA; Translational Neuroradiology Unit, National Institute of Neurological Disorders and Stroke, Bethesda, MD 20892, USA.
²⁴ Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA.

Abstract

In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website² as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.

Keywords: Magnetic resonance imaging; Multiple sclerosis.

Publication types

Research Support, N.I.H., Intramural
Research Support, Non-U.S. Gov't
Research Support, N.I.H., Extramural

MeSH terms

Adult
Algorithms
Female
Humans
Image Processing, Computer-Assisted
Imaging, Three-Dimensional
Longitudinal Studies
Magnetic Resonance Imaging
Male
Middle Aged
Multiple Sclerosis / diagnostic imaging*
Observer Variation
White Matter / diagnostic imaging

Abstract

Publication types

MeSH terms

Grants and funding