A multi-resolution approach for spinal metastasis detection using deep Siamese neural networks

Highlights

• A multi-resolution approach is proposed to detect spinal metastasis in MRI.

• The multi-resolution approach is implemented using deep Siamese neural networks.

• A slice-based aggregation method is used to minimize the number of false positives.

• The proposed approach detects spinal metastasis accurately and effectively.

Abstract

Spinal metastasis, a metastatic cancer of the spine, is the most common malignant disease in the spine. In this study, we investigate the feasibility of automated spinal metastasis detection in magnetic resonance imaging (MRI) by using deep learning methods. To accommodate the large variability in metastatic lesion sizes, we develop a Siamese deep neural network approach comprising three identical subnetworks for multi-resolution analysis and detection of spinal metastasis. At each location of interest, three image patches at three different resolutions are extracted and used as the input to the networks. To further reduce the false positives (FPs), we leverage the similarity between neighboring MRI slices, and adopt a weighted averaging strategy to aggregate the results obtained by the Siamese neural networks. The detection performance is evaluated on a set of 26 cases using a free-response receiver operating characteristic (FROC) analysis. The results show that the proposed approach correctly detects all the spinal metastatic lesions while producing only 0.40 FPs per case. At a true positive (TP) rate of 90%, the use of the aggregation reduces the FPs from 0.375 FPs per case to 0.207 FPs per case, a nearly 44.8% reduction. The results indicate that the proposed Siamese neural network method, combined with the aggregation strategy, provide a viable strategy for the automated detection of spinal metastasis in MRI images.

Introduction

Spinal metastasis, a metastatic cancer to the spine, is a malignant process in the spine. It is 25–35 times more common than any other malignant diseases in the spine [1] and affects more than 100,000 individuals in the U.S. annually [2]. The spine is the third most common site for cancer cells to metastasize [3], following the lung and the liver [4]. More than 80% of spinal metastasis in adults came from the primary tumors, including breast (72%), prostate (84%), lung (31%), thyroid (50%), kidney (37%), and pancreas (33%) [5]; and 30–90% of cancer patients who die are found to have spinal metastasis in cadaver studies [6]. In addition, spinal metastases can also have a huge impact on quality of life, with complications including pain, fracture, and spinal cord and nerve root compression [7]. Therefore, the detection, diagnosis, and treatment of spinal metastases are clinically important both to save patients’ lives and to improve their quality of life.

Due to its excellent soft tissue resolution, magnetic resonance imaging (MRI) is the most sensitive imaging modality for evaluating spinal lesions [8], [9]. Various studies have shown that early stages of spinal metastasis in the bone marrow can be detected with MRI before any bone deterioration [5]. In MRI images, neoplastic involvement in the vertebral body typically shows focal bone marrow replacement with tumorous tissue, resulting in lower T1 signal than adjacent skeletal muscle and accompanying high T2 signal. Therefore, MRI images acquired with different pulse sequences (denoted as MRI sequences) can be used to locate lesions and evaluate the extent of the disease (e.g. involving single or multiple segments). However, despite the advantages mentioned above, the manual detection of spinal metastasis in MRI is time-consuming and tedious considering the large number of slices in each MRI sequence, as well as the large number of MRI sequences usually acquired for each patient. Therefore, it is now becoming essential to develop computerized algorithms for automated detection of spinal metastases in MRI sequences.

Especially, as image acquisition speed improves, many more images with a higher spatial resolution can be acquired during an examination, and as such developing computer-aided analysis methods is essential to assist radiologists in making a thorough evaluation of the entire image set within a reasonable reading time. This is a trend for all imaging modalities (and all pathologies), and particularly true for MRI because of the multiple sets of images acquired using different sequences. Although at the present time computer-aided analysis cannot yet replace visual inspection by trained radiologists, nonetheless it can already provide an important tool for displaying the most critical information from hundreds of images, in a convenient way, to assist radiologists during diagnosis. In time, computerized methods may become as good, or even better than human experts and lead to significant economies of scale and accuracy. Thus, in short, it is important to develop computerized methods to analyze MRI and other imaging modalities in medicine.

Given the importance of automated spinal metastasis detection, a few approaches have been developed in the literature. For example, Roth et al. [10] used a deep convolutional neural network (CNN) as the 2nd tier of a two-tiered, coarse-to-fine, cascade framework to refine the candidate lesions from the first tier for sclerotic spine metastasis detection in computer tomography (CT) images. Wiese et al. [11] developed an automatic method based on a watershed algorithm and graph cut for detecting sclerotic spine metastases in CT images. And Yao et al. [12] applied a support vector machine to refine the initial detections produced with a watershed algorithm for lytic bone metastasis detection in CT images. However, as can be seen, these studies are based on CT images and do not use spinal MRI sequences.

On the MRI side, efforts for analyzing MRI sequences have focused on different problems. For example, Carballido-Gamio et al. [13] developed a normalized cut method for vertebra segmentation in spinal MRI. Huang et al. [14] proposed an AdaBoost method for vertebra detection and an iterative normalized cut algorithm for vertebra segmentation in spinal MRI. Neubert et al. [15] designed an automatic method using statistical shape analysis and registration of gray level intensity profiles for 3D intervertebral disc and vertebral body segmentation in MRI. However, in spite of these efforts and to the best of our knowledge, there is no study in the literature that is focused on detecting spinal metastases in MRI sequences.

Automated and accurate spinal metastasis detection in MRI is a difficult task, in large part because of the considerable variability in the size of vertebrae. Spinal metastases usually grow in the vertebrae, which are divided into five regions: cervical, thoracic, lumbar, sacrum, and coccyx. The size of the vertebrae varies considerably both within an individual, as well as across individuals. For example, Zhou et al. [16] investigated the lumbar vertebrae from 126 CT images and found that the upper vertebral width is $40.9\pm 3.6\mathrm{mm}$ in females and $46.1\pm 3.2\mathrm{mm}$ in males at L3, $46.7\pm 4.7\mathrm{mm}$ in females and $50.8\pm 3.7\mathrm{mm}$ in males at L4, and $50.4\pm 4.4\mathrm{mm}$ in females and $54.5\pm 4.9\mathrm{mm}$ in males at L5.

In recent years [17], neural networks and deep learning have been used to successfully tackle a variety of problems in engineering, ranging from computer vision [18], [19], [20] to speech recognition [21], as well as in the natural sciences, in areas ranging from high energy physics [22], [23], to chemistry [24], [25], and to biology [26], [27]. Thus it is natural to consider applying deep learning methods also to biomedical images. For example, Ciresan et al. applied a deep learning architecture to each pixel for addressing problems of membrane segmentation in electron microscopy images [28]; Shen et al. developed multi-scale convolutional neural network for lung nodule detection in CT images [29]; Wang et al. adopted a GoogLeNet-based method for automated detection and diagnosis of metastatic breast cancer in whole slide images of sentinel lymph node biopsies [30]; and Wang et al. devised a 12-layer convolutional neural network for cardiovascular disease detection in mammograms [31]. Therefore it is natural to hypothesize that neural networks and deep learning methods can be harnessed for the effective detection of spinal metastases in MRI sequences.

Thus, in short, the purpose of this study is to develop an accurate computerized method to locate metastatic cancer in the spine using deep learning methods. Specifically, a multi-resolution analysis is proposed to deal with the large variability of vertebral sizes, and a Siamese neural network [32], [33] is developed to incorporate the multi-resolution representation of each MRI slice. More precisely, for each location under consideration, a set of image patches centered at this location and at different resolutions are extracted from the MRI slice, resulting in a multi-resolution representation of the input. Then the multi-resolution image patches are fed into a Siamese neural network to predict the probability that the corresponding location corresponds to a metastatic lesion. The Siamese neural network is composed of several identical networks, each dealing with patches of a different resolution. To further remove false positives (FPs) in spinal metastasis detection, we consider the structural similarity of neighboring slices in MRI sequences and aggregate the outputs of the Siamese neural networks using a weighted averaging approach.

The rest of paper is organized as follows. The proposed Siamese neural network and slice-based aggregation methods for spinal metastasis detection are descried in Section 2 together with other methodological aspects. The data collection and experiments are described in Section 3. Finally, the results are presented in Section 4 and discussed in Section 5.