Clinical Validation and Extension of an Automated, Deep Learning–Based Algorithm for Quantitative Sinus CT Analysis

DISCUSSION

Objective quantification of sinus mucosal disease is important for several reasons. From a clinical perspective, detailed quantification would allow a more nuanced understanding of inflammatory burden and response to therapy. For researchers, quantification systems are frequently used to objectively compare disease burden among different patients in cross-sectional analysis or longitudinally within-subject to document the efficacy of therapeutic interventions. Current objective CT scoring measures may not offer the necessary sensitivity for such comparisons.

Semiquantitative visual assessment of radiologic disease severity involves visual estimation of sinus opacification and grading with standardized scoring systems. While dozens of such scoring systems exist, the most widely accepted method used today is the LMS.¹² While the LMS has demonstrated good interrater reliability and relative ease of use, a drawback of the LMS is that a score of 1 for a given cavity encompasses 99% of the radiologic disease spectrum, under which most sinus cavities will likely be classified. In other words, the LMS can distinguish those without disease and those with the most severe disease but has poor discrimination among patients with mild-to-moderate levels of disease for a given sinus cavity. Furthermore, the importance of ostiomeatal complex opacification has been reconsidered, and given the potential subjectivity of the 0-versus-2 score, it is not clear whether this measure is truly meaningful.¹³

Several groups have recognized this shortcoming and have sought to improve the LMS using computer-assisted volumetric analysis of sinus opacification.^14,15 These methods aim to calculate the percentage of sinus cavity occupied by soft-tissue density, allowing characterization of mucosal inflammation on a continuous scale. While shown to improve the clinical utility of the LMS for the most part, these staging systems have failed to gain wide acceptance due to their unwieldy implementation. These methods rely on an experienced clinician or radiologist with knowledge of complex sinus anatomy to manually segment the sinus cavities for each patient. Even when using semiautomated computerized techniques, these methods are heavily time-consuming and potentially prone to human error.

Methods that strive to create more objective computer analysis of sinus CT have surfaced only recently. Chowdhury et al¹⁶ used deep learning to classify opacification of the ostiomeatal complex on selected 2D coronal sections. Using a CNN trained on 296 CT scans from patients with CRS, the authors reported that their method was able to correctly detect ostiomeatal complex opacification 85% of the time. To our knowledge, our group was the first to develop, test, and validate a fully automated deep learning–based algorithm capable of 3D volumetric segmentation of the paranasal sinuses on CT.⁶ In this proof-of-principle study, the algorithm was tested on subjects presenting for multidisciplinary respiratory evaluation at a single institution. The results showed that algorithm-derived quantitative assessment of the total percentage sinus opacification correlated well with the current criterion standard visual grading system, the LMS.

The current study expands on this work in several ways, with an overarching goal being further demonstrating the applicability and utility of a fully automated system for disease quantification, while also exploring novel radiologic disease-specific readouts. The current algorithm is capable of individual sinus cavity segmentation, a feature that was not yet developed in our initial report of our technique. Averaging of individual sinus cavities represents an improvement in overall characterization of sinus disease burden, considering that in this work, we demonstrate an overall %SO-versus-LMS correlation of 0.85, which is an increase compared with our initial work in which we reported a correlation of 0.82. While the study cohort in this work is smaller, it is an independent cohort consisting exclusively of patients with well-characterized CRS. We used a number of disease-specific metrics that were not available in the initial study, including polyp status, endoscopic disease assessments, quality-of-life surveys, and serum/tissue eosinophil measurement. Additionally, CT scans in the present study were obtained using several different scanner manufacturers with varied reconstruction kernel protocols, illustrating potential broad applicability of such an algorithm. That this approach was capable of successfully analyzing a diverse set of scans while producing an improved association with LMS to the initial study demonstrates promise in the wide-spread application of this AI technology.

We also explored the clinical significance of novel automated CT metrics that have been proposed in the literature. The significance of Hounsfield unit values of sinus opacities has been investigated previously, with 1 study showing that when the LMS is weighted by certain Hounsfield unit values, correlation of the weighted LMS score to symptom indices increased.¹⁷ Opacification hyperdensity has also been found to be a key radiologic biomarker for certain CRS subtypes, such as allergic fungal rhinosinusitis¹⁸ and eosinophilic mucin rhinosinusitis. Our algorithm is capable of calculating the mHU of opacified regions across all sinus cavities for a given study, a readout that has not been studied before. In the current study, mHU correlations to clinical parameters, in particular tissue and serum eosinophils, were lower than we had expected. While we had hypothesized that radiologic density measures would have correlated with eosinophilic inflammatory markers based on the known relationship between allergic fungal concretions and eosinophilic mucin, there is, in fact, a dearth of published data to support this theory, probably, in part, because an objective radiologic metric examining this issue has not been easily obtainable until now. Furthermore, our study was not powered to examine opacification density characteristics for diseases in which this may be important, such as eosinophilic CRS or allergic fungal rhinosinusitis. More study is needed to examine how this newly available radiologic measure can be used in these conditions.

We also investigated osteitis of bony lamellae in the sinus cavities, an important clinical marker that typically indicates recalcitrant or long-standing CRS disease status and may portend worse outcomes after endoscopic sinus surgery.^19-22 This investigation was performed by assessing a 7-mm rind along the perimeter of each sinus cavity and determining what percentage of the rind was occupied by bone. We hypothesized that subjects with a greater degree of bony thickening would correlate to having a higher total GOSS, a currently applied radiologic measure of paranasal sinus osteitis. Our results showed a moderate correlation between the algorithm readout and the visual assessment. Stratifying the correlation by surgical status strengthened the correlation in postoperative subjects, possibly because these patients had more recalcitrant disease or from repeat insults from the surgery itself. Our initial method of assessment may be limited by patients who naturally have thicker bone but lack osteitic change; further refinement of this technique may be needed in the future.

We had hypothesized that an AI-based quantitative algorithm would offer a sensitive method to measure disease severity, and it seems to have performed similar to LMS measurements. Neither LMS nor quantitative %SO demonstrated any meaningful association with SNOT-22 scores in this cohort, supporting prior literature observations that likewise showed no relationship between CT disease burden and symptoms.^23-25 As has been proposed in other work, biologic factors may play only a partial role in the determination of symptom burden and expression,²⁶ and we are still a long way from deciphering how certain radiologic findings such as Hounsfield units can be used as biomarkers or as a method of endotyping patients with CRS.

Performing well compared with the criterion standard LMS, our AI-based approach has marked and obvious advantages in that it is rapid and completely automated and eliminates any bias or human error that may be part of a visual assessment. The algorithm, now demonstrating successful segmentation from scanners from multiple different manufacturers, also allows quantitation or detection of other important radiologic entities for clinical and research purposes, such as osteitis and assessment of opacification density.

Limitations of the current work include the retrospective study design. While use of a diverse set of scans may be argued as a potential drawback for a validation study, we think that this, in fact, represents a strength that showcases algorithm robustness with respect to varied scan-acquisition parameters, especially because initial validation work using a homogeneous cohort of scans has already been published by our group. Further validation and utility assessment will be of interest in a prospective multi-institutional trial setting.