Qualifying Certainty in Radiology Reports through Deep Learning–Based Natural Language Processing

Article Figures & Data

Download figure
Open in new tab
Download powerpoint
FIG 1.
Overview of the QC-RAD system workflow.
Download figure
Open in new tab
Download powerpoint
FIG 2.
Illustration of using BERT for certainty classification. The input “Findings suggestive of stroke” was classified as “Definitive-Mild.”
Download figure
Open in new tab
Download powerpoint
FIG 3.
Performance curve across the number of fine-tuning epochs. The left figure is for BERT, and the right one is for BioBERT.
Download figure
Open in new tab
Download powerpoint
FIG 4.
Receiver operating characteristic curves of individual classes on the test data set. Class 0 = Non-Definitive, class 1 = Definitive-Mild, class 2 = Definitive-Strong, and class 3 = Other.

Table 1:

Diagnostic certainty of diagnosis in the Impression section of a radiology report—categories for annotation

Certainty Categories	Interpretation	Examples
Non-Definitive	Describing differential diagnoses without indicating any confidence or only findings without any diagnosis	“Less likely differential considerations include demyelinating/inflammatory processes”
Definitive-Strong	Describing discrete diagnostic findings without hedging words	“Stable right sphenoid intraosseous lipoma”
Definitive-Mild	Describing discrete diagnostic findings with hedging words	“Findings suggestive of Arnold Chiari I malformation”
Other	Describing recommendations, imaging techniques, prior studies	“Another follow-up is recommended”

Table 2:

Data statistics of the 3 data sets^{^a}

Table 3:

Performance comparison among 3 BERT variants (with their optimal parameters) on the validation data set

Model	No of Epochs	Batch Size	Learning Rate	Macro-Sensitivity (%) (95% CI)	Macro-Specificity (%) (95% CI)	Macro-AUC (95% CI)
BERT-base	4	24	0.00003	79.46 (68.02–87.82)	93.65 (89.26–96.46)	0.928 (0.883–0.973)
BioBERT	6	32	0.00003	79.08 (67.13–87.78)	93.13 (88.58–96.13)	0.931 (0.886–0.975)
ClinicalBERT	5	32	0.00005	78.52 (66.91–87.07)	93.19 (88.57–96.25)	0.925 (0.878–0.971)

Note:—Macro indicates the average on the macro level across different categories.

Table 4:

System performance of BioBERT on the test data set^{^a}

Category	Sensitivity (%) (95% CI)	Specificity (%) (95% CI)	AUC (95% CI)
Non-Definitive	76.71 (56/73) (65.35–85.81)	90.24 (148/164) (84.64–94.32)	0.919 (0.874–0.964)
Definitive-Mild	59.52 (25/42) (43.28–74.37)	88.72 (173/195) (83.42–92.79)	0.843 (0.76–0.92)
Definitive-Strong	74.6 (47/63) (62.06–84.73)	95.4 (166/174) (91.14–97.99)	0.964 (0.931–0.997)
Other	98.31 (58/59) (90.91–99.96)	97.19 (173/178) (93.57–99.08)	0.994 (0.979–1)
Macro Avg	77.29 (65.4–86.22)	92.89 (88.19–96.05)	0.93 (0.888–0.972)

Note:—Macro Avg indicates average on the macro level across different categories.
↵a Numerators and denominators for sensitivity and specificity are included in parentheses.