Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT

Roger Y. Kim, Jason L. Oke, Lyndsey C. Pickup, Reginald F. Munden, Travis L. Dotson, Christina R. Bellinger, Avi Cohen, Michael J. Simoff, Pierre P. Massion, Claire Filippini, Fergus V. Gleeson, Anil Vachani

Research output: Contribution to journalArticlepeer-review

24 Scopus citations


Background Limited data are available regarding whether computer-aided diagnosis (CAD) improves assessment of malignancy risk in indeterminate pulmonary nodules (IPNs). Purpose To evaluate the effect of an artificial intelligence-based CAD tool on clinician IPN diagnostic performance and agreement for both malignancy risk categories and management recommendations. Materials and Methods This was a retrospective multireader multicase study performed in June and July 2020 on chest CT studies of IPNs. Readers used only CT imaging data and provided an estimate of malignancy risk and a management recommendation for each case without and with CAD. The effect of CAD on average reader diagnostic performance was assessed using the Obuchowski-Rockette and Dorfman-Berbaum-Metz method to calculate estimates of area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Multirater Fleiss κ statistics were used to measure interobserver agreement for malignancy risk and management recommendations. Results A total of 300 chest CT scans of IPNs with maximal diameters of 5-30 mm (50.0% malignant) were reviewed by 12 readers (six radiologists, six pulmonologists) (patient median age, 65 years; IQR, 59-71 years; 164 [55%] men). Readers' average AUC improved from 0.82 to 0.89 with CAD ( P < .001). At malignancy risk thresholds of 5% and 65%, use of CAD improved average sensitivity from 94.1% to 97.9% ( P = .01) and from 52.6% to 63.1% ( P < .001), respectively. Average reader specificity improved from 37.4% to 42.3% ( P = .03) and from 87.3% to 89.9% ( P = .05), respectively. Reader interobserver agreement improved with CAD for both the less than 5% (Fleiss κ, 0.50 vs 0.71; P < .001) and more than 65% (Fleiss κ, 0.54 vs 0.71; P < .001) malignancy risk categories. Overall reader interobserver agreement for management recommendation categories (no action, CT surveillance, diagnostic procedure) also improved with CAD (Fleiss κ, 0.44 vs 0.52; P = .001). Conclusion Use of computer-aided diagnosis improved estimation of indeterminate pulmonary nodule malignancy risk on chest CT scans and improved interobserver agreement for both risk stratification and management recommendations. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Yanagawa in this issue.

Original languageEnglish (US)
Pages (from-to)683-691
Number of pages9
Issue number3
StatePublished - Sep 2022


  • Aged
  • Artificial Intelligence
  • Female
  • Humans
  • Lung Neoplasms/diagnostic imaging
  • Male
  • Multiple Pulmonary Nodules/diagnostic imaging
  • Retrospective Studies
  • Sensitivity and Specificity
  • Tomography, X-Ray Computed/methods

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging


Dive into the research topics of 'Artificial Intelligence Tool for Assessment of Indeterminate Pulmonary Nodules Detected with CT'. Together they form a unique fingerprint.

Cite this