Multimodal breast lesion classification using cross-attention deep networks

Hung Q. Vo, Pengyu Yuan, Tiancheng He, Stephen T.C. Wong, Hien V. Nguyen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Accurate breast lesion risk estimation can significantly reduce unnecessary biopsies and help doctors decide optimal treatment plans. Most existing computer-aided systems rely solely on mammogram features to classify breast lesions. While this approach is convenient, it does not fully exploit useful information in clinical reports to achieve the optimal performance. Would clinical features significantly improve breast lesion classification compared to using mammograms alone? How to handle missing clinical information caused by variation in medical practice? What is the best way to combine mammograms and clinical features? There is a compelling need for a systematic study to address these fundamental questions. This paper investigates several multimodal deep networks based on feature concatenation, cross-attention, and co-attention to combine mammograms and categorical clinical variables. We show that the proposed architectures significantly increase the lesion classification performance (average area under ROC curves from 0.89 to 0.94). We also evaluate the model when clinical variables are missing.

Original languageEnglish (US)
Title of host publicationBHI 2021 - 2021 IEEE EMBS International Conference on Biomedical and Health Informatics, Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665403580
DOIs
StatePublished - 2021
Event2021 IEEE EMBS International Conference on Biomedical and Health Informatics, BHI 2021 - Virtual, Online, Greece
Duration: Jul 27 2021Jul 30 2021

Publication series

NameBHI 2021 - 2021 IEEE EMBS International Conference on Biomedical and Health Informatics, Proceedings

Conference

Conference2021 IEEE EMBS International Conference on Biomedical and Health Informatics, BHI 2021
Country/TerritoryGreece
CityVirtual, Online
Period7/27/217/30/21

Keywords

  • Attention deep networks
  • Breast cancer
  • Breast lesion
  • Multimodal deep networks

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Information Systems and Management
  • Health Informatics
  • Health(social science)

Fingerprint

Dive into the research topics of 'Multimodal breast lesion classification using cross-attention deep networks'. Together they form a unique fingerprint.

Cite this