Diverse data augmentation for learning image segmentation with cross-modality annotations

Xu Chen, Chunfeng Lian, Li Wang, Hannah Deng, Tianshu Kuang, Steve H. Fung, Jaime Gateno, Dinggang Shen, James J. Xia, Pew Thian Yap

Research output: Contribution to journalArticlepeer-review

39 Scopus citations


The dearth of annotated data is a major hurdle in building reliable image segmentation models. Manual annotation of medical images is tedious, time-consuming, and significantly variable across imaging modalities. The need for annotation can be ameliorated by leveraging an annotation-rich source modality in learning a segmentation model for an annotation-poor target modality. In this paper, we introduce a diverse data augmentation generative adversarial network (DDA-GAN) to train a segmentation model for an unannotated target image domain by borrowing information from an annotated source image domain. This is achieved by generating diverse augmented data for the target domain by one-to-many source-to-target translation. The DDA-GAN uses unpaired images from the source and target domains and is an end-to-end convolutional neural network that (i) explicitly disentangles domain-invariant structural features related to segmentation from domain-specific appearance features, (ii) combines structural features from the source domain with appearance features randomly sampled from the target domain for data augmentation, and (iii) train the segmentation model with the augmented data in the target domain and the annotations from the source domain. The effectiveness of our method is demonstrated both qualitatively and quantitatively in comparison with the state of the art for segmentation of craniomaxillofacial bony structures via MRI and cardiac substructures via CT.

Original languageEnglish (US)
Article number102060
Pages (from-to)102060
JournalMedical Image Analysis
StatePublished - Jul 2021


  • Data augmentation
  • Disentangled representation learning
  • Generative adversarial learning
  • Medical image segmentation
  • Magnetic Resonance Imaging
  • Neural Networks, Computer
  • Image Processing, Computer-Assisted
  • Humans

ASJC Scopus subject areas

  • Radiological and Ultrasound Technology
  • Health Informatics
  • Radiology Nuclear Medicine and imaging
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Diverse data augmentation for learning image segmentation with cross-modality annotations'. Together they form a unique fingerprint.

Cite this