Domain Expansion via Network Adaptation for Solving Inverse Problems

Nebiyou Yismaw1, Ulugbek S. Kamilov2, M. Salman Asif1
1. University of California Riverside
2. Washington University in St. Louis
Paper arXiv

Fig. 1: Artifact removal (AR) networks trained on MRI scans (fastMRI AR) and face images (celebA AR) suffer from performance degradation under domain shifts, resulting in poor reconstruction quality (as indicated by PSNR and SSIM values under each image). Our proposed network (Modulated AR) adapts fastMRI AR for face image reconstruction by learning rank-one factors (modulations). The network stores shared and domain-specific modulations separately. During inference, it applies the correct modulation according to the specified domain. Our proposed network retains the performance of fastMRI AR on MR images and achieves competitive reconstruction quality with celebA AR on face images.

Abstract

Deep learning-based methods deliver state-of-the-art performance for solving inverse problems that arise in computational imaging. These methods can be broadly divided into two groups: (1) learn a network to map measurements to the signal estimate, which is known to be fragile; (2) learn a prior for the signal to use in an optimization-based recovery. Despite the impressive results from the latter approach, many of these methods also lack robustness to shifts in data distribution, measurements, and noise levels. Such domain shifts result in a performance gap and in some cases introduce undesired artifacts in the estimated signal. In this paper, we explore the qualitative and quantitative effects of various domain shifts and propose a flexible and parameter efficient framework that adapt pretrained networks to such shifts. We demonstrate the effectiveness of our method for a number of natural image, MRI, and CT reconstructions tasks under domain, measurement model, and noise-level shifts. Our experiments demonstrate that our method provides significantly better performance and parameter efficiency compared to existing domain adaptation techniques.

Impact of Domain Shift

Fig. 2: Examples of image reconstruction under domain shifts. We use two artifact removal (AR) network trained for fastMRI and celebA images in deep unrolled framework. Second and third columns show both images reconstructed with fastMRI AR and celebA AR, respectively. Reconstruction quality degrades with domain shifts (PSNR and SSIM reported under each image). Our proposed network adaptation method, where we adapt celebA AR to recover an MR image (top row) and fastMRI AR to recover a celebA image (bottom row).

Learning rank-1 factors for domain adaptation

Fig. 3: Overview of our factorized network that uses modulated convolutions for domain adaptation. Our network follows the DNCNN architecture that leverages modulated convolution for domain adaptation. After trained on the source domain, the network learns low-rank modulations for each domain while keeping the base network parameters frozen. Using a domain identifier, the network selects the appropriate low-rank factors during inference and applies them to the pretrained network through element-wise multiplication.

Domain, forward model, and noise level adaptation

We performed a number of experiments to analyze the effects of shifts in different parts of the inverse problem. The shifts can occur in the data distribution \( \mathbf{x} \), the forward model \( \mathbf{A} \), and the measurement noise \( \eta \). we start with a fixed base network, which we refer to as Base AR, and learn domain-specific rank-one modulations. Base AR is trained to reconstruct MR images from \( 4\times\) radially sub-sampled Fourier measurements without any measurement noise.

Paper

BibTeX

@misc{yismaw2023domain,
      title={Domain Expansion via Network Adaptation for Solving Inverse Problems},
      author={Nebiyou Yismaw and Ulugbek S. Kamilov and M. Salman Asif},
      year={2023},
      eprint={2310.06235},
      archivePrefix={arXiv},
      primaryClass={eess.IV}
}