Computational Sensing and Information Processing Lab

Cross-modal proxy tokens approximate the class token of a missing modality by attending only to the tokens of the available modality — no modality generation, no auxiliary networks. With rank-1 LoRA adapters in frozen unimodal encoders, this beats state-of-the-art baselines across missing rates on five datasets while staying competitive with complete modalities.

A tutorial overview of robust multimodal learning, from the fundamentals of fusion, alignment, and representation learning through the architectures and models that keep performance from collapsing when modalities are noisy or missing.

SLUG unlearns targeted information by updating a single critical layer from a one-time gradient computation, using layer-importance and gradient-alignment metrics to pick that layer. It removes concrete and abstract concepts from CLIP, Stable Diffusion, and VLMs at a small fraction of the compute of existing methods.

Pretrained multimodal networks degrade sharply when a modality is absent at test time. Modulating intermediate features with fewer than 1% of the model's parameters compensates for the missing inputs, and often beats dedicated networks trained for each available modality combination — demonstrated on five tasks across seven datasets.

A mixed-modality fusion block that combines information from arbitrary modality combinations, built into a Multi-Modal Segmentation TransFormer for material and semantic segmentation. Accuracy improves monotonically as modalities are added, setting state of the art on three datasets.

Networks trained to solve inverse problems are fragile under shifts in data distribution, measurement operator, and noise level. We characterize the effects of those shifts and propose a flexible, parameter-efficient framework that adapts a pretrained network to expand its domain of validity.