Synthetic Modality Fusion in Breast Doppler Ultrasound, ACS Lab
My ACS Lab research focuses on multimodal medical image analysis for breast ultrasound. The central question is how a model can learn the relationship between tumor structure in B-mode ultrasound and vascular flow patterns in Doppler ultrasound through B-mode/Doppler fusion, especially when reliable pixel-level annotations are limited.
- Built and compared pair-based classification pipelines: Doppler-only baseline, input-level early fusion, two-stream feature fusion, one-way and bi-directional cross-attention, and gated fusion.
- Used ConvNeXt/ConvNeXtV2 backbones with stratified cross-validation, weighted cross entropy, ensemble testing, and metric analysis for Type 1/2/3 ultrasound classification.
- Designed tumor-flow interaction features using HSV-based Doppler flow masks, pseudo tumor masks, Grad-CAM, CAM2SAM, MedSAM, and relation maps such as intratumoral and peritumoral flow.
- Explored generative data construction with RePaint, LaMa, GPT-Image-1, Imagen, and Stable Diffusion to synthesize or restore ultrasound modalities while checking whether synthetic images are useful for downstream diagnosis.
- Evaluated synthetic Doppler realism with MLLM-as-a-Judge and VLM evaluation using models such as MedGemma and Qwen2.5-VL, and refined prompts through K-Means clustering of real Doppler morphology.
- Analyzed failure cases caused by class imbalance and modality conflict, then revised data construction, input design, and learning strategy to stabilize model behavior.
In short, this work is not just medical image classification. It is a reliable multimodal AI problem: deciding which modality carries useful clinical signal, how to fuse it, and how to keep generated medical images from contaminating the learned representation.
