Abstract / Summary
Precise prediction of cancer therapy response remains challenging because conventional biomarkers capture molecular features but overlook the physical state of tumors. We developed a multimodal deep-learning framework integrating ultrasound shear wave elastography (SWE) images with quantitative stiffness measurements (elastic modulus, kPa) to predict treatment outcomes in preclinical murine tumors. Each image-stiffness pair is tokenized within a transformer that learns interactions between local elastographic texture and global rigidity. A lightweight convolutional encoder extracts image features, the modulus is embedded as a numeric token, and self-attention fuses both modalities for classification. Trained on 1578 baseline SWE images from five syngeneic tumor models, the model classified tumors as responders, stable, or non-responders. Across five random seeds, it achieved 92.4% ± 1.3% accuracy, macro-F1 0.92, and ROC-AUC 0.99 on a held-out test set, with well-calibrated probabilities. Matched-split ablations-image-only, stiffness-token-removed, stiffness-shuffled, late-fusion, and stiffness-only-showed that performance reflected genuine cross-modal learning rather than scalar stiffness alone; shuffling image-stiffness pairings significantly reduced accuracy (all corrected p < 0.01). Leave-one-tumor-model-out analysis demonstrated generalization to unseen tumor types, with 95.5% ± 1.5% accuracy (range 93.9-97.9%) across five held-out models. Lower baseline stiffness correlated with better response, supporting the hypothesis that mechanically normalized tumors respond more effectively. These preclinical proof-of-concept findings establish tumor mechanics as candidate predictive biomarkers and transformer-based multimodal learning as a scalable approach to biomechanically informed response prediction, while requiring validation in human cohorts.