A Fine-Grained Benchmark for Ophthalmic Foundation Models with Point-Based Classification and Retrieval

Abstract General-purpose vision foundation models have been evaluated across a wide range of tasks to demonstrate the versatility of their learned representations. By contrast, ophthalmic foundation models have largely been assessed using a narrow set of downstream tasks, primarily disease classification and segmentation, providing only a limited view of their representational capabilities. Considering how clinicians interpret ophthalmic images, where disease-relevant clinical signs are identified before diagnosis, we introduced an ophthalmic foundation model evaluation framework comprising two tasks: 1) point-supervised disease classification and 2) point-guided clinical sign-based image retrieval. Each task incorporates sparse point annotations indicating disease-relevant evidence locations. We applied this approach to diabetic retinopathy using colour fundus photography (CFP) and age-related macular degeneration using both CFP and optical coherence tomography (OCT) scans. Our evaluation spanned 11 foundation models across four newly curated and six established datasets. The results reveal that existing ophthalmic foundation models, despite their strong performance on disease classification, still struggle with the clinical sign retrieval task, although they outperform general-purpose foundation models. Our findings highlight the need to improve these models' capabilities at the fine-grained sign-level. We publicly release our benchmark to facilitate further research on ophthalmic foundation models.