Abstract / Summary
Most of the deep learning studies on glaucoma diagnosis test their models on images from thesame dataset that was used for training. Such results say little about what happens when the modelmeets a different camera, clinic or patient. In this work we measure that gap for five ImageNetpre-trained CNNs: ResNet50, DenseNet121, MobileNetV2, EfficientNet-B0 and InceptionV3. Eachnetwork was fine-tuned with identical settings on one of three public fundus datasets (ACRIMA,ORIGA and RIM-ONE DL) and tested on images of its own dataset and on the complete other twodatasets. Every experiment was repeated with three random seeds. Averaged over the five models,the in-domain AUC was 0.921 and the cross-source AUC was 0.701, a drop of 0.219 (SD 0.015;95% CI 0.183 to 0.256). No architecture performed significantly better than the others aftercorrection for multiple comparisons. At the decision threshold chosen on source validation data,mean specificity fell from 0.89 to 0.49 on other datasets. A fourth dataset, G1020, was used only fortesting; all 45 evaluations on it gave an AUC between 0.46 and 0.53. We conclude that in-domainaccuracy of fine-tuned CNNs is a poor guide to how a model will perform at a new site, and that anyglaucoma screening study should include external validation and local threshold calibration.