Abstract / Summary
BACKGROUND: Artificial intelligence (AI) medical scribes rely on speech-to-text (STT) models for transcription. Evaluations of STT models in non-English settings remain scarce. We benchmarked ten STT models on medical consultations from Latin American (LatAm) Spanish and assessed whether fine-tuning improves transcription accuracy. METHODS: Ten videos depicting medical consultations. Human transcriptions were the ground truth. Five closed-source models were evaluated: gpt-4o-transcribe, gpt-4o-mini-transcribe, gemini-2.5-pro, Eleven Labs, and Assembly AI; and so were five open-source models: Whisper Large, Whisper Large v3, Whisper Large v3 Turbo, Voxtral Mini 3B, and Canary 1B v2. Whisper Large v3, an open-source model, was fine-tuned. One video was withheld from training. Performance assessed using Word Error Rate (WER), Character Error Rate (CER), BLEU Score, ROUGE-L, BERT Score, and Semantic Similarity on the one withheld video. RESULTS: None of the fine-tuning iterations outperformed the vanilla Whisper Large v3. When evaluated on the held-out test video, the fine-tuned open-source Whisper Large v3 did not outperform any of the top closed-source models; conversely, relative to other open-source models, the fine-tuned Whisper Large v3 had superior performance across all metrics, such as in BLEU score (63% vs. 58% for the second-best model), BERT Score (89% vs. 86%), and semantic similarity (89% vs. 83%). CONCLUSIONS: Whisper Large v3 and its fine-tuned variant are the best open-source STT models for transcribing medical conversations in LatAm Spanish. These findings provide evidence for developing AI medical scribes tailored to Spanish-speaking LatAm.