Automatic Speech Recognition Performance in Psychiatric Speech: Linguistic, Clinical, and Architectural Factors

Automatic speech recognition (ASR) is essential for automated speech analysis pipelines in psychosis-spectrum disorders, yet performance on clinical speech remains understudied. We evaluated three architecturally distinct ASR models (Whisper large-v2, Canary-1B-v2, Parakeet-TDT-1.1B) on speech from 145 participants (92 psychosis-spectrum, 53 healthy controls) across seven speech tasks ranging from structured Reading Recall to spontaneous Free Speech. Word error rate (WER) was comparable between Whisper (12.1%) and Canary (12.2%), while Parakeet showed higher errors (21.8%, p < 0.001). Whisper and Canary showed higher semantic fidelity than Parakeet (BERTScore F1: 0.94 vs. 0.91, p < 0.001) and lower phonetic and semantic hallucination scores (p < 0.001). Task type was the strongest determinant of WER across all models; structured Reading Recall gave the lowest and spontaneous Free Speech the highest WER for every model. Patients showed modestly elevated WER for Parakeet, with significant diagnostic-group differences only in specific tasks. Sex and age effects on WER were small, reaching significance only for Parakeet (males; p_bonf = 0.028) and Canary (age). Positive and Negative Syndrome Scale (PANSS) scores did not significantly predict WER for Whisper or Canary, but Parakeet showed significant positive correlations with symptom severity. Second formant bandwidth variability correlated with WER. Disfluency preservation was low (recording-level means: Whisper: 21.8%, Parakeet: 4.1%) but significantly higher in patients than controls (without ah and oh, only for Canary), a difference mediated by loudness variability for disfluency but not WER. Findings are limited to English-speaking populations. In this sample, Whisper and Canary performed comparably; Whisper preserved disfluencies most.

Recommended citation: El-Mufti, N., Mackinley, M., Dzialoszynski, P., Palaniyappan, L., & Voppel, A.* (under review). Automatic Speech Recognition Performance in Psychiatric Speech: Linguistic, Clinical, and Architectural Factors.
Download Paper