When you think about ASR models and how they compare, you usually think in the terms of WER (word error rate): the lower, the better - simple as that.
However when you are a doctor and you are reading a transcript, some mistakes can be harmless while some others can be very consequential.
WER weights each word equally - but the real world rarely does that.
For a paying customer, WER is just a number - if somebody is paying money for a scribe system or a call center solution (lets not even go to intelligence / defense fields), it is often also important how the model "feels", how often it changes it mind, the type of errors it does (and how glaring its hallucinations can be), the latency you face and the impression it made on the c-suite executive that demo-tested it exactly once.
Watch a minute long transcription of a chest Xray and EKG reading (of a patient with papillary muscle rupture) being performed in parallel by both qwen asr and faster_whisper below:
Since the aim of nanosamur.ai is to be model agnostic, we are progressively adding support for more models - and now you can run multiple real time models in parallel (feature announcement), while recording / transcribing singular audio stream. nanosamur.ai hence makes for a great "Model Arena" where you can pit different models against each other for a real-time transcription and watch how they act right next to their competitor.
WER is misleading clinically here: Faster-Whisper was better on the decisive medical term ST elevation, while Qwen’s SGL elevation is meaningless. We see here an issue with an LLM-based ASR model: qwen took the liberty of changing cephalization → stenosis which frankly is a faux pas.
| Model | Sample WER | Important strengths | Important errors |
|---|---|---|---|
| Qwen | 2.7% | Better prose, EKG, correct “should be noted” |
cephalization → stenosis; ST → SGL twice |
| Faster-Whisper | 4.7% | Correct cephalization; correctly recognized both ST elevations |
lead → leed/leaf; inserted not in “should not be noted”; EKG → NKJ |
Apart from the transcription mishaps, with this real time "arena fight" one can observe an interesting thing: faster-whisper goes silent at around 28 second mark and the lag spikes. This is on the very challenging medical jargon that has low-confidence decode and (highly likely) leads to multiple temperature fallbacks:
"There are T wave inversions in lead III and aVF as well as about 0.5 to 1 millimeter ST elevations in lead III, and about 0.5 millimeter ST elevation in lead aVF."
While the faster whisper butchered the "leads", it at least did not perform such as glaring mishaps as conjuring "SGL elevation", and its transcription makes sense.
So there we have it: WER will always be useful, but there is also a qualitative side to each particular field in which you would use an ASR model, and you have to assess them accordingly. And when push comes to shove - and to budgeting decisions - it is always good to have a platform that is model agnostic, because if users feel they have a choice between models, they will be much less nervous should they (inevitably) spot some mistakes.