Abstract

General-purpose speech recognisers make many errors on Czech medical speech. We adapted two models using 12.7 hours of medically themed Czech audio:

  • an offline model, Whisper large-v3;
  • a realtime streaming model, Nemotron 3.5 ASR streaming 0.6B.

Most training labels were old transcripts with mistakes that nobody corrected. A single reviewer corrected 3.9 hours by hand, mainly for evaluation.

On three held-out recordings with unseen speakers:

  • Whisper with a LoRA adapter reduced the word error rate (WER) from 11.0% to 6.9%. The adapter adds 0.25% extra parameters and trained in 10 minutes on one laptop GPU. Medical-term accuracy rose from 82% to 87%.
  • Nemotron, fully fine-tuned and run as a 160 ms streaming model, went from 33.1% to 16.7% WER.

Both models were converted to their production runtimes and deployed with small accuracy loss. Four results may be useful to others:

  • uncorrected old transcripts were almost as good as corrected labels for training;
  • adapter size did not matter;
  • a vocabulary prompt did not help the adapted model and harmed the stock model;
  • a small amount of corrected in-domain data still gave a significant gain after more uncorrected data had stopped helping.
Figure 1: word error rate and medical-term accuracy on the test set, stock vs fine-tuned, for Whisper large-v3 and Nemotron streaming

1. Introduction

Hospitals want speech recognition for dictation, consultation notes and meeting minutes. For Czech, the usual models are weak on three things:

  • medical vocabulary (drug names, diagnoses, procedures);
  • conversational speech: hesitations, repetitions, colloquial word endings;
  • the spelling conventions a clinical record expects.

Collecting and transcribing clinical audio is expensive. We asked how far a small amount of in-domain audio goes, when most of its labels come from an automatic transcriber and only a few hours are corrected by a human.

We study two deployment targets of one speech platform:

  • Offline transcription of finished recordings, where accuracy matters most. Whisper large-v3 (1.55B parameters, encoder–decoder) served through WhisperX / faster-whisper.
  • Realtime transcription while people speak, where latency matters. Nemotron 3.5 ASR streaming 0.6B, a cache-aware FastConformer transducer (RNN-T) run in a C++ runtime with 160 ms chunks.

2. Data

Source. 37 recordings of medically themed Czech audio, 12.7 hours in total.

  • Each recording is 15–35 minutes of spontaneous conversation between two or more speakers on a medical topic.
  • Topics include cardiology, hepatology, transplantation, allergy, oncology, infectious diseases and pharmacy.
  • The audio is recorded with good microphones in quiet rooms.

Each recording comes with a Czech transcript with segment timings.

Segmentation. Each recording was cut into clips of 5–28 seconds at transcript segment boundaries, so that a clip does not cut a sentence where possible. Clips outside this range (mostly non-speech sounds or single words) were dropped. Audio is 16 kHz mono.

Splits by recording and speaker. Dev and test have 3 recordings each. No speaker who appears in dev or test appears in any training recording.

Split Recordings Clips Hours Labels
Train 31 1,856 10.4 305 clips (1.7 h, 5 recordings) corrected by hand; 1,551 clips (8.7 h, 26 recordings) uncorrected automatic transcripts
Dev 3 202 1.1 all corrected by hand
Test 3 194 1.0 all corrected by hand

Human review. One reviewer listened to 737 clips (3.9 h) and corrected the original transcript where it differed from what was said.

  • Fillers ("eh", "e", "no"), repetitions and colloquial forms were kept verbatim.
  • 265 clips (36%) needed a correction.
  • On the dev and test recordings, the original transcripts differ from the corrected text in about 1.2–1.3% of words.

Text conventions. Labels keep case, punctuation and digits as in the original transcripts. Scoring lowercases the text, removes punctuation and keeps diacritics.

3. Methods

3.1 Whisper large-v3 with LoRA

  • Base model: openai/whisper-large-v3 at a pinned revision, frozen in bf16.
  • LoRA adapters: rank 8, alpha 16, dropout 0.05, on the query and value projections of every attention block in encoder and decoder (192 matrices). That is 3.9M trainable parameters, 0.25% of the model.
  • Optimisation:
    • AdamW, learning rate 5e-4, weight decay 0.01;
    • 5% linear warmup, then linear decay over 3 epochs;
    • effective batch 8 (micro-batch 4 × 2 accumulation);
    • gradient clipping 1.0, seed 42.
  • The decoder is trained with the Czech language token and the transcribe task, without timestamps.
  • Model selection: greedy dev WER every 40 steps, early stopping with patience 3.

The final model was trained on train plus dev (2,058 clips, 11.5 h). With no dev set left, training stops at a fixed step: the best dev step of the previous run, scaled to the larger data set (311 steps, about 1.2 epochs). Training took 10 minutes on an RTX 5090 Laptop GPU (24 GB), with a peak of 14 GB.

For deployment, the adapter was merged into the base weights in fp32 and saved in fp16. It was then converted to CTranslate2 fp16, the format faster-whisper and WhisperX load.

3.2 Nemotron streaming transducer

  • Base model: nvidia/nemotron-3.5-asr-streaming-0.6b, a cache-aware FastConformer encoder with an RNN-T decoder, a 13k-token vocabulary and a language prompt. It is the model already used for realtime transcription. The production setting is 160 ms chunks (attention context [56, 1]).
  • Two arms:
    • A. LoRA: rank 32 on all 144 encoder linear layers; decoder and joint trained densely. LoRA is merged before export, because the runtime has no adapter support.
    • B. Full fine-tune: all 638M parameters.
  • Shared settings: AdamW, cosine schedule with 5% warmup, peak learning rate 3e-4 (LoRA) or 3e-5 (full), bf16, 8 epochs, about 40 s of audio per step.
  • The language prompt is fixed to cs-CZ. The model's own multi-latency training is kept: the right context is sampled per step.
  • Model selection on dev at the 160 ms production setting, every half epoch.

The standard RNN-T loss needed about 13 GB per 30 s clip, because it keeps the full time × label × vocabulary tensor. We replaced it with an exact, memory-light implementation:

  • the joint network is evaluated in time chunks under checkpointing;
  • only the blank and next-label probabilities are kept.

It matches the reference loss to four decimals. Each run took 75–82 minutes.

For deployment, the fine-tuned model was converted to the runtime's Q8_0 GGUF format with the converter version used in production. Converting the stock model this way reproduces the production file bit for bit.

3.3 Evaluation

  • Decoding:
    • Whisper: beam 5, Czech forced, no timestamps.
    • Nemotron: greedy, cache-aware streaming with the cs-CZ prompt.
  • Metrics:
    • WER, and WER without fillers. Nemotron emits no fillers before fine-tuning, so WER without fillers is the fair comparison there.
    • Character error rate (CER).
    • Medical-term accuracy: the share of occurrences of 153 hand-picked medical word stems in the dev/test references that a system writes exactly right.
  • Significance: paired bootstrap with 5,000 resamples, once over clips and once over recordings. We call a difference significant when the 95% interval excludes zero.
  • Discipline: all decisions were made on dev. The final test numbers were computed once per finished model.

4. Results

4.1 Main result

Test set: 3 recordings, 194 clips.

System WER WER w/o fillers CER Terms exact
Whisper large-v3, stock 11.00 9.80 5.19 82.0
Whisper large-v3 + LoRA (train only) 7.40 6.68 2.41 84.0
Whisper large-v3 + LoRA (train + dev), final 6.93 6.13 — 87.0
Nemotron streaming 160 ms, stock 33.11 32.21 13.57 50.8
Nemotron 160 ms, LoRA arm 17.35 16.75 5.39 70.7
Nemotron 160 ms, full fine-tune 16.74 16.13 5.15 74.6
Nemotron 1.12 s, full fine-tune (not deployed) 15.06 14.41 4.59 79.3
Figure 1: word error rate and medical-term accuracy on the test set, stock vs fine-tuned, for Whisper large-v3 and Nemotron streaming
Figure 1. Test set, stock vs fine-tuned. Left: word error rate (lower is better). Right: share of medical terms written exactly right (higher is better). Whisper is the final model; Nemotron is the full fine-tune at the 160 ms production setting.

Relative WER reduction: 37% for Whisper and 49% for Nemotron.

  • Nemotron vs stock: the paired interval for the gain is −17.7 to −14.5 points over clips and −23.2 to −12.4 over recordings.
  • Whisper vs stock: paired intervals between systems on this test set are about ±0.3–0.4 points, so the 4-point gain is far outside them.
  • Final Whisper model vs the train-only adapter: 0.47 points better (clip interval −0.82 to −0.15, recording interval −0.57 to −0.30). All three test recordings improved.

4.2 What drove the Whisper gain

We trained in steps, scoring each on dev and, for completeness, on the test set of that time. Early steps used a single-recording test set, so the absolute numbers differ from Table 4.1.

Step Change Test WER (1 recording)
Whisper medium, stock — 22.1
+ LoRA on 1.7 h reviewed data adaptation 17.9
+ 5.7 h uncorrected data (7.4 h) more data 15.3
Whisper large-v3, stock larger base 12.9
large-v3 + LoRA on 7.4 h both 8.8
Figure 2: Whisper word error rate by training step on the earlier single-recording test set
Figure 2. Whisper word error rate at each step, on the earlier single-recording test set (about 20 minutes). Gray: stock models; blue: adapted. The larger base model gave the biggest single drop; adapting it then removed about a third of its remaining errors.

Base model size mattered most. The stock large-v3 model already beat the adapted medium model.

Corrected labels were barely better than automatic ones. On the same 305 training clips, two seeds each, dev WER was 15.5% with uncorrected labels and 15.2% with corrected labels. The difference is not significant: uncorrected labels gave about 92% of the gain. So we scaled training with uncorrected transcripts and spent review time on dev and test.

More uncorrected data stopped helping at about 7 hours. Going from 7.4 h to 10.4 h changed nothing (dev +0.02, test −0.14 points, within noise). The final model's gain came from adding 1.1 h of corrected speech from three new recordings (Section 4.1).

Adapter size did not matter. Same data and recipe, dev / test WER:

Adapter Trainable params Dev Test
r8, query/value 3.9M 6.63 7.40
r32, query/value 15.7M 6.70 7.77
r16, all linear layers 28.8M 6.98 7.68

The larger adapters were no better and pointed the wrong way.

Where the remaining errors are. After adaptation, most errors are substitutions. Many of them are convention differences rather than misheard words:

  • "e" vs "eh";
  • colloquial vs standard forms ("kterej" vs "který", "bejt" vs "být");
  • spelling variants ("bakterie" vs "baktérie");
  • digits vs words.

Adaptation fixed dropped fillers and repetitions, which the stock model removes. It also fixed many medical terms.

4.3 Vocabulary prompting

Whisper can be conditioned on a text prompt, which WhisperX exposes as initial_prompt / hotwords. We tested it on dev and test with these prompts:

  • rare words from the short title and summary that accompany each recording;
  • the same words from a different recording, as a placebo;
  • an oracle list of half of the evaluation terms;
  • the full summary text.
Prompt (adapted model) Dev WER change Test WER change
keywords of the recording −0.10 [−0.29, +0.09] −0.04 [−0.22, +0.13]
keywords of another recording −0.11 [−0.29, +0.07] −0.08 [−0.24, +0.08]
oracle term list (half) −0.19 [−0.39, +0.01] −0.14 [−0.36, +0.07]
full summary +0.14 [−0.23, +0.68] +0.03 [−0.19, +0.26]

None of these changes is significant, and the correct keywords did no better than the placebo.

  • The oracle list raised accuracy on the listed terms by 2–6.5 points, but slightly lowered accuracy on unlisted terms.
  • The full summary made the model skip opening sentences that repeat it.

On the stock model, every prompt raised WER, by 0.4 to 1.4 points (significant on test for keywords), because the model skips the start of the clip. Our reading: adaptation teaches the vocabulary better than a prompt can, and short prompts are only worth it for session-specific names.

4.4 Streaming model

Zero-shot, the streaming model was weak on this speech: 32% WER without fillers on test. It often drifted into Slovak, wrote numbers as words, and emitted no fillers. Fine-tuning:

  • halved the error;
  • removed the Slovak drift completely (31 Slovak marker words on dev before, 0 after);
  • taught the model digits and fillers;
  • raised medical-term accuracy from 51% to 75%.

The full fine-tune beat the LoRA arm by 0.6–0.7 points. This is significant over clips; on test, the recording-level interval touches zero. Longer context (1.12 s) gave another 1.4–1.7 points, but adds latency.

After adaptation, the streaming model is still 9.5 points behind Whisper on test, measured without fillers (interval +8.5 to +10.5). The gap is roughly what one would expect between a 0.6B model seeing 160 ms ahead and a 1.55B model reading the whole clip.

4.5 From training framework to production runtime

Model Training framework Production runtime Note
Whisper final (test) 6.93 (Transformers, bf16) 7.06 (CTranslate2 fp16) 154 of 194 clips identical
Whisper stock (test) 11.00 11.71 same runtime check
Nemotron full (dev, w/o fillers) 14.08 (NeMo fp32) 14.31 (Q8 runtime, deployed service) after the fix below

In the C++ runtime, the fully fine-tuned streaming model sometimes repeated its last word when the audio stopped mid-word (5 of 202 dev clips). Pushing one second of silence before the stream is closed removed all of these loops. It also brought the runtime within 0.2 points of the reference implementation.

4.6 Compute

Everything ran on one laptop GPU (RTX 5090 Laptop, 24 GB).

Job Time Peak GPU memory
Whisper LoRA, final run 10 min 14 GB
Whisper evaluation, beam 5, one split (about 1 h of audio) 7 min 12 GB
Nemotron fine-tune (per arm, incl. 17 dev evaluations) 75–82 min 9–13 GB

The whole study, including every arm and the evaluation runs, used well under a day of GPU time.

5. Discussion and limitations

  • One domain, one recording setup. The audio is clean, well-paced conversation. Dictation, ward rounds, telephone audio and noisy rooms are untested. We expect smaller gains there until similar data is added.
  • Small evaluation set. Dev and test have three recordings each, about one hour of speech, so the bootstrap over recordings is coarse.
    • Several early systems were scored on an older single-recording test set, which is one of the three current test recordings.
    • The final comparisons use the full three-recording set, and the training stop step was chosen on dev.
  • References start from original uncorrected transcripts. Corrected references inherit their segmentation and conventions.
    • That probably favours a model trained on such transcripts.
    • Part of the gain is the model learning the transcription convention (fillers, spelling) rather than hearing better.
    • WER without fillers and term accuracy are reported to partly separate these effects. A second annotator would help.
  • Other languages were not measured. Both deployed models replace the stock weights for all languages.
    • Whisper was trained with a small adapter and still transcribes an English sample word for word as before, but that is a smoke test.
    • The streaming model was fully fine-tuned on Czech only, without mixing in other languages, so some forgetting is likely.
    • A multilingual regression test is the next step before wider use.
  • Little hyperparameter tuning. The streaming model was trained with one learning rate per arm, and Whisper's learning rate was set early. Small additional gains from tuning are possible.

6. Conclusion

A few hours of in-domain speech are enough to adapt a strong multilingual recogniser to Czech medical speech.

  • Uncorrected original transcripts are adequate training labels.
  • Human correction is best spent on evaluation data and a small amount of training data from new speakers.
  • A rank-8 LoRA on Whisper large-v3 reduced WER by about a third, in ten minutes of training on a laptop.
  • The same data halved the error of a production streaming model.
  • Both models run in their existing production runtimes without code changes to the serving software.

Appendix A. Reproducibility

  • Whisper base: openai/whisper-large-v3, revision 06f233fe06e710322aca913c1bc4249a0d71fce1.
  • Nemotron base: nvidia/nemotron-3.5-asr-streaming-0.6b, revision 1c8deaecc64b91f034d73e08dd8b64625eb3395d.
  • Software:
    • Whisper training: PyTorch (CUDA 13.0), Transformers 5.22, PEFT 0.21.
    • Whisper serving: CTranslate2 4.8, faster-whisper 1.2, WhisperX 3.8.
    • Nemotron training: NeMo toolkit 3.1.
    • Nemotron serving: NeMo-Speech.cpp 0.1.0.