Did fine-tuning actually make your model better?

2026-09-10 · Technical explanations · 6 min read

Your model now recognizes a new dialect, but the Mandarin it used to handle has become worse. The training loss is falling and the checkpoint is called best. Can you ship it? The useful question is whether the application improved under the same trustworthy evaluation rules.

This synthetic example makes the trade-off concrete: A has an overall CER of 12.4%, versus B at 12.9%, but A regresses by 6 percentage points on spontaneous Cantonese. Every number is invented for teaching. No acoustic training was run and these are not SenseVoice measurements.

Lowest loss, lowest aggregate CER and readiness to ship are different conclusions. Passing this example does not validate a real model. Your application must define acceptable old-domain regression in advance.

First, validate the ruler

In a SenseVoice continual-learning report, the developer withdrew earlier new-dialect CER results: references and training labels came from the same unverified process. Even whether the labels were verbatim transcripts or written paraphrases had not been checked. This can invalidate checkpoint selection without causing a training error.

Listen to a sample and record the labeling policy: number formatting, fillers, dialect words and whether paraphrasing is allowed. Separate speakers between training and evaluation, and independently check the origin of the references. Scoring a model against its own generated transcripts is not independent validation.

Distinguish the validation sets used for tuning and checkpoint selection from a separate test set reserved for final acceptance. If references or CER normalization change, re-evaluate the original model and every candidate under that same policy. Do not subtract an old baseline from a newly defined score. Repeatedly choosing configurations on a test set makes it part of selection.

Then, stop one aggregate score from making the decision

Suppose the application serves Mandarin, Cantonese and a new dialect. Each slice below has exactly 1,000 reference characters. CER counts substitutions, deletions and insertions, divided by reference characters. For this synthetic example only, each old-domain slice may regress by at most 1 percentage point. That limit is not an official recommendation.

Total errors divided by total reference characters gives 19%, 12.4% and 12.9% for the original, A and B. A wins on both aggregate and new-dialect CER, yet violates the old-domain constraint. B qualifies for independent testing and service acceptance, not automatic deployment. With unequal real slice sizes, do not simply average percentages. Report every slice and its sample size so a large population cannot hide a smaller language.

Keep units explicit: moving from 12% CER to 18% adds 6 percentage points, a 50% relative increase. Relative regression is undefined when the baseline is zero. Small differences on small samples may also be noise: use more independent speakers and listen to errors rather than treating decimal places as proof of reliability.

Finally, read best literally and test the real output path

FunASR #3677 fixed missing selection metrics being treated as zero; the fix is included in 1.4.15. The maintained SenseVoice recipe explicitly ranks by total validation loss. Its acc_rich is not ASR accuracy or CER. The name model.pt.best refers only to the configured, valid metric; it cannot replace application constraints.

Use a new output directory when changing the ranking metric. Updating software cannot restore deleted checkpoints or automatically repair historical artificial-zero rankings. Re-evaluate candidates that remain. Averaged weights are another candidate to test, not an assumed improvement over every source checkpoint.

Run fixed audio through model output → exported runtime or server → client. After adding a language tag, the model may return valid text while an old downstream allowlist discards it. Preserve intermediate outputs and check the tokenizer, model mapping and parsing contract. Do not disable all validation merely to obtain nonempty text, or assume an existing vocabulary token is an unused slot that can be repurposed freely.

Keep a rollback-ready acceptance record: original and candidate weight hashes, data and annotation-policy versions, normalization rules, per-slice results, the fixed request and final client output. Use only appropriately authorized data. Check old-domain retention and silent empty-output failures in the real path before a gradual rollout. None of this claims that catastrophic forgetting or CTC blank bias has been solved.

Next, open the version-pinned SenseVoice continual fine-tuning guide. Write down your validation slices, separate test set and acceptance limits before starting the next training run.