Dictionary loading failed? Check blank lines and duplicate keys
Does your custom dictionary fail before model inference even starts? Check the file before changing model parameters. In a pipeline that uses a CharTokenizer segmentation dictionary, a blank line, a duplicate key or a hidden character can change loading or mapping behavior. A separate file inspection makes the problem reproducible without starting a GPU workload.
For example, hello h e l l o maps the key hello to the units that follow it. A later hello hh overrides that mapping in the current loader. A line containing only hello creates an empty mapping. None of these file-level observations tells you whether a recognizer can correctly recognize the spoken word.
This article covers FunASR's CharTokenizer and its seg_dict/seg_dict_file path. It is not a universal vocabulary or hotword API for every model. The inspector reads local UTF-8 text using the Python standard library. It does not import FunASR, load weights, access the network or modify files. The examples are constructed text fixtures, not speech-accuracy, training-quality or production-service evaluations.
1. Confirm the dictionary format first
In the pinned CharTokenizer implementation, the loader splits each line on whitespace: the first item is the key and the remaining items form its mapping. Tabs and repeated spaces can separate fields. There is no comment syntax here: a nonempty line beginning with # is still a dictionary entry.
The tokenization path lowercases input words before lookup, while the loader does not automatically lowercase dictionary keys. Uppercase keys therefore deserve inspection. Do not blindly normalize the whole file: mapped units must still match the target model's token vocabulary. The tiny example in this article is not a replacement vocabulary for arbitrary models.
2. Locate suspicious lines without changing the file
Run the following as an independent inspect_dictionary.py script with a dictionary you are authorized to inspect. It reports one-based line numbers for blank lines, keys without mapped units, duplicate keys, BOM characters inside keys and keys containing uppercase characters. Invalid UTF-8 raises a decoding error instead of silently replacing damaged bytes.
import json
import sys
def inspect_seg_dict(path):
report = {
"entries": 0,
"blank_lines": [],
"empty_values": [],
"duplicate_keys": [],
"bom_lines": [],
"uppercase_keys": [],
}
seen = {}
with open(path, "r", encoding="utf-8") as source:
for number, line in enumerate(source, 1):
fields = line.strip().split()
if not fields:
report["blank_lines"].append(number)
continue
key = fields[0]
if len(fields) == 1:
report["empty_values"].append(number)
if "\ufeff" in key:
report["bom_lines"].append(number)
if key != key.lower():
report["uppercase_keys"].append(number)
if key in seen:
report["duplicate_keys"].append({
"key": key,
"previous_line": seen[key],
"line": number,
})
seen[key] = number
report["entries"] = len(seen)
return report
if __name__ == "__main__":
print(json.dumps(inspect_seg_dict(sys.argv[1]),
ensure_ascii=False, indent=2))
python inspect_dictionary.py /path/to/seg_dict.txt
entries counts unique original keys; it is not the model's vocabulary size. previous_line identifies the most recent earlier occurrence of a duplicate, so three occurrences produce two overwrite reports. An empty file reports zero entries. An empty warning list is not proof of a usable dictionary. The inspector neither checks output units against the model vocabulary nor simulates the full unknown-word fallback.
3. Resolve the cause instead of rewriting everything
- Blank lines: check whether the installed code includes the fix. When compatibility with an older environment is necessary, review and remove accidental blank lines in a separate copy, retaining the original and the diff.
- Duplicate keys or empty mappings: examine the source and generation process, and choose the intended mapping explicitly. Do not let last-entry-wins silently become a business rule.
- BOM or encoding errors: confirm the encoding and export tool before performing an explicit conversion on a copy. Ignoring decoding errors can conceal a damaged dictionary.
- Uppercase keys: verify the case policy against real input and the model's token vocabulary. Structural checks do not establish semantic correctness.
4. Separate the merged fix from the installed package
The #3905 blank-line integration preserves the original #3903 implementation and test, and adds the regression to the existing CI entry point. It checks for an empty split result before accessing the first item. Relevant CPU regressions, a native CUDA CTC input chain and both NumPy CI configurations were accepted within their stated scope. This is not a pretrained ASR accuracy evaluation.
At the November 15, 2026 maintenance acceptance, the fix was on main while the accepted root package was still FunASR 1.4.26. The existing wheel must not be described as containing this new fix. The inspector above does not depend on the fix. Before upgrading, check the target release notes, actual installed version and a minimal reproduction instead of assuming that a merged PR describes your local installation.
The accompanying tests execute the code in both language sources and built pages. They cover whitespace and CRLF, tabs, Chinese keys, repeated overrides, empty mappings, BOM, uppercase keys, empty files, invalid UTF-8, ordinary # entries and the command-line entry point, and verify that input bytes remain unchanged. This is not a security audit of arbitrary malicious or oversized files.
After structural checks, validate the token mapping and authorized real samples in an isolated environment. A useful bug report includes a redacted minimal dictionary, versions and the error stack. Dictionaries may contain private names or terminology; do not publish the full file by default.
If you meant recognition hotwords rather than segmentation mappings, continue with hotword request paths and package boundaries.