Deployment contract
ONNX / C++ CPU runtime
Deploy offline, online, and two-pass ASR on general CPUs with ONNX Runtime and C++ paths.
Workload boundary
Decide whether it fits your production constraints
Good fit
- High-concurrency CPU services
- Embedded or local applications
- An existing Paraformer online path already meets quality goals
Not a fit
- LLM decoding and open-ended instructions
- Moving proven CPU traffic to GPU without tests
- Workflows that depend on Python-only business plugins
| Models | Paraformer / Paraformer-online / SenseVoiceSmall / FSMN-VAD / CT-Transformer punctuation |
|---|---|
| Hardware | cpu |
| Operating systems | Linux / macOS / Windows |
| Interfaces | C++ / ONNX Runtime / Python ONNX / HTTP / WebSocket |
Run path
From installation to known-audio verification
These commands come from the current verification registry. Pin dependencies, model, and hardware before rollout.
Install
cd runtime/onnxruntime && cmake -B build -DCMAKE_BUILD_TYPE=Releasecmake --build build -jLaunch
python runtime/python/onnxruntime/demo_sencevoice_small.pyHealth check
test -x runtime/onnxruntime/build/bin/funasr-onnx-offlineSmoke test
Minimum verification before promotion
python runtime/python/onnxruntime/demo_paraformer_offline.pyOperations and capacity
Move from runnable to operable
Operational checks
- Pin ONNX Runtime and model versions
- Tune model, decoder, and I/O threads separately
- Monitor realtime ratio, CPU saturation, and queue depth
Capacity variables
- Audio duration, language, channels, and VAD segment distribution
- Concurrency, queue time, warmup, and model-cache state
- Exact hardware, driver, runtime, and thread configuration
Troubleshooting
- Start with a single-file offline example
- More threads do not guarantee linear throughput
- Evaluate deterministic text post-processing before changing models for fixed terms
Security boundary
Treat production ingress as untrusted
- Bound local file and network inputs
- Limit request size and concurrency in service mode
- Keep model directories read-only and verify their source
Known limitation
Model conversion, operator support, and thread settings vary by platform; measure capacity on the target CPU with real audio.
Public benchmarks are reproduction starting points, not substitutes for target-workload testing.
Evidence and feedback