Deployment contract
llama.cpp / GGUF standalone
Use cross-platform release packages or source builds to run FunASR GGUF models on CPUs, desktop GPUs, and edge devices.
Workload boundary
Decide whether it fits your production constraints
Good fit
- No Python ML environment
- Desktop applications and offline edge deployment
- Linux, macOS, and Windows release packages
Not a fit
- vLLM-style large GPU batch scheduling
- A universal prebuilt CUDA package for an unverified GPU architecture
- Workflows that require the full Python model ecosystem
| Models | SenseVoiceSmall-GGUF / Paraformer-GGUF / Fun-ASR-Nano-GGUF / FSMN-VAD-GGUF |
|---|---|
| Hardware | cpu / desktop-edge-gpu |
| Operating systems | Linux / macOS / Windows |
| Interfaces | CLI / local HTTP server |
Run path
From installation to known-audio verification
These commands come from the current verification registry. Pin dependencies, model, and hardware before rollout.
Install
cmake -B build -DCMAKE_BUILD_TYPE=Releasecmake --build build -jLaunch
./build/bin/llama-funasr-sensevoice -m funasr-gguf/sensevoice-small-q8.gguf --vad funasr-gguf/fsmn-vad.gguf -a sample.wavHealth check
./build/bin/llama-funasr-sensevoice --helpSmoke test
Minimum verification before promotion
./runtime/llama.cpp/tests/run_regression.shOperations and capacity
Move from runnable to operable
Operational checks
- Verify release assets with SHA-256
- Keep models and binaries on the same release manifest
- Retain the previous binary and model directory for rollback
Capacity variables
- Audio duration, language, channels, and VAD segment distribution
- Concurrency, queue time, warmup, and model-cache state
- Exact hardware, driver, runtime, and thread configuration
Troubleshooting
- Validate the q8 CPU path first
- Match CUDA or Vulkan to the local driver
- Use repository regression scripts to separate model and build failures
Security boundary
Treat production ingress as untrusted
- Read local audio and models by default
- Bind the HTTP server to a private address and limit uploads
- Do not load GGUF files from untrusted sources
Known limitation
Prebuilt GPU packages cover only the documented backend and architecture; other devices require a source build and hardware validation.
Public benchmarks are reproduction starting points, not substitutes for target-workload testing.
Evidence and feedback
