Deployment contract

SenseVoice TensorRT / Triton

Build SenseVoiceSmall as a native FP16 TensorRT engine and serve batched GPU inference through Triton.

Start deployment Review benchmark method

Maturity: Production verified
FunASR: main@6408aaa9
Runtime: TensorRT 10.0.1 / Triton 24.05
Verified: 2026-08-04

Workload boundary

Decide whether it fits your production constraints

Good fit

SenseVoice batch transcription on a fixed NVIDIA GPU fleet
Triton dynamic batching, health checks, and model repositories
Optimization profiles tuned to production batch and audio distributions

Not a fit

CPU, macOS, Windows, or general edge targets
Copying one engine across different GPU models
Deployments that only have dynamically quantized model_quant.onnx

Models	SenseVoiceSmall
Hardware	nvidia-gpu / kubernetes
Operating systems	Linux
Interfaces	Triton gRPC/HTTP / TensorRT plan / Docker

Run path

From installation to known-audio verification

These commands come from the current verification registry. Pin dependencies, model, and hardware before rollout.

Install

git clone https://github.com/modelscope/FunASR.git && cd FunASR && git checkout 6408aaa96d54f89db467346010eff3c7d1fc485a

pip install -e . "onnx>=1.16" soundfile "tritonclient[grpc]"

python - <<'PY'
from funasr import AutoModel

model = AutoModel(model="iic/SenseVoiceSmall", device="cuda:0")
model.export(
    type="onnx",
    quantize=False,
    device="cuda:0",
    output_dir="./sensevoice_onnx",
    max_seq_len=4096,
)
PY

curl -fL https://huggingface.co/FunAudioLLM/SenseVoiceSmall/resolve/main/chn_jpn_yue_eng_ko_spectok.bpe.model \
  -o runtime/triton_gpu/model_repo_sense_voice_small/scoring/chn_jpn_yue_eng_ko_spectok.bpe.model \
&& echo "aa87f86064c3730d799ddf7af3c04659151102cba548bce325cf06ba4da4e6a8  runtime/triton_gpu/model_repo_sense_voice_small/scoring/chn_jpn_yue_eng_ko_spectok.bpe.model" | sha256sum -c -

Launch

python runtime/triton_gpu/scripts/build_sensevoice_tensorrt.py \
    ./sensevoice_onnx/model.onnx \
    runtime/triton_gpu/model_repo_sense_voice_small/encoder/1/model.plan \
    --precision fp16 \
    --min-batch 1 --opt-batch 8 --max-batch 16 \
    --min-frames 1 --opt-frames 512 --max-frames 4096 \
    --workspace-gb 8

cp runtime/triton_gpu/model_repo_sense_voice_small/encoder/config.pbtxt.tensorrt \
   runtime/triton_gpu/model_repo_sense_voice_small/encoder/config.pbtxt

cd runtime/triton_gpu && tritonserver --model-repository ./model_repo_sense_voice_small \
  --pinned-memory-pool-byte-size=512000000 \
  --cuda-memory-pool-byte-size=0:1024000000

Health check

curl -fsS http://localhost:8000/v2/health/ready

curl -fsS http://localhost:8000/v2/models/sensevoice/ready

Smoke test

Minimum verification before promotion

python - <<'PY'
import numpy as np
import soundfile as sf
import tritonclient.grpc as grpc

audio, sample_rate = sf.read(
    "runtime/triton_gpu/client/test_wavs/mid.wav", dtype="float32"
)
assert sample_rate == 16000
values = {
    "WAV": audio[None, :],
    "WAV_LENS": np.array([[audio.size]], dtype=np.int32),
    "LANGUAGE": np.array([[0]], dtype=np.int32),
    "TEXT_NORM": np.array([[1]], dtype=np.int32),
}
inputs = []
for name, value in values.items():
    tensor = grpc.InferInput(name, value.shape, grpc.np_to_triton_dtype(value.dtype))
    tensor.set_data_from_numpy(value)
    inputs.append(tensor)
result = grpc.InferenceServerClient("localhost:8001").infer("sensevoice", inputs)
transcript = result.as_numpy("TRANSCRIPTS")[0][0].decode()
assert transcript.strip(), "SenseVoice returned an empty transcript"
print(transcript)
PY

Operations and capacity

Move from runnable to operable

Operational checks

Build on the same GPU architecture and TensorRT version used in production
Retune optimization profiles for real batch, concurrency, and audio duration
Retain ONNX, engine, build log, SHA-256, and Triton configuration for rollback

Capacity variables

Audio duration, language, channels, and VAD segment distribution
Concurrency, queue time, warmup, and model-cache state
Exact hardware, driver, runtime, and thread configuration

Troubleshooting

Confirm the export used quantize=False; model_quant.onnx is unsupported
Run the ONNX checker and verify SenseVoice tensor names and dtypes before investigating parser errors
When Triton cannot load the model, keep model.plan and config.pbtxt.tensorrt in the same encoder version directory

Security boundary

Treat production ingress as untrusted

Bind Triton workers to private networks and provide authentication, TLS, rate limits, and audit at the gateway
Limit audio size, duration, channels, sample rate, and concurrency
Load only verified ONNX, TensorRT plans, tokenizers, and model repositories

Known limitation

A TensorRT plan is not portable across arbitrary GPU architectures or TensorRT versions; rebuild and revalidate accuracy, memory, and capacity on the target stack.

Public benchmarks are reproduction starting points, not substitutes for target-workload testing.

Evidence and feedback

Verify this contract against primary material

Report a deployment issue View related benchmark records