Python SDK: AutoModel

简体中文 | Installation | Model selection

from funasr import AutoModel runs models inside your Python process. It is not an HTTP client and does not implement the entire OpenAI API. This guide describes the implementation in this checkout, not every historical FunASR release or upstream checkpoint.

Construction and Inference Are Different

AutoModel(**kwargs) resolves a model configuration, looks up its registered implementation, loads weights, and optionally constructs VAD, punctuation, and speaker models. model.generate(input, input_len=None, progress_callback=None, **cfg) runs those already-loaded components. Passing a new model, vad_model, or device to generate() is not a supported way to rebuild or move the pipeline; construct a separate instance.

The wrapper accepts **kwargs, not a universally validated parameter schema. A keyword being accepted does not mean the selected model uses it. Configuration files can override the fallback defaults below. See AutoModel source, the registry, hub resolution, and alias maps.

Construction Parameters

Per-Call Parameters

generate() restores saved construction configuration before merging call options. Repeat request-specific options on each call; do not rely on the previous call's language, batch settings, or hotwords. This configuration reset does not establish thread safety or reset every model attribute: for example, a speaker-mode fallback mutates self.spk_mode. Serialize access to a shared instance unless your own concurrency tests establish otherwise.

Local Files and Batching

Install the SDK using the installation guide. The three examples below are standalone scripts with command-line arguments and require prepared local model snapshots and real audio. Their syntax and wrapper contracts are checked without downloading weights; they are not reported as full inference tests.

For this example, invoke the script with a local ASR directory followed by one or more audio paths, for example python transcribe.py /models/paraformer recording.wav recording2.wav. The checks fail before model construction if those paths do not exist.

import argparse
from pathlib import Path
from funasr import AutoModel

parser = argparse.ArgumentParser()
parser.add_argument("model_dir")
parser.add_argument("audio", nargs="+")
args = parser.parse_args()
model_dir = Path(args.model_dir).expanduser().resolve(strict=True)
if not model_dir.is_dir():
    raise ValueError("model_dir must be a complete local model directory")
audio = [str(Path(path).expanduser().resolve(strict=True)) for path in args.audio]
model = AutoModel(model=str(model_dir), device="cpu", disable_update=True)
results = model.generate(input=audio, batch_size=1)
for result in results:
    print(result.get("key"), result.get("text", ""))

batch_size=1 processes a list serially through direct inference; raise it only for a model that supports batching and after measuring memory use. batch_size_s has a different meaning and only controls the wrapper's VAD path. VAD segmentation is not incremental microphone streaming, and it does not guarantee unlimited recording length or bounded total memory.

Use a one-dimensional mono float32 waveform with a known sample rate. For models using the common audio loader, fs in generate() describes the array's source sample rate (fallback 16000); the model frontend supplies the target rate. Do not pass a stereo matrix or a list of numbers and assume it will be treated as one waveform: a Python list is normally a list of inputs. File decoding depends on the installed audio backends. Top-level bytes are handled by load_bytes: recognized containers are decoded/resampled to 16 kHz, otherwise bytes are interpreted as int16 PCM without sample-rate metadata. Prefer a decoded array with an explicit rate for ambiguous inputs. See audio loading and byte-input tests.

prepare_data_iterator also accepts manifests. A .scp line can be utterance_id /path/to/audio.wav; .jsonl records use {"source": "/path/to/audio.wav", "key": "utterance_id"}. .json is not parsed as an arbitrary JSON array by this iterator. Do not expose unrestricted paths or URLs directly to untrusted callers.

VAD, Timestamps, and Speakers

Run this pipeline example with ASR, VAD, punctuation, speaker directories, then audio, in that order. It is intended for a compatible local Paraformer/SeACo-Paraformer, FSMN-VAD, CT-Transformer punctuation, and CAM++ setup, not every model combination.

import argparse
from pathlib import Path
from funasr import AutoModel

parser = argparse.ArgumentParser()
for name in ("asr_dir", "vad_dir", "punc_dir", "spk_dir", "audio"):
    parser.add_argument(name)
args = parser.parse_args()
paths = {name: Path(value).expanduser().resolve(strict=True)
         for name, value in vars(args).items()}
if not all(paths[name].is_dir() for name in ("asr_dir", "vad_dir", "punc_dir", "spk_dir")):
    raise ValueError("All model arguments must be complete local directories")
model = AutoModel(
    model=str(paths["asr_dir"]),
    vad_model=str(paths["vad_dir"]),
    punc_model=str(paths["punc_dir"]),
    spk_model=str(paths["spk_dir"]),
    spk_mode="punc_segment",
    device="cpu",
    disable_update=True,
)
for result in model.generate(input=str(paths["audio"]), return_spk_res=True):
    print(result.get("text", ""))
    for sentence in result.get("sentence_info", []):
        print(sentence.get("start"), sentence.get("end"),
              sentence.get("spk"), sentence.get("text", ""))

With vad_model configured, the wrapper detects regions, sorts them by duration for ASR batching, restores their order, and offsets timestamps to the original recording. Generic speaker clustering runs in this VAD path; setting only spk_model does not add diarization to direct inference. punc_segment needs usable punctuation and timestamps; missing punctuation or timestamp fields can change the mode to vad_segment. VAD boundaries are speech-region boundaries, not necessarily speaker changes. Labels such as spk=0 are anonymous within a result, not verified identities or stable IDs across recordings.

For sentence_timestamp=True without speakers, the current wrapper can return VAD-aligned sentence records when both punctuation and token timestamps are absent, or when punctuation/timestamp lengths fail alignment. Other combinations can return an empty sentence_info, including timestamps present but no punctuation result. This flag does not create a forced aligner.

Result Fields

generate() returns a Python list of model-result dictionaries, not an OpenAI response object. Do not assume every model produces every field, or index results[0] without checking: some VAD empty-text paths can skip a result. No-speech VAD results commonly contain {"key": ..., "text": "", "timestamp": []}.

Additional fields such as words, ctc_timestamps, or speaker embeddings are model-specific. Do not relabel SDK milliseconds as service seconds without conversion. See timestamp regression tests and the model sources below.

Language, Hotwords, and Alignment Are Model-Specific

These are control conventions, not promises of language coverage, accuracy, or identical checkpoint contents across hubs. Nano disables CTC output when required tensors are missing; see checkpoint validation. MOSS-Transcribe-Diarize is a third-party OpenMOSS model with native joint transcription/diarization and a separate schema/dependency path: use the MOSS guide, not the external VAD/CAM++ example above.

Text-Level Hotword Correction

The wrapper also supports postprocess_hotwords (string/list/dict) and postprocess_hotword_file in generate(). Explicit mappings can use {"wrong": "right"} or a file line wrong=>right. postprocess_hotword_threshold=0.85, postprocess_hotword_fuzzy=True, and return_postprocess_hotword_matches=False are the fallbacks. Fuzzy matching may require pypinyin and rapidfuzz; setting postprocess_hotword_fuzzy=False selects explicit replacement only.

This runs after decoding, updates text and sentence text, and intentionally leaves timestamps aligned to the original recognition. It neither biases the acoustic decoder nor re-aligns replacements. Inspect replacement details before using corrected text for word-level subtitles. Pass these options per call: generate() sends its cfg, not the construction defaults, to this postprocessor. See implementation and tests.

Streaming Cache Lifecycle

The following example is specifically for the local Paraformer streaming checkpoint, not an offline ASR model made streaming by cache={}. Invoke it with the model directory and a nonempty mono 16 kHz audio file. It follows the upstream repository demo and streaming implementation.

import argparse
from pathlib import Path
import soundfile as sf
from funasr import AutoModel

parser = argparse.ArgumentParser()
parser.add_argument("model_dir")
parser.add_argument("audio")
args = parser.parse_args()
model_dir = Path(args.model_dir).expanduser().resolve(strict=True)
if not model_dir.is_dir():
    raise ValueError("model_dir must be a complete local streaming model directory")
speech, sample_rate = sf.read(args.audio, dtype="float32")
if sample_rate != 16000 or speech.ndim != 1 or len(speech) == 0:
    raise ValueError("Expected nonempty mono 16 kHz audio")
model = AutoModel(model=str(model_dir), device="cpu", disable_update=True)
chunk_size = [0, 10, 5]
stride = chunk_size[1] * 960
cache = {}
for start in range(0, len(speech), stride):
    end = min(start + stride, len(speech))
    result = model.generate(
        input=speech[start:end], cache=cache, is_final=end == len(speech),
        chunk_size=chunk_size, encoder_chunk_look_back=4,
        decoder_chunk_look_back=1, batch_size=1,
    )
    print(result)

For this implementation, chunk_size falls back to [0, 10, 5] and is_final to False. The middle value times 960 gives the input stride: 9600 samples at 16 kHz is 600 ms. Both encoder_chunk_look_back and decoder_chunk_look_back default to 0 in source; 4 and 1 above are explicit demo settings, not wrapper defaults.

Create a new cache={} per stream, pass the same dictionary on every ordered chunk, keep chunk/look-back settings fixed, and set is_final=True on the last chunk to flush buffered state. Paraformer streaming currently reinitializes its cache after finalization; discard it anyway when ending, cancelling, or starting a different stream. Do not share caches across recordings or simultaneous users. Keep batch_size=1: this implementation asserts one waveform per call. Returned text is for the decoded chunks in that call, not a guaranteed cumulative transcript; retain outputs in the application. WebSocket session protocols and server-side buffering are separate from this dictionary lifecycle.

Reproducibility and Offline Use

1. Pin the SDK version/commit and dependency versions separately from model artifacts. Record the hub, full model ID, resolved upstream revision, configuration, and weight hashes for ASR and every auxiliary model. An alias or moving master is not a reproducible pin.
2. The generic ModelScope loader forwards model_revision to its snapshot downloader. The generic Hugging Face helper accepts it but currently calls snapshot_download(model), without revision, local_files_only, or check_latest. Do not claim hub="hf", model_revision=... pins that path. Some model adapters have their own loading logic; inspect the selected adapter too.
3. For offline operation, prepare and verify complete local snapshots while online, including tokenizer/frontend files, configuration, weights, nested encoders/LLMs, and optional aligners. Pass existing local directories for all components. Local paths bypass generic hub resolution, but nested model code can still contact services or need missing dependencies; test under an actual network restriction.
4. disable_update=True only skips the package update check. check_latest=False is not an offline guarantee, and these generic helpers do not provide a universal local_files_only switch. URL audio and URL hotword inputs still require network access. Cached hub aliases may still trigger hub requests.
5. Keep trust_remote_code=False unless audited model code requires otherwise. ModelScope's generic loader can import remote_code (default "model") when trusted; both generic hub loaders can install a model's requirements.txt when trusted, while their code-import behavior is not identical. A local snapshot is not automatically safe code.

The FunASR software is MIT-licensed; model weights, datasets, third-party adapters, and dependencies can have different licenses. Review the selected model's own license and usage conditions before redistribution or deployment.

SDK and Service Boundaries

The Hydra inference entry point constructs AutoModel from configuration and calls generate(); it does not turn every CLI/config key into an HTTP form field. Before exposing any service, enforce authentication, TLS, upload limits, timeouts, and access restrictions at the appropriate boundary. A placeholder OpenAI-client API key is not access control.