doctate/canary/CHUNKING.md

Long-form audio handling in canary-1b-v2

TL;DR

• Canary-1b-v2 truncates audio > ~40 s on the default model.transcribe([path]) call.
• The fix is NeMo ≥ 2.5: long-form inference with overlapping-window auto-chunking ships in transcribe() itself, triggered by batch_size=1 or single-file input.
• The explicit FrameBatchMultiTaskAED path in NeMo 2.4.x is broken for canary-1b-v2 (NeMo issue #14544: CanaryBPETokenizer lacks the vocabulary attribute the streaming-utils class requires).
• Pin: nemo_toolkit[asr]>=2.7.3,<3.0 in requirements.txt.
• Currently used in this codebase: Path 2 (FrameBatchMultiTaskAED), branched on duration > 25 s. Empirically verified end-to-end on experiments/case_c414cf52/. Path 1 was not retested after the NeMo upgrade — see Decision below.
• VRAM cost: ~9.7 GB on RTX 3060 with batch_size=8. Canary cannot share a 12 GB GPU with doctate-whisper (~3.9 GB) — sequential operation required, or reduce FrameBatchMultiTaskAED.batch_size.

Problem

The default model.transcribe([audio_path]) path silently truncates audio longer than roughly 40 s. Observed empirically in our experiments/case_c414cf52/asr_comparison.md benchmark with NeMo 2.4.1: audios at 66 s, 78 s, and 81 s lost their final 25–40 s of content. The model emits no warning, container logs are clean — only the output length reveals the truncation.

The HuggingFace model card claims "automatic chunking enabled at ~40 s with 1 s overlap." That claim is true — but only in NeMo ≥ 2.5, and only on specific call shapes (single-file or batch_size=1). Our pinned nemo_toolkit==2.4.1 did not have the feature merged into a release yet.

Why it happens

Canary-1b-v2's FastConformer encoder has a fixed positional-embedding window. When the input exceeds it, the encoder produces features only for the first window. The decoder emits text up to whatever it generated from those features and then stops. No exception, no warning.

The official NeMo answer (per maintainer @nithinraok in issue #14544):

can you switch to main branch. Its not part of release version yet.

That comment dates from August 2025, when canary-1b-v2 was new and only the main branch had the long-form handling. By NeMo 2.5 it shipped to release; by 2.7.x (our current pin) it is stable.

Path 1 — auto-chunking via transcribe() (recommended, NeMo ≥ 2.5)

The transcribe() method itself supports dynamic chunking with overlapping windows in NeMo ≥ 2.5. It activates automatically under either condition:

• a single audio file passed as a string (not a list), OR
• batch_size=1 set explicitly

Quote from NVIDIA-NeMo's official example docstring:

Canary-1b-v2 supports long‑form inference via the .transcribe() method. It will use dynamic chunking with overlapping windows for better performance. This behavior is enabled automatically for long‑form inference when transcribing a single audio file or when batch_size is set to 1.

Minimal call:

results = model.transcribe(
    [wav_path],
    source_lang="de",
    target_lang="de",
    pnc="yes",
    batch_size=1,        # triggers long-form path
)
text = results[0].text

This path is preferred: it is shorter, has no manifest plumbing, and uses NeMo's tested overlapping-window strategy.

Path 2 — explicit FrameBatchMultiTaskAED (fallback)

For batch inference over many files, manifest-driven workflows, or when fine control over chunk length is needed, NeMo exposes FrameBatchMultiTaskAED + get_buffered_pred_feat_multitaskAED.

Caveat — broken in NeMo 2.4.x for canary-1b-v2: FrameBatchMultiTaskAED.__init__ calls len(asr_model.tokenizer.vocabulary), but CanaryBPETokenizer (the tokenizer of canary-1b-v2) has no vocabulary attribute. NeMo issue #14544 tracks this. In v2.7.3 the same line was extended to fall back on tokenizer.vocab_size first, so this path is fixed from NeMo 2.5 onward. Verified by diffing streaming_utils.py:740–760 between v2.4.1 and v2.7.3.

Key parameters from the official example:

Parameter	Default	Notes
chunk_len_in_secs	40.0	window length
total_buffer	same as frame_len	overlap budget
batch_size	8	chunks processed in parallel
model_stride	8	FastConformer downsampling factor

Pre-conditions on the model config (set once at startup):

import copy
from omegaconf import OmegaConf

_model_cfg = copy.deepcopy(model._cfg)
OmegaConf.set_struct(_model_cfg.preprocessor, False)
_model_cfg.preprocessor.dither = 0.0
_model_cfg.preprocessor.pad_to = 0
OmegaConf.set_struct(_model_cfg.preprocessor, True)
assert _model_cfg.preprocessor.normalize == "per_feature"

Setup pattern (condensed from the official script):

from nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED
from nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED

frame_asr = FrameBatchMultiTaskAED(
    asr_model=model,
    frame_len=40.0,
    total_buffer=40.0,
    batch_size=8,
)
feature_stride = _model_cfg.preprocessor["window_stride"]
model_stride_in_secs = feature_stride * 8  # 0.01 * 8 = 0.08 for FastConformer

with torch.amp.autocast(model.device.type, enabled=True, dtype=torch.bfloat16):
    with torch.no_grad():
        hyps = get_buffered_pred_feat_multitaskAED(
            frame_asr,
            _model_cfg.preprocessor,
            model_stride_in_secs,
            model.device,
            manifest=manifest_path,   # JSONL with audio_filepath, source_lang, target_lang, pnc, taskname
            filepaths=None,
            timestamps=False,
        )

For German ASR, the manifest is required — filepaths=-only mode defaults to English. One JSON object per line:

{"audio_filepath": "/path/audio.wav", "duration": 80.9, "taskname": "asr",
 "source_lang": "de", "target_lang": "de", "pnc": "yes"}

Caveat — timestamps require shorter chunks

When timestamps=True, the official script forces chunk_len_in_secs=10.0:

When timestamps is True, it's recommended to use chunk_len_in_secs=10.0 for optimal results.

Larger windows produce less reliable per-segment timestamps. If we expose timestamps in /inference, chunk length must drop to 10 s for that mode.

Decision — what this codebase actually uses

Path 2 (FrameBatchMultiTaskAED) is implemented in main.py (_transcribe_buffered), branched on duration > CHUNKED_THRESHOLD_SECS (default 25 s). It was implemented before we upgraded NeMo, when Path 1 was unavailable in 2.4.1. After upgrading to 2.7.3 we kept Path 2 because it is the path we have empirical evidence for.

Path 1 was not retested after the NeMo upgrade. It would likely work in 2.7.3 — the official docs and example script promise so — and would let us delete _transcribe_buffered and the manifest plumbing. Reasons to revisit Path 1 later:

• Code simplification (~50 lines of main.py could go away).
• Lower VRAM (Path 2's batch_size=8 reserves buffers for 8 parallel chunks; Path 1's overlapping-window strategy may use less).
• One fewer setup-per-call (FrameBatchMultiTaskAED is currently instantiated each request).

Reasons to keep Path 2:

• It works, with measurements and full transcripts.
• Explicit control over chunk_len_in_secs (currently 40 s).
• The fallback for short audio (model.transcribe([wav], ...) for duration ≤ 25 s) is still useful for low latency on short clips.

The migration risk is non-zero (we'd need to re-run the case_c414cf52 benchmark to confirm output quality), so it's a "later" task rather than a "now" task.

What we ruled out

• Hard fixed-cut chunking at second boundaries (the approach in EvilFreelancer/docker-canary-serve, using wave.readframes() with no overlap): cuts mid-word at chunk boundaries. NeMo's overlapping-window paths are strictly better and exist now in release versions.
• Streaming variant (speech_to_text_aed_streaming_infer.py, ~20 KB): cache-aware streaming for online use cases. Overkill for our offline transcription workflow.
• Monkey-patching tokenizer.vocabulary to work around the NeMo 2.4.x bug. Possible but a code smell — the upstream fix in 2.5+ is the right solution.

Empirical history (this codebase)

Date	Attempt	Result
2026-04-29	NeMo 2.4.1 + model.transcribe([wav], ...)	Truncation > 50 s
2026-04-29	+ batch_size=1	No change — feature not in 2.4.1
2026-04-30	+ explicit FrameBatchMultiTaskAED (Path 2) on 2.4.1	AttributeError: CanaryBPETokenizer.vocabulary (NeMo bug)
2026-04-30	NeMo bumped to ≥ 2.7.3, Path 2 active	✓ all 6 audios fully transcribed; ~2 s inference per audio (~30× realtime); 9.7 GB VRAM

References

• NeMo issue tracking the 2.4.x tokenizer bug: #14544
• Official NVIDIA NeMo Canary chunked/streaming docs: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/streaming_decoding/canary_chunked_and_streaming_decoding.html
• Official chunked AED inference example: speech_to_text_aed_chunked_infer.py
• Streaming variant: speech_to_text_aed_streaming_infer.py
• HuggingFace model card: https://huggingface.co/nvidia/canary-1b-v2
• Truncation evidence in this repo: experiments/case_c414cf52/asr_comparison.md
• Reference project we initially looked at (we did not adopt their cut strategy): EvilFreelancer/docker-canary-serve