doctate/canary/CHUNKING.md

# Long-form audio handling in canary-1b-v2

## TL;DR

- Canary-1b-v2 truncates audio > ~40 s on the default `model.transcribe([path])` call.
- The fix is **NeMo ≥ 2.5**: long-form inference with overlapping-window
  auto-chunking ships in `transcribe()` itself, triggered by
  `batch_size=1` or single-file input.
- The explicit `FrameBatchMultiTaskAED` path in NeMo `2.4.x` is broken for
  `canary-1b-v2` (NeMo issue #14544: `CanaryBPETokenizer` lacks the
  `vocabulary` attribute the streaming-utils class requires).
- Pin: `nemo_toolkit[asr]>=2.7.3,<3.0` in `requirements.txt`.
- **Currently used in this codebase**: Path 2 (`FrameBatchMultiTaskAED`),
  branched on `duration > 25 s`. Empirically verified end-to-end on
  `experiments/case_c414cf52/`. Path 1 was not retested after the NeMo
  upgrade — see Decision below.
- **VRAM cost**: ~9.7 GB on RTX 3060 with `batch_size=8`. Canary cannot
  share a 12 GB GPU with `doctate-whisper` (~3.9 GB) — sequential operation
  required, or reduce `FrameBatchMultiTaskAED.batch_size`.

## Problem

The default `model.transcribe([audio_path])` path silently truncates audio
longer than roughly 40 s. Observed empirically in our
`experiments/case_c414cf52/asr_comparison.md` benchmark with NeMo 2.4.1:
audios at 66 s, 78 s, and 81 s lost their final 25–40 s of content. The
model emits no warning, container logs are clean — only the output length
reveals the truncation.

The HuggingFace model card claims "automatic chunking enabled at ~40 s with
1 s overlap." That claim is true — but **only in NeMo ≥ 2.5**, and only on
specific call shapes (single-file or `batch_size=1`). Our pinned
`nemo_toolkit==2.4.1` did not have the feature merged into a release yet.

## Why it happens

Canary-1b-v2's FastConformer encoder has a fixed positional-embedding
window. When the input exceeds it, the encoder produces features only for
the first window. The decoder emits text up to whatever it generated from
those features and then stops. No exception, no warning.

The official NeMo answer (per maintainer @nithinraok in
[issue #14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544)):

> can you switch to main branch. Its not part of release version yet.

That comment dates from August 2025, when canary-1b-v2 was new and only the
`main` branch had the long-form handling. By NeMo 2.5 it shipped to
release; by 2.7.x (our current pin) it is stable.

## Path 1 — auto-chunking via `transcribe()` (recommended, NeMo ≥ 2.5)

The `transcribe()` method itself supports dynamic chunking with overlapping
windows in NeMo ≥ 2.5. It activates automatically under either condition:

- a single audio file passed as a string (not a list), OR
- `batch_size=1` set explicitly

Quote from NVIDIA-NeMo's
[official example docstring](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py):

> Canary-1b-v2 supports long‑form inference via the `.transcribe()` method.
> It will use dynamic chunking with overlapping windows for better
> performance. This behavior is enabled automatically for long‑form
> inference when transcribing a single audio file or when batch_size is
> set to 1.

Minimal call:

```python
results = model.transcribe(
    [wav_path],
    source_lang="de",
    target_lang="de",
    pnc="yes",
    batch_size=1,        # triggers long-form path
)
text = results[0].text
```

This path is preferred: it is shorter, has no manifest plumbing, and uses
NeMo's tested overlapping-window strategy.

## Path 2 — explicit `FrameBatchMultiTaskAED` (fallback)

For batch inference over many files, manifest-driven workflows, or when
fine control over chunk length is needed, NeMo exposes
`FrameBatchMultiTaskAED` + `get_buffered_pred_feat_multitaskAED`.

**Caveat — broken in NeMo 2.4.x for canary-1b-v2**:
`FrameBatchMultiTaskAED.__init__` calls
`len(asr_model.tokenizer.vocabulary)`, but `CanaryBPETokenizer` (the
tokenizer of canary-1b-v2) has no `vocabulary` attribute. NeMo issue
[#14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544) tracks this. In
`v2.7.3` the same line was extended to fall back on `tokenizer.vocab_size`
first, so this path is **fixed from NeMo 2.5 onward**. Verified by diffing
`streaming_utils.py:740–760` between v2.4.1 and v2.7.3.

Key parameters from the official example:

| Parameter | Default | Notes |
|---|---|---|
| `chunk_len_in_secs` | `40.0` | window length |
| `total_buffer` | same as `frame_len` | overlap budget |
| `batch_size` | `8` | chunks processed in parallel |
| `model_stride` | `8` | FastConformer downsampling factor |

Pre-conditions on the model config (set once at startup):

```python
import copy
from omegaconf import OmegaConf

_model_cfg = copy.deepcopy(model._cfg)
OmegaConf.set_struct(_model_cfg.preprocessor, False)
_model_cfg.preprocessor.dither = 0.0
_model_cfg.preprocessor.pad_to = 0
OmegaConf.set_struct(_model_cfg.preprocessor, True)
assert _model_cfg.preprocessor.normalize == "per_feature"
```

Setup pattern (condensed from the official script):

```python
from nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED
from nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED

frame_asr = FrameBatchMultiTaskAED(
    asr_model=model,
    frame_len=40.0,
    total_buffer=40.0,
    batch_size=8,
)
feature_stride = _model_cfg.preprocessor["window_stride"]
model_stride_in_secs = feature_stride * 8  # 0.01 * 8 = 0.08 for FastConformer

with torch.amp.autocast(model.device.type, enabled=True, dtype=torch.bfloat16):
    with torch.no_grad():
        hyps = get_buffered_pred_feat_multitaskAED(
            frame_asr,
            _model_cfg.preprocessor,
            model_stride_in_secs,
            model.device,
            manifest=manifest_path,   # JSONL with audio_filepath, source_lang, target_lang, pnc, taskname
            filepaths=None,
            timestamps=False,
        )
```

For German ASR, the manifest is required — `filepaths=`-only mode defaults
to English. One JSON object per line:

```json
{"audio_filepath": "/path/audio.wav", "duration": 80.9, "taskname": "asr",
 "source_lang": "de", "target_lang": "de", "pnc": "yes"}
```

## Caveat — timestamps require shorter chunks

When `timestamps=True`, the official script forces `chunk_len_in_secs=10.0`:

> When `timestamps` is True, it's recommended to use `chunk_len_in_secs=10.0`
> for optimal results.

Larger windows produce less reliable per-segment timestamps. If we expose
timestamps in `/inference`, chunk length must drop to 10 s for that mode.

## Decision — what this codebase actually uses

**Path 2 (`FrameBatchMultiTaskAED`)** is implemented in `main.py`
(`_transcribe_buffered`), branched on `duration > CHUNKED_THRESHOLD_SECS`
(default 25 s). It was implemented *before* we upgraded NeMo, when Path 1
was unavailable in 2.4.1. After upgrading to 2.7.3 we kept Path 2 because
it is the path we have empirical evidence for.

**Path 1 was not retested** after the NeMo upgrade. It would likely work
in 2.7.3 — the official docs and example script promise so — and would
let us delete `_transcribe_buffered` and the manifest plumbing. Reasons
to revisit Path 1 later:

- Code simplification (~50 lines of `main.py` could go away).
- Lower VRAM (Path 2's `batch_size=8` reserves buffers for 8 parallel
  chunks; Path 1's overlapping-window strategy may use less).
- One fewer setup-per-call (FrameBatchMultiTaskAED is currently
  instantiated each request).

Reasons to keep Path 2:

- It works, with measurements and full transcripts.
- Explicit control over `chunk_len_in_secs` (currently 40 s).
- The fallback for short audio (`model.transcribe([wav], ...)` for
  `duration ≤ 25 s`) is still useful for low latency on short clips.

The migration risk is non-zero (we'd need to re-run the case_c414cf52
benchmark to confirm output quality), so it's a "later" task rather than
a "now" task.

## What we ruled out

- **Hard fixed-cut chunking** at second boundaries (the approach in
  `EvilFreelancer/docker-canary-serve`, using `wave.readframes()` with no
  overlap): cuts mid-word at chunk boundaries. NeMo's overlapping-window
  paths are strictly better and exist now in release versions.
- **Streaming variant** (`speech_to_text_aed_streaming_infer.py`, ~20 KB):
  cache-aware streaming for online use cases. Overkill for our offline
  transcription workflow.
- **Monkey-patching `tokenizer.vocabulary`** to work around the NeMo 2.4.x
  bug. Possible but a code smell — the upstream fix in 2.5+ is the right
  solution.

## Empirical history (this codebase)

| Date | Attempt | Result |
|---|---|---|
| 2026-04-29 | NeMo 2.4.1 + `model.transcribe([wav], ...)` | Truncation > 50 s |
| 2026-04-29 | + `batch_size=1` | No change — feature not in 2.4.1 |
| 2026-04-30 | + explicit `FrameBatchMultiTaskAED` (Path 2) on 2.4.1 | `AttributeError: CanaryBPETokenizer.vocabulary` (NeMo bug) |
| 2026-04-30 | NeMo bumped to ≥ 2.7.3, Path 2 active | ✓ all 6 audios fully transcribed; ~2 s inference per audio (~30× realtime); 9.7 GB VRAM |

## References

- NeMo issue tracking the 2.4.x tokenizer bug:
  [#14544](https://github.com/NVIDIA-NeMo/NeMo/issues/14544)
- Official NVIDIA NeMo Canary chunked/streaming docs:
  <https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/streaming_decoding/canary_chunked_and_streaming_decoding.html>
- Official chunked AED inference example:
  [`speech_to_text_aed_chunked_infer.py`](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_chunked_infer.py)
- Streaming variant:
  [`speech_to_text_aed_streaming_infer.py`](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_chunked_inference/aed/speech_to_text_aed_streaming_infer.py)
- HuggingFace model card: <https://huggingface.co/nvidia/canary-1b-v2>
- Truncation evidence in this repo: `experiments/case_c414cf52/asr_comparison.md`
- Reference project we initially looked at (we did not adopt their cut
  strategy): [`EvilFreelancer/docker-canary-serve`](https://github.com/EvilFreelancer/docker-canary-serve)