Colour = training domain. Continuous (VAE-latent) codecs have no bitrate and are not shown. The line is the Pareto front (best score reachable at or below each bitrate). Bitrate is the one measured on the benchmark.
Methodology
Data. codecbench-multilingual-v1:
1,200 utterances (3.4 h) from the FLEURS test split (CC-BY-4.0), 50 per language, 24 languages covering 10 scripts
(Latin, Cyrillic, Greek, Arabic, Hebrew, Devanagari, Han, Kana, Hangul, Thai) and 4 tonal languages.
Utterances are 3–15 s, one per sentence id, gender-alternating where FLEURS provides both genders, ranked by a fixed
SHA-256 key (no RNG). Recordings whose original audio Whisper-large-v3 transcribes with >50 % error are removed.
FLEURS is 16 kHz, so this track measures speech-band (0–8 kHz) reconstruction for every codec.
Protocol. 16 kHz audio → resampled to the codec's native rate → encode → decode → resampled to 16 kHz. Batch size 1, fp32; a constant delay (up to ±100 ms) is compensated by envelope + waveform cross-correlation before the intrusive metrics.
Metrics
- Intelligibility: Whisper-large-v3 (language forced, greedy). WER, or CER for zh / yue / ja / ko / th, corpus-level per language. ΔWER = WER(recon) − WER(original) in absolute points, which removes per-language ASR difficulty.
- Speaker: cosine similarity of WavLM-Large + ECAPA-TDNN embeddings (the seed-tts-eval SIM model).
- Quality: UTMOS22-strong and DNSMOS P.835 OVRL. Both are English-trained, so the score uses retention relative to the original recording.
- Fidelity: PESQ-WB, STOI/ESTOI, F0 correlation / RMSE (cents), V/UV error, multi-scale mel distance, SI-SDR (SI-SDR is shown but not aggregated, since generative codecs do not preserve phase). F0 is scored only on utterances whose original is voiced; if the reconstruction loses the pitch entirely, its F0 correlation counts as 0.
Score
Fixed anchors, so adding a codec never changes another codec's score. Per language:
intelligibility = (WER_orig + 1) / (WER_recon + 1)(WER in %)speaker = SIMquality = mean(retention(UTMOS), retention(DNSMOS)),retention = (x − 1) / (x_orig − 1)on per-language meansfidelity = mean((PESQ_wb − 1) / 3.5, STOI, F0corr)
Each part is clipped to [0, 1]; the language score is their mean, and Score is the macro-average over the selected languages. Worst lang and EN gap (English score − mean non-English score) expose multilingual robustness. A codec is only ranked when it has results for every selected language. The 95 % CI in Codec details is a bootstrap over utterances. The aggregation is re-implemented independently and cross-checked against every published result (max difference 3e-16).
Bitrate is measured on the benchmark: the bits actually emitted (Σ tokens × log2 codebook size, per scale for multi-scale codecs) divided by the audio duration. It can be slightly above the nominal rate (padding, overlapping chunks, dynamic-rate codecs); the nominal rate is shown in Codec details. Per-utterance side information that some codecs transmit (e.g. global speaker tokens) is listed there and not included. Continuous (VAE) codecs have no bitrate.
Params count every module needed at inference, including frozen SSL encoders (w2v-BERT, HuBERT, WavLM) and vocoders.
Caveats
- Some FLEURS test languages contain a single gender (male only: de, pt, nl, cs, tr, id, vi, uk, fa, he; female only: ar, th).
- UTMOS and DNSMOS are absolute-scale English models: compare the retention-based quality score across languages rather than raw values.
- Codecs marked ! have a known caveat (e.g. an unofficial operating point, or a gap to published numbers we could not resolve) — see Codec details.
Submit / reproduce
pip install -e ".[all]" # codecbench
python -m codecbench.bench.run --codecs "<your_codec_name>" # reconstruct + evaluate + aggregate
Add a codec by subclassing codecbench.BaseCodec (implement _load, _encode,
_decode) and listing its variants in a module-level CATALOG; scripts/verify_codecs.py
sanity-checks the wrapper (frame rate, bitrate, length, aligned STOI/PESQ). Then open a PR adding
results/<codec>.json to the results dataset.