Configuration Parsing Warning:Config file tokenizer_config.json cannot be fetched (too big)
VLA Tokenizer — Qwen3 v3 (separate vocab for <listen>)
Successor to tokenizer-vla-qwen3-v2. Purely additive — every token ID from v2 is unchanged; this only registers 28,672 new tokens.
Vocab size: 303,233 (274,561 from v2 + 28,672 new <listen>-shifted
tokens).
What changed and why
An ablation study (2 vs 3 vs 5, see project progress notes) compared 3
variants of <listen> token format on real DailyTalk/Emolia audio:
| Variant | Listen format | ASR accuracy | TTS decode reliability |
|---|---|---|---|
| Baseline | truncated 3 tok/frame, shared vocab with <speak> |
0.153 | 62.5% |
| Full-listen, shared vocab | full 7 tok/frame, same ids as <speak> |
0.750 | 12.5% |
| Full-listen, separate vocab (this tokenizer) | full 7 tok/frame, shifted +1,000,000 | 0.476 | 62.5% |
Sharing token ids between <listen> and <speak> for identical SNAC codes
gave the biggest comprehension jump but badly damaged speech-generation
reliability (likely cross-talk — the model conflates "I heard code X" with
"I should say code X"). Giving <listen> a genuinely separate embedding
space (this tokenizer) recovers most of the TTS reliability while keeping
a real comprehension gain over baseline.
New tokens: <snac_{N+1000000}> for N in the same 7 SNAC bands already
used by <speak> (128266-132361, 132362-136457, 136458-140553,
140554-144649, 144650-148745, 148746-152841, 152842-156937) — i.e.
<snac_1128266> through <snac_1156937>, 28,672 ids total. Every existing
<snac_N> id (unshifted, used by <speak>) is unchanged.
Compatibility
All 274,561 token IDs from tokenizer-vla-qwen3-v2 are unchanged (verified
— strict superset, additions only, no renumbering). Text tokenized with the
v2 tokenizer remains valid under this one. Verified via
tokenizers.Tokenizer.encode() spot-check: existing wrapper tags
(<seed2_N>, <cosmos_N>, <agent>, <listen>, <speak>, unshifted
<snac_N>) and the new shifted-listen band edges (<snac_1128266>,
<snac_1156937>) all encode as single atomic token ids.
Base
Qwen/Qwen3-1.7B-Base tokenizer, extended the same way as prior versions (see tokenizer-vla-qwen3's card for the full original token-family breakdown: seed2/cosmos/avclm/agent/ snac/caption/speech).
Related
| Resource | Link |
|---|---|
| Previous tokenizer | tokenizer-vla-qwen3-v2 |