Ko-Voice-Command-Reader v0.1

🇰🇷 한국어 카드: MODEL_CARD.ko.md

A small reader (3.47 M parameters, 13.9 MB) on top of the frozen EmbeddingGemma 2 audio tower. It turns a Korean spoken command into a structured prediction: intent, action, facet, the resource it refers to (linked against your registry), and a policy decision (CALL_TOOL / CONFIRM / ASK_SLOT / DELEGATE / REFUSE / NO_COMMAND) with the tool it would map to.

This model predicts. It does not execute anything and has no tool-calling runtime. Route its output through your own permission and approval layer before any tool runs.


Encoder	`google/embeddinggemma-2` audio tower, layer 12, frozen, 304.8 M params — not included, downloaded from the original repo (Apache-2.0)
Reader (this repo)	4-layer Transformer (d=256) + intent/action/facet/type heads + jamo CTC head — 3,466,151 params, `model.safetensors`
Entity linking	jamo CTC posteriors aligned against the names of the visible registry entries → calibrated → `LINK / AMBIG / NONE`
Input	16 kHz mono speech (≤ 30 s) + a registry JSON
Output	JSON: intent, action, facet, type word, entity (status, name, ids, top candidates), decision, tool, target_id, latency

Usage

pip install -r requirements.txt
python inference.py --audio examples/01_novel_clear.wav --registry examples/01_novel_clear.registry.json

Registry format (entries with access: "none" are invisible: they are removed before scoring and are never linked or echoed):

[{"id": "nd-1", "name": "알파 노드", "type": "node", "access": "write", "aliases": []},
 {"id": "ep-1", "name": "번역 서버", "type": "endpoint", "access": "read", "aliases": ["번역기"]}]

The encoder runs in bf16 by default because the reader was trained on bf16-encoded features (--eg2-dtype float32 is available; outputs agreed on 99.25 % of clips). examples/ holds synthetic (Supertonic-3) clips with their registry and the expected output (examples.json).

Why it links names it has never heard

The previous version learned a name scorer and memorised the 39 training names; entity linking on unseen names was 0.005. This version drops the name scorer. A jamo CTC head is trained on the whole utterance, and each registry name is scored by local CTC alignment. A new resource only has to be added to the registry; no retraining.

Results

All names in the evaluation sets are unseen in training. test_new additionally uses a TTS engine never seen in training (piper-kss). Linking thresholds were calibrated on dev only, under a safety constraint (wrong-entity link ≤ 1 %, link on an unregistered or unmentioned name ≤ 2 % on dev). Single training seed. Synthetic speech only.

test_new — 7,898 utterances (2,137 with a linkable target)

Metric	previous version	this model
Entity linking (correct target)	0.005	0.698
Decision accuracy (system)	0.727	0.874
Tool + decision accuracy	0.686	0.869
Clear normal commands handled (N1)	0.009	0.692
Unnecessary clarification on clear commands (N2)	0.926	0.283
Action head accuracy	–	0.961
Jamo CER (greedy CTC)	–	0.073
Invisible resource exposed in output	0	0

Behaviour on commands that must not run as-is (test_new)

Metric	value
Execution proposed on rows whose gold is non-executable	0.020 (124/6,123)
Execution proposed when the action is ambiguous	0.010 (3/308)
Ambiguous action → asks for clarification	0.786 (242/308)
`stop` commands proposed for execution after the reference gate	0 / 131

dev (the calibration set, optimistic): entity linking 0.903, decision 0.876, N1 0.805.

End-to-end check and speed (CPU)

Raw WAV → EG2 → reader → linking with the released files, on a diagnostic probe of 400 clips built to be hard (confusable names, homophones, Korean–English mixed names, unregistered and invisible names, held-out voices, piper-kss).

Released r2 files, 400 clips (median 2.37 s audio), 8 CPU threads	bf16 encoder (default)	fp32 encoder
Decision accuracy	0.760	0.760
Entity linking (211 linkable clips)	0.597	0.602
EG2 encoder latency p50 / p90	143 ms / 184 ms	182 ms / 198 ms
Reader + linking latency p50 / p90	5.8 ms / 7.3 ms	6.0 ms / 7.1 ms
Identical outputs bf16 vs fp32	99.25 %

Model compute ≈ 149 ms per 2.4 s utterance (p50, warm process).

Model compute only. Audio loading, resampling, model loading and I/O are not included. The first call in a fresh process took about 5 s for the encoder on the same CPU (one-time initialisation; measured by running inference.py once per clip in a clean environment).

Linking with an ASR transcript (optional)

On the same items, Qwen3-ASR-1.7B transcripts matched against the registry reach 0.809 on test_new; this model's CTC linking alone reaches 0.750, and combining both reaches 0.858 (4,728 ASR-covered items, oracle heads, measured with the earlier r1 checkpoint of this architecture). The CTC path needs no ASR model; CPU Qwen3-ASR took 12.6 s per clip in the same environment.

Limitations

Synthetic speech only. Trained on Supertonic-3 voices; evaluated on Supertonic-3 and piper-kss. Real speakers, noise, far-field audio and dialects are not measured; expect lower numbers on real speech.
Guessing ambiguous actions. When a target name is linked, the action head sometimes commits to an action the user did not state. On dev only 0.197 of ambiguous-action commands are routed to clarification and 0.260 receive an execution proposal. The next version adds an action-ambiguity head. Do not auto-execute on CONFIRM.
Linking drops on an unseen TTS engine (dev 0.903 → test_new 0.698) and on hard confusable or homophone names.
Fixed 24-tool ontology (release_spec.json); new tools require retraining.
One seed; no confidence intervals across training runs.
Decisions are predictions under one policy (policy_v2), not authorisation.

Data and licenses

Utterance texts were generated in-house and are not released. No AI Hub data was used.
Training/dev audio: Supertonic-3 (BigScience Open RAIL-M). This model is released under Open RAIL-M and carries the same use-based restrictions (licenses/OPEN_RAIL_M.txt, Attachment A). See licenses/NOTICE.md.
Held-out evaluation audio (piper ko_KR-kss, KSS dataset CC BY-NC-SA 4.0) was used for evaluation only and is not released.
Encoder: google/embeddinggemma-2 (Apache-2.0), loaded from its original repository.

Use restrictions (Open RAIL-M, paragraph 5)

You may not use this model or its derivatives for the restricted uses in Attachment A of the license, including unlawful use, harming minors, impersonation, and fully automated decisions that adversely affect a person's legal rights. Redistribution must carry these restrictions.