llm-d-sc-sensitivity-v2.1

sensitivity classifier for llm-d semantic classification. Labels: PUBLIC, INTERNAL, CONFIDENTIAL, REGULATED, NEVER_EGRESS.

Architecture: sequence-classification head (requires a runtime that reads logits), base sentence-transformers/all-MiniLM-L6-v2.

Accuracy

Read the real-traffic row first.

eval set	n	accuracy	95% CI	macro F1
real traffic (WildChat, unanimous 3-model jury)	284	0.8627	0.818 – 0.898	0.4342
enterprise, secrets-handling situations	707	0.8034	0.772 – 0.831	0.7417
enterprise (unconditioned generation, unanimous jury)	744	0.7285	0.695 – 0.759	0.6163
legacy hand-authored held-out	75	0.9200	0.836 – 0.963	0.9186

Hand-authored minus real traffic: +0.057. Hand-authored held-out prompts are written in the same clean register as the anchors; real users send truncated pastes, fragments and roleplay preambles. The real-traffic row is the one that predicts production behaviour.

Per-tier recall — read this before the accuracy

For an egress or capacity gate the errors are not interchangeable: missing a NEVER_EGRESS prompt leaks a live credential, while over-flagging a PUBLIC one costs a cheap round trip. Aggregate accuracy is carried by the largest class and hides this.

tier	recall
PUBLIC	0.74
INTERNAL	0.81
CONFIDENTIAL	0.84
REGULATED	0.71
NEVER_EGRESS	0.90

Run-to-run variance. Selected on entsec-gold, the eval declared primary for this signal BEFORE the sweep, not on best-of-all-evals. The paired seed (se-w-esc0.0-seed11) scores entsec 0.7963 / real-gold 0.8838 against this one's 0.8034 / 0.8627 -- so seed 11 is BETTER on real traffic. Report both. Sensitivity's measured seed spread is 0.14 points on entsec, so the 0.7 point gap here is larger than seed noise and the two seeds genuinely trade evals.

The change from the previous release (llm-d-sc-sensitivity-v2) is the removal of tier-escalated class weighting, nothing else. That flag was introduced as a SAFETY measure and had never been audited against the safety metric that justified it. Swept at 0.0 / 0.5 / 1.0 over two seeds, the result is monotone: each step costs about a point of accuracy (0.7999 / 0.7900 / 0.7808 median) while buying under a point of gate containment, and it also RAISES over-block by 1 to 2.4 points at every gate. Removing it improves accuracy and reduces unnecessary blocking at once.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

42237 rows from sensitivity-v2:6783+sensitivity-real:8361+sensitivity-enterprise:31168+sensitivity-real-contested:618, mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: {'INTERNAL': 14813, 'REGULATED': 6935, 'PUBLIC': 9843, 'NEVER_EGRESS': 4569, 'CONFIDENTIAL': 6077}. Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 3.91 ms, p99 4.58 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.