Instructions to use StandardThinking/StandardOne-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use StandardThinking/StandardOne-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="StandardThinking/StandardOne-3B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("StandardThinking/StandardOne-3B")
model = AutoModelForMultimodalLM.from_pretrained("StandardThinking/StandardOne-3B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use StandardThinking/StandardOne-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "StandardThinking/StandardOne-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "StandardThinking/StandardOne-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/StandardThinking/StandardOne-3B

SGLang

How to use StandardThinking/StandardOne-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "StandardThinking/StandardOne-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "StandardThinking/StandardOne-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "StandardThinking/StandardOne-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "StandardThinking/StandardOne-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use StandardThinking/StandardOne-3B with Docker Model Runner:
```
docker model run hf.co/StandardThinking/StandardOne-3B
```

MyeongHoJeong commited on Sep 26

Commit

815f789

verified ·

1 Parent(s): 3c08bbf

Update model card

Browse files

Files changed (20) hide show

QUICKSTART.md +2 -1
README.md +42 -27
SHA256SUMS +19 -18
docs/BENCHMARKS.md +2 -2
docs/assets/00-benchmark-card.png +2 -2
docs/assets/00-benchmark-card.svg +143 -143
docs/assets/05-gain-over-base.png +2 -2
docs/assets/05-gain-over-base.svg +52 -52
docs/assets/06-vs-jev.png +2 -2
docs/assets/06-vs-jev.svg +67 -67
docs/public-classification-suites.md +2 -0
evidence/calibration-leaderboard-ablate-nosys.json +533 -209
evidence/calibration-leaderboard-merged-native.json +176 -176
evidence/calibration-per-type-served.json +537 -0
evidence/prompt-wording-choice.json +81 -0
evidence/served-nosys-easy-report.json +72 -72
evidence/served-nosys-hard-report.json +222 -222
evidence/served-nosys-original-report.json +116 -116
release-manifest.json +16 -2
server/README.md +7 -3

QUICKSTART.md CHANGED Viewed

@@ -105,7 +105,8 @@ jev-adapter --engine-url http://127.0.0.1:30000 --model standard-one-3b --alias
   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-3B-Instruct-2512-BF16 \
   --tokenizer-revision b6d637bef2393152b3da2b2fde72eecdee30557e \
-  --prompt-wording native --native-system-prompt none --default-temperature 1.55
 ```
 4. Check health, then send a request or run JevBench exactly as in steps 6-9 above, with

   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-3B-Instruct-2512-BF16 \
   --tokenizer-revision b6d637bef2393152b3da2b2fde72eecdee30557e \
+  --prompt-wording native --native-system-prompt none --default-temperature 0.95 \
+  --temperature-by-type choice=0.95,noul=1.05,score=0.85
 ```
 4. Check health, then send a request or run JevBench exactly as in steps 6-9 above, with

README.md CHANGED Viewed

@@ -29,7 +29,6 @@ tags:
 > **Updated weights (v2, 2026-09-26).** If you downloaded this model before, download it again or pin
 > `revision="v2"`. Earlier versions stay available under the tags `v1` and `v1.1`.
-> The benchmark figures and the serving temperature on this card are still those of v1.1 and are being updated for v2.
 **Version:** v2
@@ -51,14 +50,14 @@ for the measurement conditions and limitations.
 ![Standard One benchmark card: JevBench public tiers, held-out suites, stated-distribution probability, hard-tier calibration, latency and throughput for Standard One 8B, Standard One 3B and Jev 1.13.](docs/assets/00-benchmark-card.png)
 *The figure combines results from different measurement paths. See [Benchmarks](#benchmarks) for
-served versus offline conditions; measured 24–25 September 2026.*
 ## At a glance
 - Send a state and a bounded rubric to receive probabilities for the supplied labels:
   `choice` selects among labeled options, `noul` is yes/no, and `score` uses an ordinal scale.
   The endpoint scores the labels in one forward pass without decoding answer text.
-- In the same-run **offline** comparison with its untuned base, 3B improves on five of six suites and declines by 0.90 percentage points on public hard. These are not served-endpoint results.
 - Probabilities are temperature-scaled and calibration-checked (hard-tier ECE, distribution
   total-variation) — see Benchmarks below.
 - The shared training mixture covers English, Japanese, Chinese, Spanish, French, German,
@@ -104,9 +103,24 @@ Adapter:
   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-3B-Instruct-2512-BF16 \
   --tokenizer-revision b6d637bef2393152b3da2b2fde72eecdee30557e \
-  --prompt-wording native --native-system-prompt none --default-temperature 1.55
 ```
 Try it:
 ```bash
@@ -132,7 +146,8 @@ Response shape (example values, default temperature applied):
   "usage": {"input_tokens": 96, "output_tokens": 0},
   "metadata": {
     "confidence_method": "1 - normalized_entropy",
-    "temperature": 1.55,
     "evaluations": 1,
     "adapter_elapsed_ms": 22.4
   }
@@ -145,7 +160,7 @@ More (client command, 8B variant, request format): see
 ## Benchmarks
 **Served endpoint results (the release configuration).** Merged BF16 weights through SGLang 0.5.20
-and `jev-adapter`, native wording, no system prompt, one option order, T=1.55. Jev 1.13 was
 measured on the same items through its hosted endpoint; its probabilities are raw, with no
 temperature applied. These are our measurements, not official sealed-set JevBench scores.
@@ -153,11 +168,11 @@ temperature applied. These are our measurements, not official sealed-set JevBenc
 |---|---|---|
 | JevBench public easy (48) | **100.00 %** | 100.00 % |
 | JevBench public standard (72) | **88.89 %** | 98.61 % |
-| JevBench public hard (111) | **45.95 %** | 72.07 % |
-| judge proxy (600: routing + answer adequacy) | **89.00 %** | 90.50 % |
-| realistic transfer set (600) | **90.83 %** | 86.67 % |
-| stated-distribution probability (1,036) | **80.50 %** | 72.97 % |
-| hard proxy (600) | **42.17 %** | 54.83 % |
 **Offline comparison with the untuned base.** This separate `transformers` runner used native
 wording, no system prompt, one option order, T=1. The base and tuned checkpoint were scored by the
@@ -167,14 +182,14 @@ same offline path; these numbers are indicative of the base-model change, not th
 |---|---|---|
 | JevBench public easy (48) | 97.92 % | **100.00 %** |
 | JevBench public standard (72) | 70.83 % | **93.06 %** |
-| JevBench public hard (111) | 47.75 % | **46.85 %** |
-| judge proxy (600: routing + answer adequacy) | 63.17 % | **88.83 %** |
-| realistic transfer set (600) | 62.50 % | **90.67 %** |
-| stated-distribution probability (1,036) | 31.56 % | **79.83 %** |
-Against the untuned base, five suites improve in this offline run; public hard falls by 0.90 percentage points. Served and offline probabilities differ even on identical prompts, so use
-the served table for expected endpoint behavior. Hard-tier ECE at served T is 0.198 (T=1.55) against
-Jev 1.13's 0.099 raw, and mean TV to the stated distributions is 0.139 at served T against Jev's
 0.192 raw; full calibration table: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#calibration).
 **Speed** — raw serial latency on one H200 with SGLang 0.5.20, using a 242-decision profile averaging
@@ -191,7 +206,7 @@ for speed only; no accuracy comparison is implied.
 Throughput has been measured for the 8B only (see `docs/BENCHMARKS.md`); no equivalent sweep has been run for the 3B.
-On the public classification and decision suites (400 cases/suite, seed 13, served endpoints): AG News 83.0 %, typed decisions 69.0 %, MASSIVE intent mean 83.3 %, email spam 93.0 %, phishing 89.8 %. Full table, per-language and per-workflow breakdown: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#public-classification-and-decision-suites) and [`docs/public-classification-suites.md`](docs/public-classification-suites.md).
 A JevBench v1.4.1 run has been requested; the sealed-set result is not yet available. Full report: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
@@ -200,7 +215,7 @@ A JevBench v1.4.1 run has been requested; the sealed-set result is not yet avail
 - **Base model:** `mistralai/Ministral-3-3B-Instruct-2512-BF16`, revision `b6d637bef2393152b3da2b2fde72eecdee30557e` (Apache-2.0).
 - **Adapter:** LoRA r=16, α=32, dropout 0, on `q_proj k_proj v_proj o_proj gate_proj up_proj down_proj` of the language-model projections only (vision tower and multimodal projector excluded), **24,707,072** trainable parameters, PEFT 0.21.0. Adapter file `adapter_model.safetensors`, 135,113,048 bytes, sha256 `a8e3eb341e27c1a49a282e327c2a2038906abb0eee771e80debbb4a4c45f7afc`.
 - **Merged BF16 checkpoint:** merging the adapter into the base changed **182 tensors**, none of them outside the language-model projections, maximum absolute weight change **0.0023**.
-- **Serving details:** native chat-template wording, served without a system prompt, at a fixed temperature (T = 1.55, fitted on held-out calibration data); served model name `standard-one-3b` behind stock SGLang 0.5.20 via `jev-adapter` (`POST /v1/systemone`); single, caller-supplied option order, no rotation ensemble; 8,192-token context.
 | Path | Contents |
 |---|---|
@@ -223,18 +238,18 @@ An exact-text overlap audit against the public JevBench tiers found 0 exact scen
 ## Limitations
-- **Public hard tier:** the served 3B score is **45.95 %**, versus **72.07 %** for Jev 1.13. In the
-  separate offline base comparison, Standard One 3B scores **46.85 %**, **0.90 percentage points** below the untuned base's **47.75 %**.
-- Served probabilities are temperature-scaled by a single global value; if you apply this model to a
   materially different question distribution, re-fitting that temperature is advisable rather than
-  assuming 1.55 transfers.
 - At most 26 options per question (one uppercase letter per option, `A`–`Z`).
 - The sealed JevBench set has not been measured for this model.
-- Served and offline probabilities can differ on identical prompts (mean total-variation ≈0.07–0.09 on the
   hard tier); served numbers are treated as authoritative.
 - Korean is a small share of multilingual training alongside English.
 - The card reports text benchmarks; it does not establish decision accuracy on image inputs.
-- Ten-way support triage (35–40 %) and RAG passage relevance (57–60 %) are weak zero-shot; fine-tune
   for those.
 ## Licence

 > **Updated weights (v2, 2026-09-26).** If you downloaded this model before, download it again or pin
 > `revision="v2"`. Earlier versions stay available under the tags `v1` and `v1.1`.
 **Version:** v2
 ![Standard One benchmark card: JevBench public tiers, held-out suites, stated-distribution probability, hard-tier calibration, latency and throughput for Standard One 8B, Standard One 3B and Jev 1.13.](docs/assets/00-benchmark-card.png)
 *The figure combines results from different measurement paths. See [Benchmarks](#benchmarks) for
+served versus offline conditions; measured 24–26 September 2026.*
 ## At a glance
 - Send a state and a bounded rubric to receive probabilities for the supplied labels:
   `choice` selects among labeled options, `noul` is yes/no, and `score` uses an ordinal scale.
   The endpoint scores the labels in one forward pass without decoding answer text.
+- In the same-run **offline** comparison with its untuned base, 3B improves on all six suites. These are not served-endpoint results.
 - Probabilities are temperature-scaled and calibration-checked (hard-tier ECE, distribution
   total-variation) — see Benchmarks below.
 - The shared training mixture covers English, Japanese, Chinese, Spanish, French, German,
   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-3B-Instruct-2512-BF16 \
   --tokenizer-revision b6d637bef2393152b3da2b2fde72eecdee30557e \
+  --prompt-wording native --native-system-prompt none --default-temperature 0.95 \
+  --temperature-by-type choice=0.95,noul=1.05,score=0.85
 ```
+### Prompt wording
+`jev-adapter` can phrase a request in two ways. Both were measured on the same served endpoint (no system prompt) and each has its
+own fitted temperatures; the recommended default is `native` unless `served` scores at least 1.0 percentage point higher on
+the suites below and an offline check agrees.
+| `--prompt-wording` | What the prompt looks like | Temperatures (default; choice / noul / score) | Mean accuracy, 10 suites |
+|---|---|---|---:|
+| `native` (recommended default) | `State:` / `Question:` / `Options:` headers, options as `A. name: description` | 0.95; 0.95 / 1.05 / 0.85 | 69.9 % |
+| `served` | `Context:` / `Question:` / `Options:` headers, options as `A: name: description` | 0.90; 0.90 / 1.05 / 0.95 | 69.8 % |
+The 10 suites: judge proxy, hard proxy, stated-distribution probability, realistic transfer set, MuSiQue (multiple choice), SQuAD 2.0 unanswerable questions, ContractNLI, PAWS-X (English), a held-out hard decision set and a consistency set. None of them is a JevBench tier, and no JevBench item was used to choose the wording or the temperatures.
+To use `served`, pass `--prompt-wording served --default-temperature 0.90 --temperature-by-type choice=0.90,noul=1.05,score=0.95`.
 Try it:
 ```bash
   "usage": {"input_tokens": 96, "output_tokens": 0},
   "metadata": {
     "confidence_method": "1 - normalized_entropy",
+    "temperature": 0.95,
+    "temperature_by_type": {"choice": 0.95, "noul": 1.05, "score": 0.85},
     "evaluations": 1,
     "adapter_elapsed_ms": 22.4
   }
 ## Benchmarks
 **Served endpoint results (the release configuration).** Merged BF16 weights through SGLang 0.5.20
+and `jev-adapter`, native wording, no system prompt, one option order, per-answer-type temperatures (choice 0.95, noul 1.05, score 0.85). The wording and the temperatures were chosen on non-JevBench data. Jev 1.13 was
 measured on the same items through its hosted endpoint; its probabilities are raw, with no
 temperature applied. These are our measurements, not official sealed-set JevBench scores.
 |---|---|---|
 | JevBench public easy (48) | **100.00 %** | 100.00 % |
 | JevBench public standard (72) | **88.89 %** | 98.61 % |
+| JevBench public hard (111) | **46.85 %** | 72.07 % |
+| judge proxy (600: routing + answer adequacy) | **88.17 %** | 90.50 % |
+| realistic transfer set (600) | **90.67 %** | 86.67 % |
+| stated-distribution probability (1,036) | **78.76 %** | 72.97 % |
+| hard proxy (600) | **44.67 %** | 54.83 % |
 **Offline comparison with the untuned base.** This separate `transformers` runner used native
 wording, no system prompt, one option order, T=1. The base and tuned checkpoint were scored by the
 |---|---|---|
 | JevBench public easy (48) | 97.92 % | **100.00 %** |
 | JevBench public standard (72) | 70.83 % | **93.06 %** |
+| JevBench public hard (111) | 47.75 % | **49.55 %** |
+| judge proxy (600: routing + answer adequacy) | 63.17 % | **88.00 %** |
+| realistic transfer set (600) | 62.50 % | **88.50 %** |
+| stated-distribution probability (1,036) | 31.56 % | **79.25 %** |
+Against the untuned base, all six suites improve on this offline run. Served and offline probabilities differ even on identical prompts, so use
+the served table for expected endpoint behavior. Hard-tier ECE at the served temperatures is 0.225 against
+Jev 1.13's 0.099 raw, and mean TV to the stated distributions is 0.117 at served T against Jev's
 0.192 raw; full calibration table: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#calibration).
 **Speed** — raw serial latency on one H200 with SGLang 0.5.20, using a 242-decision profile averaging
 Throughput has been measured for the 8B only (see `docs/BENCHMARKS.md`); no equivalent sweep has been run for the 3B.
+On the public classification and decision suites (400 cases/suite, seed 13, served endpoints): AG News 84.2 %, typed decisions 67.8 %, MASSIVE intent mean 82.9 %, email spam 93.2 %, phishing 89.5 %. Full table, per-language and per-workflow breakdown: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#public-classification-and-decision-suites) and [`docs/public-classification-suites.md`](docs/public-classification-suites.md).
 A JevBench v1.4.1 run has been requested; the sealed-set result is not yet available. Full report: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
 - **Base model:** `mistralai/Ministral-3-3B-Instruct-2512-BF16`, revision `b6d637bef2393152b3da2b2fde72eecdee30557e` (Apache-2.0).
 - **Adapter:** LoRA r=16, α=32, dropout 0, on `q_proj k_proj v_proj o_proj gate_proj up_proj down_proj` of the language-model projections only (vision tower and multimodal projector excluded), **24,707,072** trainable parameters, PEFT 0.21.0. Adapter file `adapter_model.safetensors`, 135,113,048 bytes, sha256 `a8e3eb341e27c1a49a282e327c2a2038906abb0eee771e80debbb4a4c45f7afc`.
 - **Merged BF16 checkpoint:** merging the adapter into the base changed **182 tensors**, none of them outside the language-model projections, maximum absolute weight change **0.0023**.
+- **Serving details:** native chat-template wording, served without a system prompt, at fixed per-answer-type temperatures (choice 0.95, noul 1.05, score 0.85; fitted on held-out and public-train calibration data, no JevBench item); served model name `standard-one-3b` behind stock SGLang 0.5.20 via `jev-adapter` (`POST /v1/systemone`); single, caller-supplied option order, no rotation ensemble; 8,192-token context.
 | Path | Contents |
 |---|---|
 ## Limitations
+- **Public hard tier:** the served 3B score is **46.85 %**, versus **72.07 %** for Jev 1.13. In the
+  separate offline base comparison, Standard One 3B scores **49.55 %**, **1.80 percentage points** above the untuned base's **47.75 %**.
+- Served probabilities are temperature-scaled by one value per answer type; if you apply this model to a
   materially different question distribution, re-fitting that temperature is advisable rather than
+  assuming these values transfer.
 - At most 26 options per question (one uppercase letter per option, `A`–`Z`).
 - The sealed JevBench set has not been measured for this model.
+- Served and offline probabilities can differ on identical prompts (mean total-variation ≈0.08 on the
   hard tier); served numbers are treated as authoritative.
 - Korean is a small share of multilingual training alongside English.
 - The card reports text benchmarks; it does not establish decision accuracy on image inputs.
+- Ten-way support triage (42 %) and RAG passage relevance (57 %) are weak zero-shot; fine-tune
   for those.
 ## Licence

SHA256SUMS CHANGED Viewed

@@ -1,40 +1,41 @@
 1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747  LICENSE
 916927708f20ec40252055f98ad7f09f5dcfacadc31e99d85bcac19f285b7bad  MERGE_REPORT.json
 f10343f951ecb3a72670e12720de6e04d59e9cb872e84b2ceddd7d54853bb7e7  NOTICE
-372d9160e95493d1d8e5c94d82b12fb0693fd36e4a50f5206d287ee74f189046  QUICKSTART.md
-faf66b200ab0a00f0e7f7c7174af1996c45376d3e529ed08ea7582b232c1e602  README.md
 331b249682cd52226c50e533f59825184997f3b31f9a62b2c3fea940db6999c5  SYSTEM_PROMPT.txt
 0701cfbdc2b7d44fdbad104dff604faee4b0543e8247624568777fe465746f9b  chat_template.jinja
 c4446cf970fefd8b65f2e3119c30d406ea7026dca362dfb319c9aa08643131c3  config.json
-141067bb117600df9e09a46098e6a1f26e685ea5eab77f73b20352bce6d503d6  docs/BENCHMARKS.md
-96127670ad1e5bdd336a33118d6246330c27ac9e4fcb3f4f22767033809cd889  docs/assets/00-benchmark-card.png
-28c67a8c861f421fae5889af259d0850e7fb6eadb11f247adef72b94bf00b187  docs/assets/00-benchmark-card.svg
 7313f2b0645ee401d082572986dcee0c08d3daed1ae79fa650cfa685570ad7e2  docs/assets/02-latency-vs-qwen.png
 0b48a0dd4b4c9ac7c1842836940348edc2e1ba697184943f0c84c3e3d3c7f2d0  docs/assets/02-latency-vs-qwen.svg
 5f430f8841e393548f080833524840adb93deef1c6f75dca93d0571262a2ac68  docs/assets/04-throughput.png
 dd888be78c0e32894cf04a8e3fbf8757e7218873668fa4443ba47c7c1c48b6dc  docs/assets/04-throughput.svg
-2d056ba120ea239ee7a2f6840187589f779d5e6266185d835c36a4693984e4b8  docs/assets/05-gain-over-base.png
-0009567d9bd46390f95b6e8e8f1defd02c86b49c62bb40fec28431bf25725f4d  docs/assets/05-gain-over-base.svg
-837f07a1934509e3dffd4bc1b824d7c983b8dccd47457e5fd881a17f6b2e047c  docs/assets/06-vs-jev.png
-085a13c149a8245f046beeca014fe90b85a8faa379639c2cdedcd443ad941a4f  docs/assets/06-vs-jev.svg
-773cdf5156019a276bb7b7153a1962f38a9439a323644cd7a7291fb08a60788f  docs/public-classification-suites.md
-95188c27727bf11bde728f1e4dde96e5023501efb67fd2a790e41011cff16984  evidence/calibration-leaderboard-ablate-nosys.json
-2d70606bee5c231976b8d3efacbb6bde460373a5285ec23290140a12a963016e  evidence/calibration-leaderboard-merged-native.json
-0f433655d8e129dbbc11c31e3ed19cba8047719bffb2882b9c4e0b8eb4bc7085  evidence/served-nosys-easy-report.json
-8a7dc4ee7b607a650d787ce513b22064b0227c723514d56702c1529f505982b2  evidence/served-nosys-hard-report.json
 7b9b1ee2f93166a817ead61bbc0f4c62984afe7d6c8affefd051007582d0a485  evidence/served-nosys-latency-idle-gpu.json
-5b217fe780b4c8c7cb292d57630b0f0a0bb5dcbe275f731272d6f0804cc14a68  evidence/served-nosys-original-report.json
 e0923390059f84a9180b00e5501778acc45ea9856cd7f2fd68208b360927c677  generation_config.json
 7029a24151316b12d84b0e666c6af7d46bfe07e9549d59e513bb34afabb43073  model-00001-of-00002.safetensors
 ad184b56624eaca81a22c7fb7bef034edd25bd435cfb173a76d0db1b2c67ff2b  model-00002-of-00002.safetensors
 55641440eb385d8bc83a319b0a92f60fe45e6aef4b6992245eee91142237786e  model.safetensors.index.json
 d15587c57574d2a92c66ff2552c6db5173d4c440ad47f7454dc51bad0ecaeb59  params.json
 ece2373c2ae391bce18785a1810543bc6173a1f28b6767cffab2c35dbea5f002  processor_config.json
-f9913e0d49fc053df656058d13af2523b6263f9396efad7a6fa06e7a121e1acf  release-manifest.json
-eef2a88a000d2ec294379755770c147327cb79d3d7278c711899bd36797d7a02  server/.gitignore
 1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747  server/LICENSE
 0f1f023b916d38d7c12af3a71e09f5e82b819a7e156e4992118bb160ee63ede1  server/NOTICE
-8db88b70be27a581c975b872abc0ba7eaafa092cc38db4debd3970bc83fb013a  server/README.md
 88e4dea8cc92f467a03bdd45a5fb968e402517116b09e24edcea85bb15f121e5  server/benchmarks/DECISION_BENCHMARK_SELECTION.md
 7fea6b9dc44e34ce448f1d80a9f2d273b76ab9b0ac1859c032fa4faa4f458501  server/benchmarks/METRIC_VALIDATION.md
 bb88f12ef294680e389147d781bf1f8680ea0a5ca9ed554730d8ea29f1c25eb9  server/benchmarks/PUBLIC_DATASETS.md

 1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747  LICENSE
 916927708f20ec40252055f98ad7f09f5dcfacadc31e99d85bcac19f285b7bad  MERGE_REPORT.json
 f10343f951ecb3a72670e12720de6e04d59e9cb872e84b2ceddd7d54853bb7e7  NOTICE
+627517b41bcfc5845e85228d227629869c420b174309b02c4dce93a7c985d7d0  QUICKSTART.md
+1a7bc5b929d9131d653dc467d0b83962309528e7db792ed2dc272293eace8a43  README.md
 331b249682cd52226c50e533f59825184997f3b31f9a62b2c3fea940db6999c5  SYSTEM_PROMPT.txt
 0701cfbdc2b7d44fdbad104dff604faee4b0543e8247624568777fe465746f9b  chat_template.jinja
 c4446cf970fefd8b65f2e3119c30d406ea7026dca362dfb319c9aa08643131c3  config.json
+63f74ef21e242575af5ba24e3bdb5fe343f6fb382ddb4ab040aeca91b1b6c590  docs/BENCHMARKS.md
+1395bca8ffbf3b3fd6ecffd28dedaf424f7b859123e0d4f19bc10ef406b9746d  docs/assets/00-benchmark-card.png
+f5b9f65e777b3443e14bd2678574b30b266a1ca15dddfe131ce44279b220f0e6  docs/assets/00-benchmark-card.svg
 7313f2b0645ee401d082572986dcee0c08d3daed1ae79fa650cfa685570ad7e2  docs/assets/02-latency-vs-qwen.png
 0b48a0dd4b4c9ac7c1842836940348edc2e1ba697184943f0c84c3e3d3c7f2d0  docs/assets/02-latency-vs-qwen.svg
 5f430f8841e393548f080833524840adb93deef1c6f75dca93d0571262a2ac68  docs/assets/04-throughput.png
 dd888be78c0e32894cf04a8e3fbf8757e7218873668fa4443ba47c7c1c48b6dc  docs/assets/04-throughput.svg
+4900de357045839e7cd48331a8ae4db8a8ce722a41a462ec2ccbbca03587f84e  docs/assets/05-gain-over-base.png
+e24a379bc590f91ec757edec44b7c286e762966095093dc31b67903bbab7337e  docs/assets/05-gain-over-base.svg
+0274d259fe3ea27ba6a5122452ab15970ba51575cf2e4f278e983e1c3cbe1528  docs/assets/06-vs-jev.png
+034a5d015b98b34e55b8639271e9fcf4676820f1301f99d48e179b4381ebf4e6  docs/assets/06-vs-jev.svg
+e6302b40bb4acd94fbac3d7fc08c92c4b5930b1e4127b5e7e8bff9c03b6ccd24  docs/public-classification-suites.md
+a6423137b37e89b106336034107c41c8915818c421be869fcf76684cf67d667e  evidence/calibration-leaderboard-ablate-nosys.json
+b6fc93a0c07713654c0b98a34380b78d558db774158b3f681634e1c9c92e2421  evidence/calibration-leaderboard-merged-native.json
+af92bc8ad5010acc9c7d134080325dc09d7e4343dfee72bc4336adf496227e91  evidence/calibration-per-type-served.json
+e622df488a08b425d5a464b2ab7e8b2d60b3b703baef8e4bdf927357adb87c2e  evidence/prompt-wording-choice.json
+b07691c0eda24ef08288f58cc6e0906f4c5b7ced57ebb9cb49888e4e2ed8672c  evidence/served-nosys-easy-report.json
+60dad968740e62f6c19dd6d530e4299784b1a26e22e7e60f26bee42367ee6398  evidence/served-nosys-hard-report.json
 7b9b1ee2f93166a817ead61bbc0f4c62984afe7d6c8affefd051007582d0a485  evidence/served-nosys-latency-idle-gpu.json
+d6c43f4f53bf63ca49dbed9355f601d4791776b4bb99c38452abca5c7aa41b4c  evidence/served-nosys-original-report.json
 e0923390059f84a9180b00e5501778acc45ea9856cd7f2fd68208b360927c677  generation_config.json
 7029a24151316b12d84b0e666c6af7d46bfe07e9549d59e513bb34afabb43073  model-00001-of-00002.safetensors
 ad184b56624eaca81a22c7fb7bef034edd25bd435cfb173a76d0db1b2c67ff2b  model-00002-of-00002.safetensors
 55641440eb385d8bc83a319b0a92f60fe45e6aef4b6992245eee91142237786e  model.safetensors.index.json
 d15587c57574d2a92c66ff2552c6db5173d4c440ad47f7454dc51bad0ecaeb59  params.json
 ece2373c2ae391bce18785a1810543bc6173a1f28b6767cffab2c35dbea5f002  processor_config.json
+bed3c4c85545d51574d9b1c1040fa5f28dea801a491978ed39e044c19df5260e  release-manifest.json
 1495e1e757ef4d0925a2350563cf5754bb23c51701a8ec4fb3c5cdcbedae6747  server/LICENSE
 0f1f023b916d38d7c12af3a71e09f5e82b819a7e156e4992118bb160ee63ede1  server/NOTICE
+e94c7fd37026302738bec03a3a5e8d1e6ffc52a10b4b0c442de9139f81789443  server/README.md
 88e4dea8cc92f467a03bdd45a5fb968e402517116b09e24edcea85bb15f121e5  server/benchmarks/DECISION_BENCHMARK_SELECTION.md
 7fea6b9dc44e34ce448f1d80a9f2d273b76ab9b0ac1859c032fa4faa4f458501  server/benchmarks/METRIC_VALIDATION.md
 bb88f12ef294680e389147d781bf1f8680ea0a5ca9ed554730d8ea29f1c25eb9  server/benchmarks/PUBLIC_DATASETS.md

docs/BENCHMARKS.md CHANGED Viewed

@@ -1,7 +1,7 @@
 # Standard One — full benchmark report
-> **Note.** The benchmark figures in this report are for v1.1 and are being updated for v2. The training-data provenance and
-> overlap-audit sections below already describe the v2 weights.
 This report covers both released systems, **Standard One 8B** (`StandardThinking/StandardOne-8B` / `-LoRA`) and
 **Standard One 3B** (`StandardThinking/StandardOne-3B` / `-LoRA`). It is our own measurement, not an official JevBench

 # Standard One — full benchmark report
+> **Note.** The Standard One 3B v2 figures are on its model card. The benchmark figures in this report are still those of v1.1 and
+> will be updated when both sizes are on v2; the training-data provenance and overlap-audit sections already describe the v2 weights.
 This report covers both released systems, **Standard One 8B** (`StandardThinking/StandardOne-8B` / `-LoRA`) and
 **Standard One 3B** (`StandardThinking/StandardOne-3B` / `-LoRA`). It is our own measurement, not an official JevBench

docs/assets/00-benchmark-card.png CHANGED Viewed

Git LFS Details

SHA256: 96127670ad1e5bdd336a33118d6246330c27ac9e4fcb3f4f22767033809cd889
Pointer size: 131 Bytes
Size of remote file: 299 kB

Git LFS Details

SHA256: 1395bca8ffbf3b3fd6ecffd28dedaf424f7b859123e0d4f19bc10ef406b9746d
Pointer size: 131 Bytes
Size of remote file: 298 kB

docs/assets/00-benchmark-card.svg CHANGED Viewed

docs/assets/05-gain-over-base.png CHANGED Viewed

Git LFS Details

SHA256: 2d056ba120ea239ee7a2f6840187589f779d5e6266185d835c36a4693984e4b8
Pointer size: 131 Bytes
Size of remote file: 176 kB

Git LFS Details

SHA256: 4900de357045839e7cd48331a8ae4db8a8ce722a41a462ec2ccbbca03587f84e
Pointer size: 131 Bytes
Size of remote file: 180 kB

docs/assets/05-gain-over-base.svg CHANGED Viewed

docs/assets/06-vs-jev.png CHANGED Viewed

Git LFS Details

SHA256: 837f07a1934509e3dffd4bc1b824d7c983b8dccd47457e5fd881a17f6b2e047c
Pointer size: 131 Bytes
Size of remote file: 266 kB

Git LFS Details

SHA256: 0274d259fe3ea27ba6a5122452ab15970ba51575cf2e4f278e983e1c3cbe1528
Pointer size: 131 Bytes
Size of remote file: 266 kB

docs/assets/06-vs-jev.svg CHANGED Viewed

docs/public-classification-suites.md CHANGED Viewed

@@ -1,5 +1,7 @@
 # Public classification and decision suites
 Measured 25 September 2026 through the served endpoints (merged BF16, SGLang 0.5.20, jev-adapter, native wording, no system prompt, single option order). Each suite: 400 cases sampled with seed 13 from the public test split, one request per case; ECE (10-bin, top label) uses the served temperature (8B T=1.65, 3B T=1.55). Accuracy and macro-F1 are argmax metrics and do not depend on the temperature. banking77 (77 labels) is not runnable: the endpoint accepts at most 26 options per question.
 ## Public datasets

 # Public classification and decision suites
+> **Note.** The Standard One 3B v2 figures for these suites are on its model card; the tables below are still those of v1.1.
 Measured 25 September 2026 through the served endpoints (merged BF16, SGLang 0.5.20, jev-adapter, native wording, no system prompt, single option order). Each suite: 400 cases sampled with seed 13 from the public test split, one request per case; ECE (10-bin, top label) uses the served temperature (8B T=1.65, 3B T=1.55). Accuracy and macro-F1 are argmax metrics and do not depend on the temperature. banking77 (77 labels) is not runnable: the endpoint accepts at most 26 options per question.
 ## Public datasets

evidence/calibration-leaderboard-ablate-nosys.json CHANGED Viewed

@@ -1,237 +1,399 @@
 {
  "arm_fitted": "original",
  "inputs": [
-  "runs/ablate-hard25-3b-r1-nosys/predictions.jsonl"
  ],
  "fit_suites": [
   "distprob-calib-v2",
   "internal-eval-hardproxy-calib-v1"
  ],
- "temperature": 1.5,
  "objective": "leaderboard",
- "axis_proxy_at_T1": 83.82616050141019,
- "axis_proxy_at_T": 87.03644794016614,
  "at_T1": {
   "jevbench-easy": {
    "original": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.001149104215916778,
-    "calibration_axis_ece_part": 99.77017915681664
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.001149104215916778,
-    "calibration_axis_ece_part": 99.77017915681664
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.016422420620663625,
-    "calibration_axis_ece_part": 96.71551587586727
    },
    "rotation": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.016422420620663625,
-    "calibration_axis_ece_part": 96.71551587586727
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
-    "accuracy": 0.46846846846846846,
-    "ece_10bin": 0.2863840041801175,
-    "calibration_axis_ece_part": 42.7231991639765
    },
    "rotation": {
     "questions": 111,
-    "accuracy": 0.46846846846846846,
-    "ece_10bin": 0.2863840041801175,
-    "calibration_axis_ece_part": 42.7231991639765
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.8883333333333333,
-    "ece_10bin": 0.033608477626319376,
-    "calibration_axis_ece_part": 93.27830447473613
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.8883333333333333,
-    "ece_10bin": 0.033608477626319376,
-    "calibration_axis_ece_part": 93.27830447473613
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.43666666666666665,
-    "ece_10bin": 0.11217360972848735,
-    "calibration_axis_ece_part": 77.56527805430254
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.43666666666666665,
-    "ece_10bin": 0.11217360972848735,
-    "calibration_axis_ece_part": 77.56527805430254
    }
   },
   "internal-eval-hardproxy-v2": {
    "original": {
     "questions": 900,
-    "accuracy": 0.8066666666666666,
-    "ece_10bin": 0.0440976735832413,
-    "calibration_axis_ece_part": 91.18046528335174,
     "probability_items": 80,
-    "mean_tv_distance": 0.1277800123178266,
-    "fidelity_part": 87.22199876821735,
-    "calibration_axis_estimate": 89.20123202578455
    },
    "rotation": {
     "questions": 900,
-    "accuracy": 0.8066666666666666,
-    "ece_10bin": 0.0440976735832413,
-    "calibration_axis_ece_part": 91.18046528335174,
     "probability_items": 80,
-    "mean_tv_distance": 0.1277800123178266,
-    "fidelity_part": 87.22199876821735,
-    "calibration_axis_estimate": 89.20123202578455
    }
   },
   "internal-eval-hardproxy-v3": {
    "original": {
     "questions": 960,
-    "accuracy": 0.6416666666666667,
-    "ece_10bin": 0.140114867135236,
-    "calibration_axis_ece_part": 71.9770265729528,
     "probability_items": 120,
-    "mean_tv_distance": 0.3854807681079812,
-    "fidelity_part": 61.45192318920187,
-    "calibration_axis_estimate": 66.71447488107734
    },
    "rotation": {
     "questions": 960,
-    "accuracy": 0.6416666666666667,
-    "ece_10bin": 0.140114867135236,
-    "calibration_axis_ece_part": 71.9770265729528,
     "probability_items": 120,
-    "mean_tv_distance": 0.3854807681079812,
-    "fidelity_part": 61.45192318920187,
-    "calibration_axis_estimate": 66.71447488107734
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.48,
-    "ece_10bin": 0.08306918196174966,
-    "calibration_axis_ece_part": 83.38616360765006
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.48,
-    "ece_10bin": 0.08306918196174966,
-    "calibration_axis_ece_part": 83.38616360765006
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
-    "accuracy": 0.7827510917030568,
-    "ece_10bin": 0.17041662070579264,
-    "calibration_axis_ece_part": 65.91667585884147,
     "probability_items": 916,
-    "mean_tv_distance": 0.15733842604829681,
-    "fidelity_part": 84.26615739517031,
-    "calibration_axis_estimate": 75.09141662700588
    },
    "rotation": {
     "questions": 916,
-    "accuracy": 0.7827510917030568,
-    "ece_10bin": 0.17041662070579264,
-    "calibration_axis_ece_part": 65.91667585884147,
     "probability_items": 916,
-    "mean_tv_distance": 0.15733842604829681,
-    "fidelity_part": 84.26615739517031,
-    "calibration_axis_estimate": 75.09141662700588
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
-    "accuracy": 0.7982625482625483,
-    "ece_10bin": 0.1934569384835919,
-    "calibration_axis_ece_part": 61.30861230328162,
     "probability_items": 1036,
-    "mean_tv_distance": 0.11388774889852549,
-    "fidelity_part": 88.61122511014744,
-    "calibration_axis_estimate": 74.95991870671453
    },
    "rotation": {
     "questions": 1036,
-    "accuracy": 0.7982625482625483,
-    "ece_10bin": 0.1934569384835919,
-    "calibration_axis_ece_part": 61.30861230328162,
     "probability_items": 1036,
-    "mean_tv_distance": 0.11388774889852549,
-    "fidelity_part": 88.61122511014744,
-    "calibration_axis_estimate": 74.95991870671453
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
-    "accuracy": 0.9066666666666666,
-    "ece_10bin": 0.0825985048108747,
-    "calibration_axis_ece_part": 83.48029903782506,
     "probability_items": 120,
-    "mean_tv_distance": 0.11120485898001545,
-    "fidelity_part": 88.87951410199845,
-    "calibration_axis_estimate": 86.17990656991176
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.9066666666666666,
-    "ece_10bin": 0.0825985048108747,
-    "calibration_axis_ece_part": 83.48029903782506,
     "probability_items": 120,
-    "mean_tv_distance": 0.11120485898001545,
-    "fidelity_part": 88.87951410199845,
-    "calibration_axis_estimate": 86.17990656991176
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.009742608154769303,
-    "calibration_axis_ece_part": 98.05147836904614
    },
    "rotation": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.009742608154769303,
-    "calibration_axis_ece_part": 98.05147836904614
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
-    "accuracy": 0.675,
-    "ece_10bin": 0.11965060424992888,
-    "calibration_axis_ece_part": 76.06987915001422
    },
    "rotation": {
     "questions": 120,
-    "accuracy": 0.675,
-    "ece_10bin": 0.11965060424992888,
-    "calibration_axis_ece_part": 76.06987915001422
    }
   }
  },
@@ -240,223 +402,385 @@
    "original": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.008738382420806143,
-    "calibration_axis_ece_part": 98.25232351583877
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.008738382420806143,
-    "calibration_axis_ece_part": 98.25232351583877
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.05473490425907009,
-    "calibration_axis_ece_part": 89.05301914818598
    },
    "rotation": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.05473490425907009,
-    "calibration_axis_ece_part": 89.05301914818598
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
-    "accuracy": 0.46846846846846846,
-    "ece_10bin": 0.209595391205915,
-    "calibration_axis_ece_part": 58.080921758817006
    },
    "rotation": {
     "questions": 111,
-    "accuracy": 0.46846846846846846,
-    "ece_10bin": 0.209595391205915,
-    "calibration_axis_ece_part": 58.080921758817006
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.8883333333333333,
-    "ece_10bin": 0.02419928317439568,
-    "calibration_axis_ece_part": 95.16014336512086
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.8883333333333333,
-    "ece_10bin": 0.02419928317439568,
-    "calibration_axis_ece_part": 95.16014336512086
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.43666666666666665,
-    "ece_10bin": 0.05620456615385875,
-    "calibration_axis_ece_part": 88.75908676922825
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.43666666666666665,
-    "ece_10bin": 0.05620456615385875,
-    "calibration_axis_ece_part": 88.75908676922825
    }
   },
   "internal-eval-hardproxy-v2": {
    "original": {
     "questions": 900,
-    "accuracy": 0.8066666666666666,
-    "ece_10bin": 0.07746394431116115,
-    "calibration_axis_ece_part": 84.50721113776777,
     "probability_items": 80,
-    "mean_tv_distance": 0.14965105729945485,
-    "fidelity_part": 85.03489427005452,
-    "calibration_axis_estimate": 84.77105270391115
    },
    "rotation": {
     "questions": 900,
-    "accuracy": 0.8066666666666666,
-    "ece_10bin": 0.07746394431116115,
-    "calibration_axis_ece_part": 84.50721113776777,
     "probability_items": 80,
-    "mean_tv_distance": 0.14965105729945485,
-    "fidelity_part": 85.03489427005452,
-    "calibration_axis_estimate": 84.77105270391115
    }
   },
   "internal-eval-hardproxy-v3": {
    "original": {
     "questions": 960,
-    "accuracy": 0.6416666666666667,
-    "ece_10bin": 0.10491496455454948,
-    "calibration_axis_ece_part": 79.01700708909011,
     "probability_items": 120,
-    "mean_tv_distance": 0.31211445746013206,
-    "fidelity_part": 68.7885542539868,
-    "calibration_axis_estimate": 73.90278067153847
    },
    "rotation": {
     "questions": 960,
-    "accuracy": 0.6416666666666667,
-    "ece_10bin": 0.10491496455454948,
-    "calibration_axis_ece_part": 79.01700708909011,
     "probability_items": 120,
-    "mean_tv_distance": 0.31211445746013206,
-    "fidelity_part": 68.7885542539868,
-    "calibration_axis_estimate": 73.90278067153847
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.48,
-    "ece_10bin": 0.03986368848484867,
-    "calibration_axis_ece_part": 92.02726230303027
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.48,
-    "ece_10bin": 0.03986368848484867,
-    "calibration_axis_ece_part": 92.02726230303027
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
-    "accuracy": 0.7827510917030568,
-    "ece_10bin": 0.23232298669097232,
-    "calibration_axis_ece_part": 53.53540266180554,
     "probability_items": 916,
-    "mean_tv_distance": 0.17954366422697995,
-    "fidelity_part": 82.045633577302,
-    "calibration_axis_estimate": 67.79051811955377
    },
    "rotation": {
     "questions": 916,
-    "accuracy": 0.7827510917030568,
-    "ece_10bin": 0.23232298669097232,
-    "calibration_axis_ece_part": 53.53540266180554,
     "probability_items": 916,
-    "mean_tv_distance": 0.17954366422697995,
-    "fidelity_part": 82.045633577302,
-    "calibration_axis_estimate": 67.79051811955377
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
-    "accuracy": 0.7982625482625483,
-    "ece_10bin": 0.2562140570862304,
-    "calibration_axis_ece_part": 48.75718858275392,
     "probability_items": 1036,
-    "mean_tv_distance": 0.1372665400776386,
-    "fidelity_part": 86.27334599223614,
-    "calibration_axis_estimate": 67.51526728749504
    },
    "rotation": {
     "questions": 1036,
-    "accuracy": 0.7982625482625483,
-    "ece_10bin": 0.2562140570862304,
-    "calibration_axis_ece_part": 48.75718858275392,
     "probability_items": 1036,
-    "mean_tv_distance": 0.1372665400776386,
-    "fidelity_part": 86.27334599223614,
-    "calibration_axis_estimate": 67.51526728749504
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
-    "accuracy": 0.9066666666666666,
-    "ece_10bin": 0.12869745290106477,
-    "calibration_axis_ece_part": 74.26050941978704,
     "probability_items": 120,
-    "mean_tv_distance": 0.13473642194515204,
-    "fidelity_part": 86.52635780548479,
-    "calibration_axis_estimate": 80.39343361263592
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.9066666666666666,
-    "ece_10bin": 0.12869745290106477,
-    "calibration_axis_ece_part": 74.26050941978704,
     "probability_items": 120,
-    "mean_tv_distance": 0.13473642194515204,
-    "fidelity_part": 86.52635780548479,
-    "calibration_axis_estimate": 80.39343361263592
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.029814957164165524,
-    "calibration_axis_ece_part": 94.0370085671669
    },
    "rotation": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.029814957164165524,
-    "calibration_axis_ece_part": 94.0370085671669
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
-    "accuracy": 0.675,
-    "ece_10bin": 0.05513521009547392,
-    "calibration_axis_ece_part": 88.97295798090522
    },
    "rotation": {
     "questions": 120,
-    "accuracy": 0.675,
-    "ece_10bin": 0.05513521009547392,
-    "calibration_axis_ece_part": 88.97295798090522
    }
   }
  }
-}

 {
  "arm_fitted": "original",
  "inputs": [
+  "runs/ablate-r5-3b-nosys/predictions.jsonl"
  ],
  "fit_suites": [
   "distprob-calib-v2",
   "internal-eval-hardproxy-calib-v1"
  ],
+ "temperature": 1.1,
  "objective": "leaderboard",
+ "axis_proxy_at_T1": 86.94695078922626,
+ "axis_proxy_at_T": 87.85838373964691,
  "at_T1": {
   "jevbench-easy": {
    "original": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.048693904493442075,
+    "calibration_axis_ece_part": 90.26121910131158
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.048693904493442075,
+    "calibration_axis_ece_part": 90.26121910131158
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
+    "ece_10bin": 0.09516724430076529,
+    "calibration_axis_ece_part": 80.96655113984694
    },
    "rotation": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
+    "ece_10bin": 0.09516724430076529,
+    "calibration_axis_ece_part": 80.96655113984694
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
+    "accuracy": 0.4954954954954955,
+    "ece_10bin": 0.198064802424603,
+    "calibration_axis_ece_part": 60.3870395150794
    },
    "rotation": {
     "questions": 111,
+    "accuracy": 0.4954954954954955,
+    "ece_10bin": 0.198064802424603,
+    "calibration_axis_ece_part": 60.3870395150794
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.88,
+    "ece_10bin": 0.030882378998731596,
+    "calibration_axis_ece_part": 93.82352420025369
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.88,
+    "ece_10bin": 0.030882378998731596,
+    "calibration_axis_ece_part": 93.82352420025369
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.455,
+    "ece_10bin": 0.05174075785326058,
+    "calibration_axis_ece_part": 89.65184842934788
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.455,
+    "ece_10bin": 0.05174075785326058,
+    "calibration_axis_ece_part": 89.65184842934788
    }
   },
   "internal-eval-hardproxy-v2": {
    "original": {
     "questions": 900,
+    "accuracy": 0.8,
+    "ece_10bin": 0.042251628113046696,
+    "calibration_axis_ece_part": 91.54967437739066,
     "probability_items": 80,
+    "mean_tv_distance": 0.1410759775058439,
+    "fidelity_part": 85.89240224941561,
+    "calibration_axis_estimate": 88.72103831340314
    },
    "rotation": {
     "questions": 900,
+    "accuracy": 0.8,
+    "ece_10bin": 0.042251628113046696,
+    "calibration_axis_ece_part": 91.54967437739066,
     "probability_items": 80,
+    "mean_tv_distance": 0.1410759775058439,
+    "fidelity_part": 85.89240224941561,
+    "calibration_axis_estimate": 88.72103831340314
    }
   },
   "internal-eval-hardproxy-v3": {
    "original": {
     "questions": 960,
+    "accuracy": 0.675,
+    "ece_10bin": 0.09072671180283924,
+    "calibration_axis_ece_part": 81.85465763943215,
     "probability_items": 120,
+    "mean_tv_distance": 0.2891946928248282,
+    "fidelity_part": 71.08053071751719,
+    "calibration_axis_estimate": 76.46759417847467
    },
    "rotation": {
     "questions": 960,
+    "accuracy": 0.675,
+    "ece_10bin": 0.09072671180283924,
+    "calibration_axis_ece_part": 81.85465763943215,
     "probability_items": 120,
+    "mean_tv_distance": 0.2891946928248282,
+    "fidelity_part": 71.08053071751719,
+    "calibration_axis_estimate": 76.46759417847467
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.48833333333333334,
+    "ece_10bin": 0.04776417422286578,
+    "calibration_axis_ece_part": 90.44716515542684
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.48833333333333334,
+    "ece_10bin": 0.04776417422286578,
+    "calibration_axis_ece_part": 90.44716515542684
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
+    "accuracy": 0.7652838427947598,
+    "ece_10bin": 0.16726222714435904,
+    "calibration_axis_ece_part": 66.5475545711282,
     "probability_items": 916,
+    "mean_tv_distance": 0.16553263576974328,
+    "fidelity_part": 83.44673642302567,
+    "calibration_axis_estimate": 74.99714549707693
    },
    "rotation": {
     "questions": 916,
+    "accuracy": 0.7652838427947598,
+    "ece_10bin": 0.16726222714435904,
+    "calibration_axis_ece_part": 66.5475545711282,
     "probability_items": 916,
+    "mean_tv_distance": 0.16553263576974328,
+    "fidelity_part": 83.44673642302567,
+    "calibration_axis_estimate": 74.99714549707693
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
+    "accuracy": 0.7924710424710425,
+    "ece_10bin": 0.19778793823299243,
+    "calibration_axis_ece_part": 60.44241235340151,
     "probability_items": 1036,
+    "mean_tv_distance": 0.1177658932696707,
+    "fidelity_part": 88.22341067303293,
+    "calibration_axis_estimate": 74.33291151321723
    },
    "rotation": {
     "questions": 1036,
+    "accuracy": 0.7924710424710425,
+    "ece_10bin": 0.19778793823299243,
+    "calibration_axis_ece_part": 60.44241235340151,
     "probability_items": 1036,
+    "mean_tv_distance": 0.1177658932696707,
+    "fidelity_part": 88.22341067303293,
+    "calibration_axis_estimate": 74.33291151321723
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.0758498921611568,
+    "calibration_axis_ece_part": 84.83002156776864,
     "probability_items": 120,
+    "mean_tv_distance": 0.1370973008896632,
+    "fidelity_part": 86.29026991103368,
+    "calibration_axis_estimate": 85.56014573940115
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.0758498921611568,
+    "calibration_axis_ece_part": 84.83002156776864,
     "probability_items": 120,
+    "mean_tv_distance": 0.1370973008896632,
+    "fidelity_part": 86.29026991103368,
+    "calibration_axis_estimate": 85.56014573940115
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
+    "ece_10bin": 0.034073542299971456,
+    "calibration_axis_ece_part": 93.18529154000571
    },
    "rotation": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
+    "ece_10bin": 0.034073542299971456,
+    "calibration_axis_ece_part": 93.18529154000571
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.08593213770611521,
+    "calibration_axis_ece_part": 82.81357245877696
    },
    "rotation": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.08593213770611521,
+    "calibration_axis_ece_part": 82.81357245877696
+   }
+  },
+  "musique-mc": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.72,
+    "ece_10bin": 0.19159002386402638,
+    "calibration_axis_ece_part": 61.68199522719473
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.72,
+    "ece_10bin": 0.19159002386402638,
+    "calibration_axis_ece_part": 61.68199522719473
+   }
+  },
+  "squad2-abstain": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.818,
+    "ece_10bin": 0.11659810384560394,
+    "calibration_axis_ece_part": 76.68037923087921
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.818,
+    "ece_10bin": 0.11659810384560394,
+    "calibration_axis_ece_part": 76.68037923087921
+   }
+  },
+  "contractnli": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.604,
+    "ece_10bin": 0.08422282355384689,
+    "calibration_axis_ece_part": 83.15543528923061
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.604,
+    "ece_10bin": 0.08422282355384689,
+    "calibration_axis_ece_part": 83.15543528923061
+   }
+  },
+  "paws-x-en": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.728,
+    "ece_10bin": 0.03883046400000004,
+    "calibration_axis_ece_part": 92.2339072
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.728,
+    "ece_10bin": 0.03883046400000004,
+    "calibration_axis_ece_part": 92.2339072
+   }
+  },
+  "calib-public-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.705,
+    "ece_10bin": 0.054123000396006614,
+    "calibration_axis_ece_part": 89.17539992079868
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.705,
+    "ece_10bin": 0.054123000396006614,
+    "calibration_axis_ece_part": 89.17539992079868
+   }
+  },
+  "shortcut-diagnostic-eval-v1": {
+   "original": {
+    "questions": 601,
+    "accuracy": 0.7254575707154742,
+    "ece_10bin": 0.04654242872523272,
+    "calibration_axis_ece_part": 90.69151425495346
+   },
+   "rotation": {
+    "questions": 601,
+    "accuracy": 0.7254575707154742,
+    "ece_10bin": 0.04654242872523272,
+    "calibration_axis_ece_part": 90.69151425495346
+   }
+  },
+  "consistency-eval-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.6208333333333333,
+    "ece_10bin": 0.04866161773395039,
+    "calibration_axis_ece_part": 90.26767645320992
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.6208333333333333,
+    "ece_10bin": 0.04866161773395039,
+    "calibration_axis_ece_part": 90.26767645320992
+   }
+  },
+  "hard-eval-v4": {
+   "original": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.09676464952640845,
+    "calibration_axis_ece_part": 80.64707009471832
+   },
+   "rotation": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.09676464952640845,
+    "calibration_axis_ece_part": 80.64707009471832
+   }
+  },
+  "multihop-v4-eval": {
+   "original": {
+    "questions": 597,
+    "accuracy": 0.4807370184254606,
+    "ece_10bin": 0.07415618395803027,
+    "calibration_axis_ece_part": 85.16876320839395
+   },
+   "rotation": {
+    "questions": 597,
+    "accuracy": 0.4807370184254606,
+    "ece_10bin": 0.07415618395803027,
+    "calibration_axis_ece_part": 85.16876320839395
+   }
+  },
+  "ood-paraphrase-eval-v1": {
+   "original": {
+    "questions": 586,
+    "accuracy": 0.6791808873720137,
+    "ece_10bin": 0.04744817654998246,
+    "calibration_axis_ece_part": 90.51036469000351,
+    "probability_items": 70,
+    "mean_tv_distance": 0.3033238459161196,
+    "fidelity_part": 69.66761540838804,
+    "calibration_axis_estimate": 80.08899004919579
+   },
+   "rotation": {
+    "questions": 586,
+    "accuracy": 0.6791808873720137,
+    "ece_10bin": 0.04744817654998246,
+    "calibration_axis_ece_part": 90.51036469000351,
+    "probability_items": 70,
+    "mean_tv_distance": 0.3033238459161196,
+    "fidelity_part": 69.66761540838804,
+    "calibration_axis_estimate": 80.08899004919579
+   }
+  },
+  "gen-v5-eval": {
+   "original": {
+    "questions": 738,
+    "accuracy": 0.7086720867208672,
+    "ece_10bin": 0.06021533660245647,
+    "calibration_axis_ece_part": 87.9569326795087
+   },
+   "rotation": {
+    "questions": 738,
+    "accuracy": 0.7086720867208672,
+    "ece_10bin": 0.06021533660245647,
+    "calibration_axis_ece_part": 87.9569326795087
    }
   }
  },
    "original": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.06461285288527517,
+    "calibration_axis_ece_part": 87.07742942294496
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.06461285288527517,
+    "calibration_axis_ece_part": 87.07742942294496
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
+    "ece_10bin": 0.1184115185905035,
+    "calibration_axis_ece_part": 76.3176962818993
    },
    "rotation": {
     "questions": 72,
     "accuracy": 0.9305555555555556,
+    "ece_10bin": 0.1184115185905035,
+    "calibration_axis_ece_part": 76.3176962818993
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
+    "accuracy": 0.4954954954954955,
+    "ece_10bin": 0.19930132460492345,
+    "calibration_axis_ece_part": 60.13973507901531
    },
    "rotation": {
     "questions": 111,
+    "accuracy": 0.4954954954954955,
+    "ece_10bin": 0.19930132460492345,
+    "calibration_axis_ece_part": 60.13973507901531
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.88,
+    "ece_10bin": 0.021633540662693674,
+    "calibration_axis_ece_part": 95.67329186746126
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.88,
+    "ece_10bin": 0.021633540662693674,
+    "calibration_axis_ece_part": 95.67329186746126
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.455,
+    "ece_10bin": 0.04300048280951459,
+    "calibration_axis_ece_part": 91.39990343809708
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.455,
+    "ece_10bin": 0.04300048280951459,
+    "calibration_axis_ece_part": 91.39990343809708
    }
   },
   "internal-eval-hardproxy-v2": {
    "original": {
     "questions": 900,
+    "accuracy": 0.8,
+    "ece_10bin": 0.06325979798087637,
+    "calibration_axis_ece_part": 87.34804040382473,
     "probability_items": 80,
+    "mean_tv_distance": 0.14619680214307737,
+    "fidelity_part": 85.38031978569227,
+    "calibration_axis_estimate": 86.3641800947585
    },
    "rotation": {
     "questions": 900,
+    "accuracy": 0.8,
+    "ece_10bin": 0.06325979798087637,
+    "calibration_axis_ece_part": 87.34804040382473,
     "probability_items": 80,
+    "mean_tv_distance": 0.14619680214307737,
+    "fidelity_part": 85.38031978569227,
+    "calibration_axis_estimate": 86.3641800947585
    }
   },
   "internal-eval-hardproxy-v3": {
    "original": {
     "questions": 960,
+    "accuracy": 0.675,
+    "ece_10bin": 0.09330695930701657,
+    "calibration_axis_ece_part": 81.33860813859668,
     "probability_items": 120,
+    "mean_tv_distance": 0.27555711918108655,
+    "fidelity_part": 72.44428808189134,
+    "calibration_axis_estimate": 76.89144811024401
    },
    "rotation": {
     "questions": 960,
+    "accuracy": 0.675,
+    "ece_10bin": 0.09330695930701657,
+    "calibration_axis_ece_part": 81.33860813859668,
     "probability_items": 120,
+    "mean_tv_distance": 0.27555711918108655,
+    "fidelity_part": 72.44428808189134,
+    "calibration_axis_estimate": 76.89144811024401
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.48833333333333334,
+    "ece_10bin": 0.036512514045423335,
+    "calibration_axis_ece_part": 92.69749719091533
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.48833333333333334,
+    "ece_10bin": 0.036512514045423335,
+    "calibration_axis_ece_part": 92.69749719091533
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
+    "accuracy": 0.7652838427947598,
+    "ece_10bin": 0.18328289452370788,
+    "calibration_axis_ece_part": 63.34342109525842,
     "probability_items": 916,
+    "mean_tv_distance": 0.16980729711621512,
+    "fidelity_part": 83.0192702883785,
+    "calibration_axis_estimate": 73.18134569181845
    },
    "rotation": {
     "questions": 916,
+    "accuracy": 0.7652838427947598,
+    "ece_10bin": 0.18328289452370788,
+    "calibration_axis_ece_part": 63.34342109525842,
     "probability_items": 916,
+    "mean_tv_distance": 0.16980729711621512,
+    "fidelity_part": 83.0192702883785,
+    "calibration_axis_estimate": 73.18134569181845
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
+    "accuracy": 0.7924710424710425,
+    "ece_10bin": 0.21368694801717578,
+    "calibration_axis_ece_part": 57.26261039656484,
     "probability_items": 1036,
+    "mean_tv_distance": 0.12240806428161144,
+    "fidelity_part": 87.75919357183885,
+    "calibration_axis_estimate": 72.51090198420184
    },
    "rotation": {
     "questions": 1036,
+    "accuracy": 0.7924710424710425,
+    "ece_10bin": 0.21368694801717578,
+    "calibration_axis_ece_part": 57.26261039656484,
     "probability_items": 1036,
+    "mean_tv_distance": 0.12240806428161144,
+    "fidelity_part": 87.75919357183885,
+    "calibration_axis_estimate": 72.51090198420184
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.08990976562750098,
+    "calibration_axis_ece_part": 82.0180468744998,
     "probability_items": 120,
+    "mean_tv_distance": 0.14216306628445002,
+    "fidelity_part": 85.783693371555,
+    "calibration_axis_estimate": 83.9008701230274
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.08990976562750098,
+    "calibration_axis_ece_part": 82.0180468744998,
     "probability_items": 120,
+    "mean_tv_distance": 0.14216306628445002,
+    "fidelity_part": 85.783693371555,
+    "calibration_axis_estimate": 83.9008701230274
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
+    "ece_10bin": 0.04066830663029119,
+    "calibration_axis_ece_part": 91.86633867394175
    },
    "rotation": {
     "questions": 360,
     "accuracy": 0.9638888888888889,
+    "ece_10bin": 0.04066830663029119,
+    "calibration_axis_ece_part": 91.86633867394175
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.09207493110817179,
+    "calibration_axis_ece_part": 81.58501377836565
    },
    "rotation": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.09207493110817179,
+    "calibration_axis_ece_part": 81.58501377836565
+   }
+  },
+  "musique-mc": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.72,
+    "ece_10bin": 0.21948894295098328,
+    "calibration_axis_ece_part": 56.10221140980334
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.72,
+    "ece_10bin": 0.21948894295098328,
+    "calibration_axis_ece_part": 56.10221140980334
+   }
+  },
+  "squad2-abstain": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.818,
+    "ece_10bin": 0.14491400493910717,
+    "calibration_axis_ece_part": 71.01719901217857
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.818,
+    "ece_10bin": 0.14491400493910717,
+    "calibration_axis_ece_part": 71.01719901217857
+   }
+  },
+  "contractnli": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.604,
+    "ece_10bin": 0.09492789828643218,
+    "calibration_axis_ece_part": 81.01442034271356
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.604,
+    "ece_10bin": 0.09492789828643218,
+    "calibration_axis_ece_part": 81.01442034271356
+   }
+  },
+  "paws-x-en": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.728,
+    "ece_10bin": 0.03295066485281044,
+    "calibration_axis_ece_part": 93.4098670294379
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.728,
+    "ece_10bin": 0.03295066485281044,
+    "calibration_axis_ece_part": 93.4098670294379
+   }
+  },
+  "calib-public-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.705,
+    "ece_10bin": 0.06966694436859198,
+    "calibration_axis_ece_part": 86.0666111262816
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.705,
+    "ece_10bin": 0.06966694436859198,
+    "calibration_axis_ece_part": 86.0666111262816
+   }
+  },
+  "shortcut-diagnostic-eval-v1": {
+   "original": {
+    "questions": 601,
+    "accuracy": 0.7254575707154742,
+    "ece_10bin": 0.0575774141959636,
+    "calibration_axis_ece_part": 88.48451716080727
+   },
+   "rotation": {
+    "questions": 601,
+    "accuracy": 0.7254575707154742,
+    "ece_10bin": 0.0575774141959636,
+    "calibration_axis_ece_part": 88.48451716080727
+   }
+  },
+  "consistency-eval-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.6208333333333333,
+    "ece_10bin": 0.04477509477458718,
+    "calibration_axis_ece_part": 91.04498104508257
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.6208333333333333,
+    "ece_10bin": 0.04477509477458718,
+    "calibration_axis_ece_part": 91.04498104508257
+   }
+  },
+  "hard-eval-v4": {
+   "original": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.08011158172685723,
+    "calibration_axis_ece_part": 83.97768365462855
+   },
+   "rotation": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.08011158172685723,
+    "calibration_axis_ece_part": 83.97768365462855
+   }
+  },
+  "multihop-v4-eval": {
+   "original": {
+    "questions": 597,
+    "accuracy": 0.4807370184254606,
+    "ece_10bin": 0.06060400738795974,
+    "calibration_axis_ece_part": 87.87919852240805
+   },
+   "rotation": {
+    "questions": 597,
+    "accuracy": 0.4807370184254606,
+    "ece_10bin": 0.06060400738795974,
+    "calibration_axis_ece_part": 87.87919852240805
+   }
+  },
+  "ood-paraphrase-eval-v1": {
+   "original": {
+    "questions": 586,
+    "accuracy": 0.6791808873720137,
+    "ece_10bin": 0.04813904474892826,
+    "calibration_axis_ece_part": 90.37219105021434,
+    "probability_items": 70,
+    "mean_tv_distance": 0.2889597615450036,
+    "fidelity_part": 71.10402384549964,
+    "calibration_axis_estimate": 80.73810744785699
+   },
+   "rotation": {
+    "questions": 586,
+    "accuracy": 0.6791808873720137,
+    "ece_10bin": 0.04813904474892826,
+    "calibration_axis_ece_part": 90.37219105021434,
+    "probability_items": 70,
+    "mean_tv_distance": 0.2889597615450036,
+    "fidelity_part": 71.10402384549964,
+    "calibration_axis_estimate": 80.73810744785699
+   }
+  },
+  "gen-v5-eval": {
+   "original": {
+    "questions": 738,
+    "accuracy": 0.7086720867208672,
+    "ece_10bin": 0.08227469775648809,
+    "calibration_axis_ece_part": 83.54506044870237
+   },
+   "rotation": {
+    "questions": 738,
+    "accuracy": 0.7086720867208672,
+    "ece_10bin": 0.08227469775648809,
+    "calibration_axis_ece_part": 83.54506044870237
    }
   }
  }
+}

evidence/calibration-leaderboard-merged-native.json CHANGED Viewed

@@ -1,207 +1,207 @@
 {
  "arm_fitted": "original",
  "inputs": [
-  "runs/merged-v11-3b-native/predictions.jsonl"
  ],
  "fit_suites": [
   "distprob-calib-v2",
   "internal-eval-hardproxy-calib-v1"
  ],
- "temperature": 1.35,
  "objective": "leaderboard",
- "axis_proxy_at_T1": 84.21389291588972,
- "axis_proxy_at_T": 88.67401841583384,
  "at_T1": {
   "jevbench-easy": {
    "original": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.0014843125501875498,
-    "calibration_axis_ece_part": 99.70313748996249
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.0014843125501875498,
-    "calibration_axis_ece_part": 99.70313748996249
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
-    "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.0228556343408844,
-    "calibration_axis_ece_part": 95.42887313182312
    },
    "rotation": {
     "questions": 72,
-    "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.0228556343408844,
-    "calibration_axis_ece_part": 95.42887313182312
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
-    "accuracy": 0.4864864864864865,
-    "ece_10bin": 0.26941446853676937,
-    "calibration_axis_ece_part": 46.11710629264613
    },
    "rotation": {
     "questions": 111,
-    "accuracy": 0.4864864864864865,
-    "ece_10bin": 0.26941446853676937,
-    "calibration_axis_ece_part": 46.11710629264613
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.89,
-    "ece_10bin": 0.021799163372850473,
-    "calibration_axis_ece_part": 95.6401673254299
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.89,
-    "ece_10bin": 0.021799163372850473,
-    "calibration_axis_ece_part": 95.6401673254299
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
-    "accuracy": 0.9,
-    "ece_10bin": 0.08024207896892373,
-    "calibration_axis_ece_part": 83.95158420621526,
     "probability_items": 120,
-    "mean_tv_distance": 0.10918668729159332,
-    "fidelity_part": 89.08133127084066,
-    "calibration_axis_estimate": 86.51645773852796
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.9,
-    "ece_10bin": 0.08024207896892373,
-    "calibration_axis_ece_part": 83.95158420621526,
     "probability_items": 120,
-    "mean_tv_distance": 0.10918668729159332,
-    "fidelity_part": 89.08133127084066,
-    "calibration_axis_estimate": 86.51645773852796
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
-    "accuracy": 0.8117760617760618,
-    "ece_10bin": 0.19675310407034366,
-    "calibration_axis_ece_part": 60.649379185931274,
     "probability_items": 1036,
-    "mean_tv_distance": 0.11023935512937952,
-    "fidelity_part": 88.97606448706205,
-    "calibration_axis_estimate": 74.81272183649666
    },
    "rotation": {
     "questions": 1036,
-    "accuracy": 0.8117760617760618,
-    "ece_10bin": 0.19675310407034366,
-    "calibration_axis_ece_part": 60.649379185931274,
     "probability_items": 1036,
-    "mean_tv_distance": 0.11023935512937952,
-    "fidelity_part": 88.97606448706205,
-    "calibration_axis_estimate": 74.81272183649666
    }
   },
   "internal-eval-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.7166666666666667,
-    "ece_10bin": 0.14650752255863214,
-    "calibration_axis_ece_part": 70.69849548827357
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.7166666666666667,
-    "ece_10bin": 0.14650752255863214,
-    "calibration_axis_ece_part": 70.69849548827357
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.435,
-    "ece_10bin": 0.11452618198956604,
-    "calibration_axis_ece_part": 77.09476360208679
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.435,
-    "ece_10bin": 0.11452618198956604,
-    "calibration_axis_ece_part": 77.09476360208679
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
-    "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.016584023903433262,
-    "calibration_axis_ece_part": 96.68319521931335
    },
    "rotation": {
     "questions": 360,
-    "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.016584023903433262,
-    "calibration_axis_ece_part": 96.68319521931335
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
-    "accuracy": 0.6833333333333333,
-    "ece_10bin": 0.09488361820723981,
-    "calibration_axis_ece_part": 81.02327635855204
    },
    "rotation": {
     "questions": 120,
-    "accuracy": 0.6833333333333333,
-    "ece_10bin": 0.09488361820723981,
-    "calibration_axis_ece_part": 81.02327635855204
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.4816666666666667,
-    "ece_10bin": 0.08163742104516354,
-    "calibration_axis_ece_part": 83.6725157909673
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.4816666666666667,
-    "ece_10bin": 0.08163742104516354,
-    "calibration_axis_ece_part": 83.6725157909673
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
-    "ece_10bin": 0.164477610622995,
-    "calibration_axis_ece_part": 67.10447787540099,
     "probability_items": 916,
-    "mean_tv_distance": 0.15244729959187872,
-    "fidelity_part": 84.75527004081212,
-    "calibration_axis_estimate": 75.92987395810655
    },
    "rotation": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
-    "ece_10bin": 0.164477610622995,
-    "calibration_axis_ece_part": 67.10447787540099,
     "probability_items": 916,
-    "mean_tv_distance": 0.15244729959187872,
-    "fidelity_part": 84.75527004081212,
-    "calibration_axis_estimate": 75.92987395810655
    }
   }
  },
@@ -210,192 +210,192 @@
    "original": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.006004891425301917,
-    "calibration_axis_ece_part": 98.79902171493961
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
-    "ece_10bin": 0.006004891425301917,
-    "calibration_axis_ece_part": 98.79902171493961
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
-    "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.043759725982928606,
-    "calibration_axis_ece_part": 91.24805480341428
    },
    "rotation": {
     "questions": 72,
-    "accuracy": 0.9305555555555556,
-    "ece_10bin": 0.043759725982928606,
-    "calibration_axis_ece_part": 91.24805480341428
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
-    "accuracy": 0.4864864864864865,
-    "ece_10bin": 0.21242280730976387,
-    "calibration_axis_ece_part": 57.51543853804723
    },
    "rotation": {
     "questions": 111,
-    "accuracy": 0.4864864864864865,
-    "ece_10bin": 0.21242280730976387,
-    "calibration_axis_ece_part": 57.51543853804723
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.89,
-    "ece_10bin": 0.029870314961169264,
-    "calibration_axis_ece_part": 94.02593700776615
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.89,
-    "ece_10bin": 0.029870314961169264,
-    "calibration_axis_ece_part": 94.02593700776615
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
-    "accuracy": 0.9,
-    "ece_10bin": 0.11506992601190205,
-    "calibration_axis_ece_part": 76.98601479761959,
     "probability_items": 120,
-    "mean_tv_distance": 0.12110827936168912,
-    "fidelity_part": 87.88917206383108,
-    "calibration_axis_estimate": 82.43759343072534
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.9,
-    "ece_10bin": 0.11506992601190205,
-    "calibration_axis_ece_part": 76.98601479761959,
     "probability_items": 120,
-    "mean_tv_distance": 0.12110827936168912,
-    "fidelity_part": 87.88917206383108,
-    "calibration_axis_estimate": 82.43759343072534
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
-    "accuracy": 0.8117760617760618,
-    "ece_10bin": 0.24602121170213875,
-    "calibration_axis_ece_part": 50.795757659572246,
     "probability_items": 1036,
-    "mean_tv_distance": 0.12238352239415004,
-    "fidelity_part": 87.76164776058499,
-    "calibration_axis_estimate": 69.27870271007862
    },
    "rotation": {
     "questions": 1036,
-    "accuracy": 0.8117760617760618,
-    "ece_10bin": 0.24602121170213875,
-    "calibration_axis_ece_part": 50.795757659572246,
     "probability_items": 1036,
-    "mean_tv_distance": 0.12238352239415004,
-    "fidelity_part": 87.76164776058499,
-    "calibration_axis_estimate": 69.27870271007862
    }
   },
   "internal-eval-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.7166666666666667,
-    "ece_10bin": 0.10387948891595138,
-    "calibration_axis_ece_part": 79.22410221680973
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.7166666666666667,
-    "ece_10bin": 0.10387948891595138,
-    "calibration_axis_ece_part": 79.22410221680973
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.435,
-    "ece_10bin": 0.06736918921772936,
-    "calibration_axis_ece_part": 86.52616215645412
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.435,
-    "ece_10bin": 0.06736918921772936,
-    "calibration_axis_ece_part": 86.52616215645412
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
-    "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.028917472692378383,
-    "calibration_axis_ece_part": 94.21650546152432
    },
    "rotation": {
     "questions": 360,
-    "accuracy": 0.9638888888888889,
-    "ece_10bin": 0.028917472692378383,
-    "calibration_axis_ece_part": 94.21650546152432
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
-    "accuracy": 0.6833333333333333,
-    "ece_10bin": 0.05340595065703371,
-    "calibration_axis_ece_part": 89.31880986859326
    },
    "rotation": {
     "questions": 120,
-    "accuracy": 0.6833333333333333,
-    "ece_10bin": 0.05340595065703371,
-    "calibration_axis_ece_part": 89.31880986859326
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
-    "accuracy": 0.4816666666666667,
-    "ece_10bin": 0.03032421514666685,
-    "calibration_axis_ece_part": 93.93515697066664
    },
    "rotation": {
     "questions": 600,
-    "accuracy": 0.4816666666666667,
-    "ece_10bin": 0.03032421514666685,
-    "calibration_axis_ece_part": 93.93515697066664
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
-    "ece_10bin": 0.21579301823334204,
-    "calibration_axis_ece_part": 56.84139635333159,
     "probability_items": 916,
-    "mean_tv_distance": 0.16587120138998968,
-    "fidelity_part": 83.41287986100103,
-    "calibration_axis_estimate": 70.1271381071663
    },
    "rotation": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
-    "ece_10bin": 0.21579301823334204,
-    "calibration_axis_ece_part": 56.84139635333159,
     "probability_items": 916,
-    "mean_tv_distance": 0.16587120138998968,
-    "fidelity_part": 83.41287986100103,
-    "calibration_axis_estimate": 70.1271381071663
    }
   }
  }

 {
  "arm_fitted": "original",
  "inputs": [
+  "runs/merged-v2r5-3b-native/predictions.jsonl"
  ],
  "fit_suites": [
   "distprob-calib-v2",
   "internal-eval-hardproxy-calib-v1"
  ],
+ "temperature": 1.05,
  "objective": "leaderboard",
+ "axis_proxy_at_T1": 86.2880616214533,
+ "axis_proxy_at_T": 87.36825835787238,
  "at_T1": {
   "jevbench-easy": {
    "original": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.04800839538567559,
+    "calibration_axis_ece_part": 90.39832092286488
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.04800839538567559,
+    "calibration_axis_ece_part": 90.39832092286488
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
+    "accuracy": 0.875,
+    "ece_10bin": 0.09771030785321377,
+    "calibration_axis_ece_part": 80.45793842935724
    },
    "rotation": {
     "questions": 72,
+    "accuracy": 0.875,
+    "ece_10bin": 0.09771030785321377,
+    "calibration_axis_ece_part": 80.45793842935724
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
+    "accuracy": 0.5045045045045045,
+    "ece_10bin": 0.2088674827913022,
+    "calibration_axis_ece_part": 58.22650344173957
    },
    "rotation": {
     "questions": 111,
+    "accuracy": 0.5045045045045045,
+    "ece_10bin": 0.2088674827913022,
+    "calibration_axis_ece_part": 58.22650344173957
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.9033333333333333,
+    "ece_10bin": 0.02388094113346994,
+    "calibration_axis_ece_part": 95.22381177330601
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.9033333333333333,
+    "ece_10bin": 0.02388094113346994,
+    "calibration_axis_ece_part": 95.22381177330601
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.08169286527671146,
+    "calibration_axis_ece_part": 83.6614269446577,
     "probability_items": 120,
+    "mean_tv_distance": 0.1351234243501839,
+    "fidelity_part": 86.48765756498162,
+    "calibration_axis_estimate": 85.07454225481966
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.08169286527671146,
+    "calibration_axis_ece_part": 83.6614269446577,
     "probability_items": 120,
+    "mean_tv_distance": 0.1351234243501839,
+    "fidelity_part": 86.48765756498162,
+    "calibration_axis_estimate": 85.07454225481966
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
+    "accuracy": 0.8127413127413128,
+    "ece_10bin": 0.2071254269264329,
+    "calibration_axis_ece_part": 58.57491461471342,
     "probability_items": 1036,
+    "mean_tv_distance": 0.11286277320639246,
+    "fidelity_part": 88.71372267936076,
+    "calibration_axis_estimate": 73.6443186470371
    },
    "rotation": {
     "questions": 1036,
+    "accuracy": 0.8127413127413128,
+    "ece_10bin": 0.2071254269264329,
+    "calibration_axis_ece_part": 58.57491461471342,
     "probability_items": 1036,
+    "mean_tv_distance": 0.11286277320639246,
+    "fidelity_part": 88.71372267936076,
+    "calibration_axis_estimate": 73.6443186470371
    }
   },
   "internal-eval-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.7733333333333333,
+    "ece_10bin": 0.08928708761683869,
+    "calibration_axis_ece_part": 82.14258247663227
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.7733333333333333,
+    "ece_10bin": 0.08928708761683869,
+    "calibration_axis_ece_part": 82.14258247663227
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.44166666666666665,
+    "ece_10bin": 0.05371810426360786,
+    "calibration_axis_ece_part": 89.25637914727844
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.44166666666666665,
+    "ece_10bin": 0.05371810426360786,
+    "calibration_axis_ece_part": 89.25637914727844
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
+    "accuracy": 0.9555555555555556,
+    "ece_10bin": 0.029217282367707872,
+    "calibration_axis_ece_part": 94.15654352645842
    },
    "rotation": {
     "questions": 360,
+    "accuracy": 0.9555555555555556,
+    "ece_10bin": 0.029217282367707872,
+    "calibration_axis_ece_part": 94.15654352645842
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.0726245848844734,
+    "calibration_axis_ece_part": 85.47508302310531
    },
    "rotation": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.0726245848844734,
+    "calibration_axis_ece_part": 85.47508302310531
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.48,
+    "ece_10bin": 0.057592540160655105,
+    "calibration_axis_ece_part": 88.48149196786898
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.48,
+    "ece_10bin": 0.057592540160655105,
+    "calibration_axis_ece_part": 88.48149196786898
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
+    "ece_10bin": 0.17627768052696383,
+    "calibration_axis_ece_part": 64.74446389460724,
     "probability_items": 916,
+    "mean_tv_distance": 0.1590536872496238,
+    "fidelity_part": 84.09463127503763,
+    "calibration_axis_estimate": 74.41954758482242
    },
    "rotation": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
+    "ece_10bin": 0.17627768052696383,
+    "calibration_axis_ece_part": 64.74446389460724,
     "probability_items": 916,
+    "mean_tv_distance": 0.1590536872496238,
+    "fidelity_part": 84.09463127503763,
+    "calibration_axis_estimate": 74.41954758482242
    }
   }
  },
    "original": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.05532780811667773,
+    "calibration_axis_ece_part": 88.93443837666446
    },
    "rotation": {
     "questions": 48,
     "accuracy": 1.0,
+    "ece_10bin": 0.05532780811667773,
+    "calibration_axis_ece_part": 88.93443837666446
    }
   },
   "jevbench-original": {
    "original": {
     "questions": 72,
+    "accuracy": 0.875,
+    "ece_10bin": 0.09604732483334082,
+    "calibration_axis_ece_part": 80.79053503333184
    },
    "rotation": {
     "questions": 72,
+    "accuracy": 0.875,
+    "ece_10bin": 0.09604732483334082,
+    "calibration_axis_ece_part": 80.79053503333184
    }
   },
   "jevbench-hard": {
    "original": {
     "questions": 111,
+    "accuracy": 0.5045045045045045,
+    "ece_10bin": 0.2094997360066108,
+    "calibration_axis_ece_part": 58.10005279867785
    },
    "rotation": {
     "questions": 111,
+    "accuracy": 0.5045045045045045,
+    "ece_10bin": 0.2094997360066108,
+    "calibration_axis_ece_part": 58.10005279867785
    }
   },
   "judge-proxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.9033333333333333,
+    "ece_10bin": 0.023621370112711884,
+    "calibration_axis_ece_part": 95.27572597745763
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.9033333333333333,
+    "ece_10bin": 0.023621370112711884,
+    "calibration_axis_ece_part": 95.27572597745763
    }
   },
   "internal-eval-v2-realistic": {
    "original": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.08832038841348458,
+    "calibration_axis_ece_part": 82.33592231730309,
     "probability_items": 120,
+    "mean_tv_distance": 0.13696264167191877,
+    "fidelity_part": 86.30373583280813,
+    "calibration_axis_estimate": 84.31982907505561
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.885,
+    "ece_10bin": 0.08832038841348458,
+    "calibration_axis_ece_part": 82.33592231730309,
     "probability_items": 120,
+    "mean_tv_distance": 0.13696264167191877,
+    "fidelity_part": 86.30373583280813,
+    "calibration_axis_estimate": 84.31982907505561
    }
   },
   "distprob-val-v2": {
    "original": {
     "questions": 1036,
+    "accuracy": 0.8127413127413128,
+    "ece_10bin": 0.21558434490288425,
+    "calibration_axis_ece_part": 56.883131019423146,
     "probability_items": 1036,
+    "mean_tv_distance": 0.11417286394255613,
+    "fidelity_part": 88.58271360574439,
+    "calibration_axis_estimate": 72.73292231258377
    },
    "rotation": {
     "questions": 1036,
+    "accuracy": 0.8127413127413128,
+    "ece_10bin": 0.21558434490288425,
+    "calibration_axis_ece_part": 56.883131019423146,
     "probability_items": 1036,
+    "mean_tv_distance": 0.11417286394255613,
+    "fidelity_part": 88.58271360574439,
+    "calibration_axis_estimate": 72.73292231258377
    }
   },
   "internal-eval-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.7733333333333333,
+    "ece_10bin": 0.09682354890222471,
+    "calibration_axis_ece_part": 80.63529021955506
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.7733333333333333,
+    "ece_10bin": 0.09682354890222471,
+    "calibration_axis_ece_part": 80.63529021955506
    }
   },
   "internal-eval-hardproxy-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.44166666666666665,
+    "ece_10bin": 0.04391492758917536,
+    "calibration_axis_ece_part": 91.21701448216493
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.44166666666666665,
+    "ece_10bin": 0.04391492758917536,
+    "calibration_axis_ece_part": 91.21701448216493
    }
   },
   "games-harness-eval-v1": {
    "original": {
     "questions": 360,
+    "accuracy": 0.9555555555555556,
+    "ece_10bin": 0.03115401530534188,
+    "calibration_axis_ece_part": 93.76919693893161
    },
    "rotation": {
     "questions": 360,
+    "accuracy": 0.9555555555555556,
+    "ece_10bin": 0.03115401530534188,
+    "calibration_axis_ece_part": 93.76919693893161
    }
   },
   "tetris-drought-eval-v1": {
    "original": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.08824054458176568,
+    "calibration_axis_ece_part": 82.35189108364686
    },
    "rotation": {
     "questions": 120,
+    "accuracy": 0.7,
+    "ece_10bin": 0.08824054458176568,
+    "calibration_axis_ece_part": 82.35189108364686
    }
   },
   "internal-eval-hardproxy-calib-v1": {
    "original": {
     "questions": 600,
+    "accuracy": 0.48,
+    "ece_10bin": 0.04594419114837251,
+    "calibration_axis_ece_part": 90.8111617703255
    },
    "rotation": {
     "questions": 600,
+    "accuracy": 0.48,
+    "ece_10bin": 0.04594419114837251,
+    "calibration_axis_ece_part": 90.8111617703255
    }
   },
   "distprob-calib-v2": {
    "original": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
+    "ece_10bin": 0.1847901784239128,
+    "calibration_axis_ece_part": 63.04196431521745,
     "probability_items": 916,
+    "mean_tv_distance": 0.16074645054580725,
+    "fidelity_part": 83.92535494541927,
+    "calibration_axis_estimate": 73.48365963031836
    },
    "rotation": {
     "questions": 916,
     "accuracy": 0.7838427947598253,
+    "ece_10bin": 0.1847901784239128,
+    "calibration_axis_ece_part": 63.04196431521745,
     "probability_items": 916,
+    "mean_tv_distance": 0.16074645054580725,
+    "fidelity_part": 83.92535494541927,
+    "calibration_axis_estimate": 73.48365963031836
    }
   }
  }

evidence/calibration-per-type-served.json ADDED Viewed

	@@ -0,0 +1,537 @@

+{
+ "arm_fitted": "original",
+ "inputs": [
+  "runs/served-v2r5-3b-native/predictions.jsonl"
+ ],
+ "fit_suites": [
+  "calib-public-v1",
+  "distprob-calib-v2",
+  "internal-eval-hardproxy-calib-v1"
+ ],
+ "temperature": {
+  "*": 0.95,
+  "choice": 0.95,
+  "noul": 1.05,
+  "score": 0.8500000000000001
+ },
+ "per_type": {
+  "*": {
+   "temperature": 0.95,
+   "axis_proxy_at_T": 88.69418853137557
+  },
+  "choice": {
+   "fit_items": 1648,
+   "temperature": 0.95
+  },
+  "noul": {
+   "fit_items": 782,
+   "temperature": 1.05
+  },
+  "score": {
+   "fit_items": 286,
+   "temperature": 0.8500000000000001
+  },
+  "combined_axis_proxy": 88.97103816972816
+ },
+ "objective": "leaderboard",
+ "at_T1": {
+  "jevbench-easy": {
+   "original": {
+    "questions": 48,
+    "accuracy": 1.0,
+    "ece_10bin": 0.04932844718295749,
+    "calibration_axis_ece_part": 90.1343105634085
+   },
+   "rotation": {
+    "questions": 48,
+    "accuracy": 1.0,
+    "ece_10bin": 0.04932844718295749,
+    "calibration_axis_ece_part": 90.1343105634085
+   }
+  },
+  "jevbench-original": {
+   "original": {
+    "questions": 72,
+    "accuracy": 0.8888888888888888,
+    "ece_10bin": 0.09455451547806448,
+    "calibration_axis_ece_part": 81.0890969043871
+   },
+   "rotation": {
+    "questions": 72,
+    "accuracy": 0.8888888888888888,
+    "ece_10bin": 0.09455451547806448,
+    "calibration_axis_ece_part": 81.0890969043871
+   }
+  },
+  "jevbench-hard": {
+   "original": {
+    "questions": 111,
+    "accuracy": 0.46846846846846846,
+    "ece_10bin": 0.2184913569926046,
+    "calibration_axis_ece_part": 56.301728601479084
+   },
+   "rotation": {
+    "questions": 111,
+    "accuracy": 0.46846846846846846,
+    "ece_10bin": 0.2184913569926046,
+    "calibration_axis_ece_part": 56.301728601479084
+   }
+  },
+  "judge-proxy-v1": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.8816666666666667,
+    "ece_10bin": 0.030726866457745777,
+    "calibration_axis_ece_part": 93.85462670845084
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.8816666666666667,
+    "ece_10bin": 0.030726866457745777,
+    "calibration_axis_ece_part": 93.85462670845084
+   }
+  },
+  "internal-eval-hardproxy-v1": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.44666666666666666,
+    "ece_10bin": 0.05193867154197662,
+    "calibration_axis_ece_part": 89.61226569160468
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.44666666666666666,
+    "ece_10bin": 0.05193867154197662,
+    "calibration_axis_ece_part": 89.61226569160468
+   }
+  },
+  "internal-eval-hardproxy-calib-v1": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.4683333333333333,
+    "ece_10bin": 0.05091091372261677,
+    "calibration_axis_ece_part": 89.81781725547665
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.4683333333333333,
+    "ece_10bin": 0.05091091372261677,
+    "calibration_axis_ece_part": 89.81781725547665
+   }
+  },
+  "distprob-calib-v2": {
+   "original": {
+    "questions": 916,
+    "accuracy": 0.759825327510917,
+    "ece_10bin": 0.15548108916652895,
+    "calibration_axis_ece_part": 68.90378216669421,
+    "probability_items": 916,
+    "mean_tv_distance": 0.16417326306216248,
+    "fidelity_part": 83.58267369378375,
+    "calibration_axis_estimate": 76.24322793023899
+   },
+   "rotation": {
+    "questions": 916,
+    "accuracy": 0.759825327510917,
+    "ece_10bin": 0.15548108916652895,
+    "calibration_axis_ece_part": 68.90378216669421,
+    "probability_items": 916,
+    "mean_tv_distance": 0.16417326306216248,
+    "fidelity_part": 83.58267369378375,
+    "calibration_axis_estimate": 76.24322793023899
+   }
+  },
+  "distprob-val-v2": {
+   "original": {
+    "questions": 1036,
+    "accuracy": 0.7876447876447876,
+    "ece_10bin": 0.18697963114403407,
+    "calibration_axis_ece_part": 62.60407377119319,
+    "probability_items": 1036,
+    "mean_tv_distance": 0.11523235926701762,
+    "fidelity_part": 88.47676407329824,
+    "calibration_axis_estimate": 75.54041892224572
+   },
+   "rotation": {
+    "questions": 1036,
+    "accuracy": 0.7876447876447876,
+    "ece_10bin": 0.18697963114403407,
+    "calibration_axis_ece_part": 62.60407377119319,
+    "probability_items": 1036,
+    "mean_tv_distance": 0.11523235926701762,
+    "fidelity_part": 88.47676407329824,
+    "calibration_axis_estimate": 75.54041892224572
+   }
+  },
+  "internal-eval-v2-realistic": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.9066666666666666,
+    "ece_10bin": 0.10114578467993941,
+    "calibration_axis_ece_part": 79.77084306401213,
+    "probability_items": 120,
+    "mean_tv_distance": 0.12265229390399424,
+    "fidelity_part": 87.73477060960057,
+    "calibration_axis_estimate": 83.75280683680634
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.9066666666666666,
+    "ece_10bin": 0.10114578467993941,
+    "calibration_axis_ece_part": 79.77084306401213,
+    "probability_items": 120,
+    "mean_tv_distance": 0.12265229390399424,
+    "fidelity_part": 87.73477060960057,
+    "calibration_axis_estimate": 83.75280683680634
+   }
+  },
+  "musique-mc": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.706,
+    "ece_10bin": 0.17993847577848524,
+    "calibration_axis_ece_part": 64.01230484430296
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.706,
+    "ece_10bin": 0.17993847577848524,
+    "calibration_axis_ece_part": 64.01230484430296
+   }
+  },
+  "squad2-abstain": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.822,
+    "ece_10bin": 0.11357821370411919,
+    "calibration_axis_ece_part": 77.28435725917616
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.822,
+    "ece_10bin": 0.11357821370411919,
+    "calibration_axis_ece_part": 77.28435725917616
+   }
+  },
+  "contractnli": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.59,
+    "ece_10bin": 0.09726769731259255,
+    "calibration_axis_ece_part": 80.5464605374815
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.59,
+    "ece_10bin": 0.09726769731259255,
+    "calibration_axis_ece_part": 80.5464605374815
+   }
+  },
+  "paws-x-en": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.724,
+    "ece_10bin": 0.03678150199999996,
+    "calibration_axis_ece_part": 92.6436996
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.724,
+    "ece_10bin": 0.03678150199999996,
+    "calibration_axis_ece_part": 92.6436996
+   }
+  },
+  "hard-eval-v4": {
+   "original": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.08417544291841819,
+    "calibration_axis_ece_part": 83.16491141631637
+   },
+   "rotation": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.08417544291841819,
+    "calibration_axis_ece_part": 83.16491141631637
+   }
+  },
+  "consistency-eval-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.6083333333333333,
+    "ece_10bin": 0.057791216912671715,
+    "calibration_axis_ece_part": 88.44175661746566
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.6083333333333333,
+    "ece_10bin": 0.057791216912671715,
+    "calibration_axis_ece_part": 88.44175661746566
+   }
+  },
+  "calib-public-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.7091666666666666,
+    "ece_10bin": 0.052337331262089004,
+    "calibration_axis_ece_part": 89.53253374758219
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.7091666666666666,
+    "ece_10bin": 0.052337331262089004,
+    "calibration_axis_ece_part": 89.53253374758219
+   }
+  }
+ },
+ "at_T": {
+  "jevbench-easy": {
+   "original": {
+    "questions": 48,
+    "accuracy": 1.0,
+    "ece_10bin": 0.04419476052413752,
+    "calibration_axis_ece_part": 91.1610478951725
+   },
+   "rotation": {
+    "questions": 48,
+    "accuracy": 1.0,
+    "ece_10bin": 0.04419476052413752,
+    "calibration_axis_ece_part": 91.1610478951725
+   }
+  },
+  "jevbench-original": {
+   "original": {
+    "questions": 72,
+    "accuracy": 0.8888888888888888,
+    "ece_10bin": 0.07585349079105076,
+    "calibration_axis_ece_part": 84.82930184178986
+   },
+   "rotation": {
+    "questions": 72,
+    "accuracy": 0.8888888888888888,
+    "ece_10bin": 0.07585349079105076,
+    "calibration_axis_ece_part": 84.82930184178986
+   }
+  },
+  "jevbench-hard": {
+   "original": {
+    "questions": 111,
+    "accuracy": 0.46846846846846846,
+    "ece_10bin": 0.2247919929190559,
+    "calibration_axis_ece_part": 55.04160141618881
+   },
+   "rotation": {
+    "questions": 111,
+    "accuracy": 0.46846846846846846,
+    "ece_10bin": 0.2247919929190559,
+    "calibration_axis_ece_part": 55.04160141618881
+   }
+  },
+  "judge-proxy-v1": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.8816666666666667,
+    "ece_10bin": 0.029894001180385665,
+    "calibration_axis_ece_part": 94.02119976392287
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.8816666666666667,
+    "ece_10bin": 0.029894001180385665,
+    "calibration_axis_ece_part": 94.02119976392287
+   }
+  },
+  "internal-eval-hardproxy-v1": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.44666666666666666,
+    "ece_10bin": 0.05741751892694225,
+    "calibration_axis_ece_part": 88.51649621461155
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.44666666666666666,
+    "ece_10bin": 0.05741751892694225,
+    "calibration_axis_ece_part": 88.51649621461155
+   }
+  },
+  "internal-eval-hardproxy-calib-v1": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.4683333333333333,
+    "ece_10bin": 0.05585026958796656,
+    "calibration_axis_ece_part": 88.82994608240668
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.4683333333333333,
+    "ece_10bin": 0.05585026958796656,
+    "calibration_axis_ece_part": 88.82994608240668
+   }
+  },
+  "distprob-calib-v2": {
+   "original": {
+    "questions": 916,
+    "accuracy": 0.759825327510917,
+    "ece_10bin": 0.14766935198566422,
+    "calibration_axis_ece_part": 70.46612960286716,
+    "probability_items": 916,
+    "mean_tv_distance": 0.16455869427124392,
+    "fidelity_part": 83.5441305728756,
+    "calibration_axis_estimate": 77.00513008787138
+   },
+   "rotation": {
+    "questions": 916,
+    "accuracy": 0.759825327510917,
+    "ece_10bin": 0.14766935198566422,
+    "calibration_axis_ece_part": 70.46612960286716,
+    "probability_items": 916,
+    "mean_tv_distance": 0.16455869427124392,
+    "fidelity_part": 83.5441305728756,
+    "calibration_axis_estimate": 77.00513008787138
+   }
+  },
+  "distprob-val-v2": {
+   "original": {
+    "questions": 1036,
+    "accuracy": 0.7876447876447876,
+    "ece_10bin": 0.17849139832630306,
+    "calibration_axis_ece_part": 64.30172033473939,
+    "probability_items": 1036,
+    "mean_tv_distance": 0.11650143104460119,
+    "fidelity_part": 88.34985689553989,
+    "calibration_axis_estimate": 76.32578861513963
+   },
+   "rotation": {
+    "questions": 1036,
+    "accuracy": 0.7876447876447876,
+    "ece_10bin": 0.17849139832630306,
+    "calibration_axis_ece_part": 64.30172033473939,
+    "probability_items": 1036,
+    "mean_tv_distance": 0.11650143104460119,
+    "fidelity_part": 88.34985689553989,
+    "calibration_axis_estimate": 76.32578861513963
+   }
+  },
+  "internal-eval-v2-realistic": {
+   "original": {
+    "questions": 600,
+    "accuracy": 0.9066666666666666,
+    "ece_10bin": 0.09839693431153797,
+    "calibration_axis_ece_part": 80.3206131376924,
+    "probability_items": 120,
+    "mean_tv_distance": 0.12166688104280965,
+    "fidelity_part": 87.83331189571904,
+    "calibration_axis_estimate": 84.07696251670572
+   },
+   "rotation": {
+    "questions": 600,
+    "accuracy": 0.9066666666666666,
+    "ece_10bin": 0.09839693431153797,
+    "calibration_axis_ece_part": 80.3206131376924,
+    "probability_items": 120,
+    "mean_tv_distance": 0.12166688104280965,
+    "fidelity_part": 87.83331189571904,
+    "calibration_axis_estimate": 84.07696251670572
+   }
+  },
+  "musique-mc": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.706,
+    "ece_10bin": 0.1647126682714215,
+    "calibration_axis_ece_part": 67.0574663457157
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.706,
+    "ece_10bin": 0.1647126682714215,
+    "calibration_axis_ece_part": 67.0574663457157
+   }
+  },
+  "squad2-abstain": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.822,
+    "ece_10bin": 0.09989379871731917,
+    "calibration_axis_ece_part": 80.02124025653616
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.822,
+    "ece_10bin": 0.09989379871731917,
+    "calibration_axis_ece_part": 80.02124025653616
+   }
+  },
+  "contractnli": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.59,
+    "ece_10bin": 0.09822783253025427,
+    "calibration_axis_ece_part": 80.35443349394914
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.59,
+    "ece_10bin": 0.09822783253025427,
+    "calibration_axis_ece_part": 80.35443349394914
+   }
+  },
+  "paws-x-en": {
+   "original": {
+    "questions": 500,
+    "accuracy": 0.724,
+    "ece_10bin": 0.03758326507352331,
+    "calibration_axis_ece_part": 92.48334698529533
+   },
+   "rotation": {
+    "questions": 500,
+    "accuracy": 0.724,
+    "ece_10bin": 0.03758326507352331,
+    "calibration_axis_ece_part": 92.48334698529533
+   }
+  },
+  "hard-eval-v4": {
+   "original": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.0970318063010429,
+    "calibration_axis_ece_part": 80.59363873979142
+   },
+   "rotation": {
+    "questions": 800,
+    "accuracy": 0.51875,
+    "ece_10bin": 0.0970318063010429,
+    "calibration_axis_ece_part": 80.59363873979142
+   }
+  },
+  "consistency-eval-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.6083333333333333,
+    "ece_10bin": 0.07001837482030217,
+    "calibration_axis_ece_part": 85.99632503593956
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.6083333333333333,
+    "ece_10bin": 0.07001837482030217,
+    "calibration_axis_ece_part": 85.99632503593956
+   }
+  },
+  "calib-public-v1": {
+   "original": {
+    "questions": 1200,
+    "accuracy": 0.7091666666666666,
+    "ece_10bin": 0.0470134244447497,
+    "calibration_axis_ece_part": 90.59731511105007
+   },
+   "rotation": {
+    "questions": 1200,
+    "accuracy": 0.7091666666666666,
+    "ece_10bin": 0.0470134244447497,
+    "calibration_axis_ece_part": 90.59731511105007
+   }
+  }
+ }
+}

evidence/prompt-wording-choice.json ADDED Viewed

	@@ -0,0 +1,81 @@

+{
+ "method": "decision on the unweighted mean accuracy over choice_suites (served predictions): native unless served leads by >= 1.0 pp and the offline cross-check agrees; report_only suites play no part",
+ "choice_suites": [
+  "judge-proxy-v1",
+  "internal-eval-hardproxy-v1",
+  "distprob-val-v2",
+  "internal-eval-v2-realistic",
+  "musique-mc",
+  "squad2-abstain",
+  "contractnli",
+  "paws-x-en",
+  "hard-eval-v4",
+  "consistency-eval-v1"
+ ],
+ "report_only_suites": [
+  "jevbench-easy",
+  "jevbench-original",
+  "jevbench-hard"
+ ],
+ "size": "3B",
+ "basis": "served predictions (SGLang 0.5.20 + jev-adapter, no system prompt, T=1, one option order)",
+ "arms": {
+  "native": {
+   "wording": "native (jev-adapter --prompt-wording native --native-system-prompt none)",
+   "served_choice_suites": {
+    "judge-proxy-v1": 88.17,
+    "internal-eval-hardproxy-v1": 44.67,
+    "distprob-val-v2": 78.76,
+    "internal-eval-v2-realistic": 90.67,
+    "musique-mc": 70.6,
+    "squad2-abstain": 82.2,
+    "contractnli": 59.0,
+    "paws-x-en": 72.4,
+    "hard-eval-v4": 51.88,
+    "consistency-eval-v1": 60.83
+   },
+   "served_mean": 69.918,
+   "report_only_jevbench_served": {
+    "jevbench-easy": 100.0,
+    "jevbench-original": 88.89,
+    "jevbench-hard": 46.85
+   },
+   "per_type_T": {
+    "choice": 0.95,
+    "noul": 1.05,
+    "score": 0.85
+   }
+  },
+  "served": {
+   "wording": "served (jev-adapter --prompt-wording served)",
+   "served_choice_suites": {
+    "judge-proxy-v1": 88.5,
+    "internal-eval-hardproxy-v1": 43.5,
+    "distprob-val-v2": 80.31,
+    "internal-eval-v2-realistic": 91.0,
+    "musique-mc": 68.0,
+    "squad2-abstain": 81.2,
+    "contractnli": 59.2,
+    "paws-x-en": 73.0,
+    "hard-eval-v4": 52.0,
+    "consistency-eval-v1": 61.58
+   },
+   "served_mean": 69.829,
+   "report_only_jevbench_served": {
+    "jevbench-easy": 97.92,
+    "jevbench-original": 93.06,
+    "jevbench-hard": 45.95
+   },
+   "per_type_T": {
+    "choice": 0.9,
+    "noul": 1.05,
+    "score": 0.95
+   }
+  }
+ },
+ "chosen": "native",
+ "chosen_wording": "native (jev-adapter --prompt-wording native --native-system-prompt none)",
+ "margin_pp": 0.089,
+ "rule": "served only if its served mean leads native by >= 1.0 pp and the offline check agrees; otherwise native",
+ "served_lead_pp": -0.089
+}

evidence/served-nosys-easy-report.json CHANGED Viewed

@@ -8,22 +8,22 @@
   "served_models": [
     "decision-model"
   ],
-  "elapsed_measurement_s": 1.8050968870520592,
-  "requests_per_second": 26.5913704379546,
-  "questions_per_second": 26.5913704379546,
   "latency_ms": {
     "n": 48,
-    "mean": 37.49669147994913,
-    "p50": 37.14705549646169,
-    "p95": 37.637978242128156,
-    "p99": 49.57394877215848
   },
   "adapter_ms": {
     "n": 48,
-    "mean": 36.123791850210786,
-    "p50": 35.78712046146393,
-    "p95": 36.14921441185288,
-    "p99": 48.11713631264866
   },
   "input_tokens_per_request": {
     "n": 48,
@@ -37,13 +37,13 @@
       "clean": {
         "n": 48,
         "acc": 1.0,
-        "nll": 0.0015276622415421612,
-        "brier": 5.584960223915265e-05,
-        "ece": 0.0015118869298550441,
-        "mean_conf": 0.998488113070145,
-        "confidence_bias": -0.0015118869298550441,
         "confident_error_rate": 0.0,
-        "coverage_at_0_9": 1.0,
         "accuracy_at_0_9": 1.0,
         "coverage_at_5pct_error": 1.0,
         "coverage_at_1pct_error": 1.0
@@ -52,13 +52,13 @@
         "extraction": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.002813571161406935,
-          "brier": 7.836254226801523e-05,
-          "ece": 0.0027875193213375837,
-          "mean_conf": 0.9972124806786624,
-          "confidence_bias": -0.0027875193213375837,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -66,11 +66,11 @@
         "fact": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.0029452787672731777,
-          "brier": 0.0001444467258469531,
-          "ece": 0.0029084513057315853,
-          "mean_conf": 0.9970915486942684,
-          "confidence_bias": -0.0029084513057315853,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
@@ -80,13 +80,13 @@
         "intent": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.00026626090898396263,
-          "brier": 5.533060241933733e-07,
-          "ece": 0.0002660503696976235,
-          "mean_conf": 0.9997339496303024,
-          "confidence_bias": -0.0002660503696976235,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -94,11 +94,11 @@
         "tool_selection": {
           "n": 12,
           "acc": 1.0,
-          "nll": 8.553812850457014e-05,
-          "brier": 3.5834817448884555e-08,
-          "ece": 8.552672265349504e-05,
-          "mean_conf": 0.9999144732773465,
-          "confidence_bias": -8.552672265349504e-05,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
@@ -110,13 +110,13 @@
         "extraction": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.002813571161406935,
-          "brier": 7.836254226801523e-05,
-          "ece": 0.0027875193213375837,
-          "mean_conf": 0.9972124806786624,
-          "confidence_bias": -0.0027875193213375837,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -124,11 +124,11 @@
         "fact": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.0029452787672731777,
-          "brier": 0.0001444467258469531,
-          "ece": 0.0029084513057315853,
-          "mean_conf": 0.9970915486942684,
-          "confidence_bias": -0.0029084513057315853,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
@@ -138,13 +138,13 @@
         "intent": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.00026626090898396263,
-          "brier": 5.533060241933733e-07,
-          "ece": 0.0002660503696976235,
-          "mean_conf": 0.9997339496303024,
-          "confidence_bias": -0.0002660503696976235,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -152,11 +152,11 @@
         "tool_selection": {
           "n": 12,
           "acc": 1.0,
-          "nll": 8.553812850457014e-05,
-          "brier": 3.5834817448884555e-08,
-          "ece": 8.552672265349504e-05,
-          "mean_conf": 0.9999144732773465,
-          "confidence_bias": -8.552672265349504e-05,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
@@ -168,20 +168,20 @@
         "clean": {
           "n": 48,
           "acc": 1.0,
-          "nll": 0.0015276622415421612,
-          "brier": 5.584960223915265e-05,
-          "ece": 0.0015118869298550441,
-          "mean_conf": 0.998488113070145,
-          "confidence_bias": -0.0015118869298550441,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         }
       },
       "macro_task_acc": 1.0,
-      "objective": -0.0015276622415421614,
       "paired_flip": {
         "pairs": 0,
         "incomplete_pairs": 0,
@@ -211,10 +211,10 @@
       },
       "latency_ms": {
         "n": 48,
-        "mean": 37.49669147994913,
-        "p50": 37.14705549646169,
-        "p95": 37.637978242128156,
-        "p99": 49.57394877215848
       }
     }
   },

   "served_models": [
     "decision-model"
   ],
+  "elapsed_measurement_s": 1.488872062007431,
+  "requests_per_second": 32.23917032554301,
+  "questions_per_second": 32.23917032554301,
   "latency_ms": {
     "n": 48,
+    "mean": 30.909886918379925,
+    "p50": 23.774297995259985,
+    "p95": 50.51140714203939,
+    "p99": 55.254196919850074
   },
   "adapter_ms": {
     "n": 48,
+    "mean": 29.532985641708365,
+    "p50": 22.54483348224312,
+    "p95": 49.124720809049904,
+    "p99": 53.83548503741622
   },
   "input_tokens_per_request": {
     "n": 48,
       "clean": {
         "n": 48,
         "acc": 1.0,
+        "nll": 0.053192280230641535,
+        "brier": 0.009221265547225526,
+        "ece": 0.048823191479344843,
+        "mean_conf": 0.9511768085206551,
+        "confidence_bias": -0.048823191479344885,
         "confident_error_rate": 0.0,
+        "coverage_at_0_9": 0.9583333333333334,
         "accuracy_at_0_9": 1.0,
         "coverage_at_5pct_error": 1.0,
         "coverage_at_1pct_error": 1.0
         "extraction": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.12001306418747816,
+          "brier": 0.02956933344542464,
+          "ece": 0.10511608947448611,
+          "mean_conf": 0.8948839105255139,
+          "confidence_bias": -0.10511608947448614,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.9166666666666666,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "fact": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.028728554729256454,
+          "brier": 0.0028036707065585732,
+          "ece": 0.02799237777415231,
+          "mean_conf": 0.9720076222258477,
+          "confidence_bias": -0.02799237777415231,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
         "intent": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.03664962443989921,
+          "brier": 0.003184709744442926,
+          "ece": 0.03530644043694742,
+          "mean_conf": 0.9646935595630527,
+          "confidence_bias": -0.035306440436947306,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.9166666666666666,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "tool_selection": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.027377877565932313,
+          "brier": 0.0013273482924759663,
+          "ece": 0.02687785823179356,
+          "mean_conf": 0.9731221417682064,
+          "confidence_bias": -0.02687785823179356,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
         "extraction": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.12001306418747816,
+          "brier": 0.02956933344542464,
+          "ece": 0.10511608947448611,
+          "mean_conf": 0.8948839105255139,
+          "confidence_bias": -0.10511608947448614,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.9166666666666666,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "fact": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.028728554729256454,
+          "brier": 0.0028036707065585732,
+          "ece": 0.02799237777415231,
+          "mean_conf": 0.9720076222258477,
+          "confidence_bias": -0.02799237777415231,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
         "intent": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.03664962443989921,
+          "brier": 0.003184709744442926,
+          "ece": 0.03530644043694742,
+          "mean_conf": 0.9646935595630527,
+          "confidence_bias": -0.035306440436947306,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.9166666666666666,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "tool_selection": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.027377877565932313,
+          "brier": 0.0013273482924759663,
+          "ece": 0.02687785823179356,
+          "mean_conf": 0.9731221417682064,
+          "confidence_bias": -0.02687785823179356,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
         "clean": {
           "n": 48,
           "acc": 1.0,
+          "nll": 0.053192280230641535,
+          "brier": 0.009221265547225526,
+          "ece": 0.048823191479344843,
+          "mean_conf": 0.9511768085206551,
+          "confidence_bias": -0.048823191479344885,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.9583333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         }
       },
       "macro_task_acc": 1.0,
+      "objective": -0.053192280230641535,
       "paired_flip": {
         "pairs": 0,
         "incomplete_pairs": 0,
       },
       "latency_ms": {
         "n": 48,
+        "mean": 30.909886918379925,
+        "p50": 23.774297995259985,
+        "p95": 50.51140714203939,
+        "p99": 55.254196919850074
       }
     }
   },

evidence/served-nosys-hard-report.json CHANGED Viewed

@@ -8,22 +8,22 @@
   "served_models": [
     "decision-model"
   ],
-  "elapsed_measurement_s": 6.400594180973712,
-  "requests_per_second": 17.342139942250448,
-  "questions_per_second": 17.342139942250448,
   "latency_ms": {
     "n": 111,
-    "mean": 57.53704737808958,
-    "p50": 41.14527994534001,
-    "p95": 107.0126669947058,
-    "p99": 121.36280137347059
   },
   "adapter_ms": {
     "n": 111,
-    "mean": 55.97996988348445,
-    "p50": 39.66773202409968,
-    "p95": 105.28535101911984,
-    "p99": 119.4705346308183
   },
   "input_tokens_per_request": {
     "n": 111,
@@ -36,45 +36,45 @@
     "jevbench-hard/public": {
       "clean": {
         "n": 111,
-        "acc": 0.4594594594594595,
-        "nll": 1.2962948856809682,
-        "brier": 0.6968902514322239,
-        "ece": 0.28535210981983,
-        "mean_conf": 0.7361970214660999,
-        "confidence_bias": 0.2767375620066404,
-        "confident_error_rate": 0.10810810810810811,
-        "coverage_at_0_9": 0.36036036036036034,
-        "accuracy_at_0_9": 0.7,
-        "coverage_at_5pct_error": 0.0990990990990991,
-        "coverage_at_1pct_error": 0.0990990990990991,
-        "score_mae": 0.6876357781329778,
-        "ranked_probability_score": 0.19267980315847502
       },
       "tasks": {
         "adversarial": {
           "n": 6,
-          "acc": 0.6666666666666666,
-          "nll": 0.3804766699344641,
-          "brier": 0.24696365952999444,
-          "ece": 0.10635190756625662,
-          "mean_conf": 0.7502708058819635,
-          "confidence_bias": 0.08360413921529686,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.5,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.6666666666666666,
           "coverage_at_1pct_error": 0.6666666666666666,
-          "score_mae": 0.6004958589124929,
-          "ranked_probability_score": 0.10993958949738573
         },
         "ambiguous": {
           "n": 7,
-          "acc": 0.2857142857142857,
-          "nll": 1.1965487237375452,
-          "brier": 0.6527176169156463,
-          "ece": 0.4011055273042717,
-          "mean_conf": 0.6739533055078394,
-          "confidence_bias": 0.3882390197935537,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 0.2857142857142857,
           "accuracy_at_0_9": 1.0,
@@ -83,74 +83,74 @@
         },
         "judge_hard": {
           "n": 17,
-          "acc": 0.7058823529411765,
-          "nll": 0.9352993338445932,
-          "brier": 0.5776553044911235,
-          "ece": 0.34600650565663504,
-          "mean_conf": 0.8837164999753305,
-          "confidence_bias": 0.177834147034154,
-          "confident_error_rate": 0.29411764705882354,
-          "coverage_at_0_9": 0.7647058823529411,
-          "accuracy_at_0_9": 0.6153846153846154,
-          "coverage_at_5pct_error": 0.23529411764705882,
-          "coverage_at_1pct_error": 0.23529411764705882
         },
         "long_policy": {
           "n": 19,
           "acc": 0.21052631578947367,
-          "nll": 1.969063478310097,
-          "brier": 1.0575910654140444,
-          "ece": 0.5443446760383921,
-          "mean_conf": 0.6978151313376447,
-          "confidence_bias": 0.48728881554817105,
-          "confident_error_rate": 0.10526315789473684,
-          "coverage_at_0_9": 0.10526315789473684,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
-          "score_mae": 0.5265190609151413,
-          "ranked_probability_score": 0.15218789918371528
         },
         "multi_hop": {
           "n": 18,
-          "acc": 0.5,
-          "nll": 1.1199038798912422,
-          "brier": 0.6376540745233639,
-          "ece": 0.3014772683765995,
-          "mean_conf": 0.6628249641338441,
-          "confidence_bias": 0.16282496413384406,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.1111111111111111,
           "accuracy_at_0_9": 1.0,
-          "coverage_at_5pct_error": 0.1111111111111111,
-          "coverage_at_1pct_error": 0.1111111111111111,
-          "score_mae": 0.38716699283188594,
-          "ranked_probability_score": 0.1118292374551618
         },
         "probability": {
           "n": 10,
           "acc": 0.5,
-          "nll": 0.7271252358087724,
-          "brier": 0.483854142289723,
-          "ece": 0.14057929069212244,
-          "mean_conf": 0.6261857999614627,
-          "confidence_bias": 0.12618579996146273,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.1,
-          "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.1,
           "coverage_at_1pct_error": 0.1
         },
         "routing_hard": {
           "n": 5,
           "acc": 1.0,
-          "nll": 0.02045794307903345,
-          "brier": 0.0024295831118786066,
-          "ece": 0.019812811096257654,
-          "mean_conf": 0.9801871889037423,
-          "confidence_bias": -0.019812811096257654,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -158,75 +158,75 @@
         "temporal_numeric": {
           "n": 15,
           "acc": 0.13333333333333333,
-          "nll": 2.2634257023598283,
-          "brier": 1.119741919350661,
-          "ece": 0.487236140970707,
-          "mean_conf": 0.6205694743040404,
-          "confidence_bias": 0.4872361409707071,
-          "confident_error_rate": 0.2,
-          "coverage_at_0_9": 0.2,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
-          "score_mae": 1.9373593402178928,
-          "ranked_probability_score": 0.626412898273334
         },
         "tradeoff": {
           "n": 6,
-          "acc": 0.16666666666666666,
-          "nll": 2.697830255531367,
-          "brier": 1.1394884476835054,
-          "ece": 0.6244343812760749,
-          "mean_conf": 0.7911010479427416,
-          "confidence_bias": 0.624434381276075,
-          "confident_error_rate": 0.3333333333333333,
-          "coverage_at_0_9": 0.5,
-          "accuracy_at_0_9": 0.3333333333333333,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0
         },
         "trap": {
           "n": 8,
-          "acc": 0.875,
-          "nll": 0.28094468518213467,
-          "brier": 0.17851500724166375,
-          "ece": 0.1602692173284523,
-          "mean_conf": 0.8835139641418724,
-          "confidence_bias": 0.008513964141872443,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.75,
           "accuracy_at_0_9": 1.0,
-          "coverage_at_5pct_error": 0.75,
-          "coverage_at_1pct_error": 0.75,
-          "score_mae": 0.1477543550053129,
-          "ranked_probability_score": 0.0035212953575381283
         }
       },
       "sources": {
         "adversarial": {
           "n": 6,
-          "acc": 0.6666666666666666,
-          "nll": 0.3804766699344641,
-          "brier": 0.24696365952999444,
-          "ece": 0.10635190756625662,
-          "mean_conf": 0.7502708058819635,
-          "confidence_bias": 0.08360413921529686,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.5,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.6666666666666666,
           "coverage_at_1pct_error": 0.6666666666666666,
-          "score_mae": 0.6004958589124929,
-          "ranked_probability_score": 0.10993958949738573
         },
         "ambiguous": {
           "n": 7,
-          "acc": 0.2857142857142857,
-          "nll": 1.1965487237375452,
-          "brier": 0.6527176169156463,
-          "ece": 0.4011055273042717,
-          "mean_conf": 0.6739533055078394,
-          "confidence_bias": 0.3882390197935537,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 0.2857142857142857,
           "accuracy_at_0_9": 1.0,
@@ -235,74 +235,74 @@
         },
         "judge_hard": {
           "n": 17,
-          "acc": 0.7058823529411765,
-          "nll": 0.9352993338445932,
-          "brier": 0.5776553044911235,
-          "ece": 0.34600650565663504,
-          "mean_conf": 0.8837164999753305,
-          "confidence_bias": 0.177834147034154,
-          "confident_error_rate": 0.29411764705882354,
-          "coverage_at_0_9": 0.7647058823529411,
-          "accuracy_at_0_9": 0.6153846153846154,
-          "coverage_at_5pct_error": 0.23529411764705882,
-          "coverage_at_1pct_error": 0.23529411764705882
         },
         "long_policy": {
           "n": 19,
           "acc": 0.21052631578947367,
-          "nll": 1.969063478310097,
-          "brier": 1.0575910654140444,
-          "ece": 0.5443446760383921,
-          "mean_conf": 0.6978151313376447,
-          "confidence_bias": 0.48728881554817105,
-          "confident_error_rate": 0.10526315789473684,
-          "coverage_at_0_9": 0.10526315789473684,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
-          "score_mae": 0.5265190609151413,
-          "ranked_probability_score": 0.15218789918371528
         },
         "multi_hop": {
           "n": 18,
-          "acc": 0.5,
-          "nll": 1.1199038798912422,
-          "brier": 0.6376540745233639,
-          "ece": 0.3014772683765995,
-          "mean_conf": 0.6628249641338441,
-          "confidence_bias": 0.16282496413384406,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.1111111111111111,
           "accuracy_at_0_9": 1.0,
-          "coverage_at_5pct_error": 0.1111111111111111,
-          "coverage_at_1pct_error": 0.1111111111111111,
-          "score_mae": 0.38716699283188594,
-          "ranked_probability_score": 0.1118292374551618
         },
         "probability": {
           "n": 10,
           "acc": 0.5,
-          "nll": 0.7271252358087724,
-          "brier": 0.483854142289723,
-          "ece": 0.14057929069212244,
-          "mean_conf": 0.6261857999614627,
-          "confidence_bias": 0.12618579996146273,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.1,
-          "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.1,
           "coverage_at_1pct_error": 0.1
         },
         "routing_hard": {
           "n": 5,
           "acc": 1.0,
-          "nll": 0.02045794307903345,
-          "brier": 0.0024295831118786066,
-          "ece": 0.019812811096257654,
-          "mean_conf": 0.9801871889037423,
-          "confidence_bias": -0.019812811096257654,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -310,70 +310,70 @@
         "temporal_numeric": {
           "n": 15,
           "acc": 0.13333333333333333,
-          "nll": 2.2634257023598283,
-          "brier": 1.119741919350661,
-          "ece": 0.487236140970707,
-          "mean_conf": 0.6205694743040404,
-          "confidence_bias": 0.4872361409707071,
-          "confident_error_rate": 0.2,
-          "coverage_at_0_9": 0.2,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
-          "score_mae": 1.9373593402178928,
-          "ranked_probability_score": 0.626412898273334
         },
         "tradeoff": {
           "n": 6,
-          "acc": 0.16666666666666666,
-          "nll": 2.697830255531367,
-          "brier": 1.1394884476835054,
-          "ece": 0.6244343812760749,
-          "mean_conf": 0.7911010479427416,
-          "confidence_bias": 0.624434381276075,
-          "confident_error_rate": 0.3333333333333333,
-          "coverage_at_0_9": 0.5,
-          "accuracy_at_0_9": 0.3333333333333333,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0
         },
         "trap": {
           "n": 8,
-          "acc": 0.875,
-          "nll": 0.28094468518213467,
-          "brier": 0.17851500724166375,
-          "ece": 0.1602692173284523,
-          "mean_conf": 0.8835139641418724,
-          "confidence_bias": 0.008513964141872443,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.75,
           "accuracy_at_0_9": 1.0,
-          "coverage_at_5pct_error": 0.75,
-          "coverage_at_1pct_error": 0.75,
-          "score_mae": 0.1477543550053129,
-          "ranked_probability_score": 0.0035212953575381283
         }
       },
       "variants": {
         "clean": {
           "n": 111,
-          "acc": 0.4594594594594595,
-          "nll": 1.2962948856809682,
-          "brier": 0.6968902514322239,
-          "ece": 0.28535210981983,
-          "mean_conf": 0.7361970214660999,
-          "confidence_bias": 0.2767375620066404,
-          "confident_error_rate": 0.10810810810810811,
-          "coverage_at_0_9": 0.36036036036036034,
-          "accuracy_at_0_9": 0.7,
-          "coverage_at_5pct_error": 0.0990990990990991,
-          "coverage_at_1pct_error": 0.0990990990990991,
-          "score_mae": 0.6876357781329778,
-          "ranked_probability_score": 0.19267980315847502
         }
       },
-      "macro_task_acc": 0.5043789621111603,
-      "objective": -1.1591075907679076,
       "paired_flip": {
         "pairs": 0,
         "incomplete_pairs": 0,
@@ -385,9 +385,9 @@
       },
       "reference_distribution": {
         "n": 10,
-        "squared_l2": 0.0923480672287547,
-        "total_variation": 0.18456016438122402,
-        "kl_reference_to_model": 0.11494476732281071,
         "note": "Exact reference-distribution fidelity, separate from hard-label Brier/ECE."
       },
       "unknowable": null,
@@ -409,10 +409,10 @@
       },
       "latency_ms": {
         "n": 111,
-        "mean": 57.53704737808958,
-        "p50": 41.14527994534001,
-        "p95": 107.0126669947058,
-        "p99": 121.36280137347059
       }
     }
   },

   "served_models": [
     "decision-model"
   ],
+  "elapsed_measurement_s": 5.122954997001216,
+  "requests_per_second": 21.667182332262378,
+  "questions_per_second": 21.667182332262378,
   "latency_ms": {
     "n": 111,
+    "mean": 46.03076461117362,
+    "p50": 38.96701900521293,
+    "p95": 108.61240749363787,
+    "p99": 123.60410959809093
   },
   "adapter_ms": {
     "n": 111,
+    "mean": 44.58860407881271,
+    "p50": 37.559089018031955,
+    "p95": 106.71706849825568,
+    "p99": 122.04659082926815
   },
   "input_tokens_per_request": {
     "n": 111,
     "jevbench-hard/public": {
       "clean": {
         "n": 111,
+        "acc": 0.46846846846846846,
+        "nll": 1.1337637630688588,
+        "brier": 0.6575415552787955,
+        "ece": 0.23352159510752035,
+        "mean_conf": 0.6867711010782002,
+        "confidence_bias": 0.2183026326097317,
+        "confident_error_rate": 0.06306306306306306,
+        "coverage_at_0_9": 0.25225225225225223,
+        "accuracy_at_0_9": 0.75,
+        "coverage_at_5pct_error": 0.07207207207207207,
+        "coverage_at_1pct_error": 0.07207207207207207,
+        "score_mae": 0.5988975177927606,
+        "ranked_probability_score": 0.16734159401413287
       },
       "tasks": {
         "adversarial": {
           "n": 6,
+          "acc": 0.8333333333333334,
+          "nll": 0.43355370552665956,
+          "brier": 0.2520495270653785,
+          "ece": 0.2819930187888528,
+          "mean_conf": 0.7433512435342755,
+          "confidence_bias": -0.08998208979905786,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.3333333333333333,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.6666666666666666,
           "coverage_at_1pct_error": 0.6666666666666666,
+          "score_mae": 0.5525046511362255,
+          "ranked_probability_score": 0.08637420914724503
         },
         "ambiguous": {
           "n": 7,
+          "acc": 0.42857142857142855,
+          "nll": 1.2174010343228778,
+          "brier": 0.6611791751218591,
+          "ece": 0.4002230753039043,
+          "mean_conf": 0.6555588405299361,
+          "confidence_bias": 0.22698741195850752,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 0.2857142857142857,
           "accuracy_at_0_9": 1.0,
         },
         "judge_hard": {
           "n": 17,
+          "acc": 0.5882352941176471,
+          "nll": 0.8909463631971223,
+          "brier": 0.5844481046015214,
+          "ece": 0.323506167851663,
+          "mean_conf": 0.8855414450299884,
+          "confidence_bias": 0.29730615091234136,
+          "confident_error_rate": 0.23529411764705882,
+          "coverage_at_0_9": 0.7058823529411765,
+          "accuracy_at_0_9": 0.6666666666666666,
+          "coverage_at_5pct_error": 0.17647058823529413,
+          "coverage_at_1pct_error": 0.17647058823529413
         },
         "long_policy": {
           "n": 19,
           "acc": 0.21052631578947367,
+          "nll": 1.6198424118360804,
+          "brier": 0.9501318928553117,
+          "ece": 0.40032160864841904,
+          "mean_conf": 0.6108479244378927,
+          "confidence_bias": 0.40032160864841904,
+          "confident_error_rate": 0.05263157894736842,
+          "coverage_at_0_9": 0.05263157894736842,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
+          "score_mae": 0.4105976174660464,
+          "ranked_probability_score": 0.13040975449243591
         },
         "multi_hop": {
           "n": 18,
+          "acc": 0.4444444444444444,
+          "nll": 1.0756646284750686,
+          "brier": 0.6212047882547728,
+          "ece": 0.17079526618894453,
+          "mean_conf": 0.5902725537487883,
+          "confidence_bias": 0.14582810930434387,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.05555555555555555,
           "accuracy_at_0_9": 1.0,
+          "coverage_at_5pct_error": 0.16666666666666666,
+          "coverage_at_1pct_error": 0.16666666666666666,
+          "score_mae": 0.10929065461430265,
+          "ranked_probability_score": 0.08843563517050587
         },
         "probability": {
           "n": 10,
           "acc": 0.5,
+          "nll": 0.7781094688183425,
+          "brier": 0.5066911105112001,
+          "ece": 0.12302537166641263,
+          "mean_conf": 0.5782217047826121,
+          "confidence_bias": 0.07822170478261214,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.0,
+          "accuracy_at_0_9": null,
           "coverage_at_5pct_error": 0.1,
           "coverage_at_1pct_error": 0.1
         },
         "routing_hard": {
           "n": 5,
           "acc": 1.0,
+          "nll": 0.11790766167390429,
+          "brier": 0.029803866265500058,
+          "ece": 0.10733065826074913,
+          "mean_conf": 0.892669341739251,
+          "confidence_bias": -0.10733065826074895,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.6,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "temporal_numeric": {
           "n": 15,
           "acc": 0.13333333333333333,
+          "nll": 1.872570080540508,
+          "brier": 1.0507975537151109,
+          "ece": 0.46625775536857367,
+          "mean_conf": 0.5772131644777224,
+          "confidence_bias": 0.4438798311443891,
+          "confident_error_rate": 0.06666666666666667,
+          "coverage_at_0_9": 0.06666666666666667,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
+          "score_mae": 1.8155008348426862,
+          "ranked_probability_score": 0.5551970640131222
         },
         "tradeoff": {
           "n": 6,
+          "acc": 0.3333333333333333,
+          "nll": 1.8645472555110578,
+          "brier": 0.9793224099311413,
+          "ece": 0.5772121379330721,
+          "mean_conf": 0.7302834506219541,
+          "confidence_bias": 0.3969501172886208,
+          "confident_error_rate": 0.16666666666666666,
+          "coverage_at_0_9": 0.3333333333333333,
+          "accuracy_at_0_9": 0.5,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0
         },
         "trap": {
           "n": 8,
+          "acc": 1.0,
+          "nll": 0.22414039721355677,
+          "brier": 0.10286538936066428,
+          "ece": 0.17351376874535934,
+          "mean_conf": 0.8264862312546406,
+          "confidence_bias": -0.1735137687453594,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.5,
           "accuracy_at_0_9": 1.0,
+          "coverage_at_5pct_error": 1.0,
+          "coverage_at_1pct_error": 1.0,
+          "score_mae": 0.2948937312312564,
+          "ranked_probability_score": 0.01322314676905235
         }
       },
       "sources": {
         "adversarial": {
           "n": 6,
+          "acc": 0.8333333333333334,
+          "nll": 0.43355370552665956,
+          "brier": 0.2520495270653785,
+          "ece": 0.2819930187888528,
+          "mean_conf": 0.7433512435342755,
+          "confidence_bias": -0.08998208979905786,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.3333333333333333,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.6666666666666666,
           "coverage_at_1pct_error": 0.6666666666666666,
+          "score_mae": 0.5525046511362255,
+          "ranked_probability_score": 0.08637420914724503
         },
         "ambiguous": {
           "n": 7,
+          "acc": 0.42857142857142855,
+          "nll": 1.2174010343228778,
+          "brier": 0.6611791751218591,
+          "ece": 0.4002230753039043,
+          "mean_conf": 0.6555588405299361,
+          "confidence_bias": 0.22698741195850752,
           "confident_error_rate": 0.0,
           "coverage_at_0_9": 0.2857142857142857,
           "accuracy_at_0_9": 1.0,
         },
         "judge_hard": {
           "n": 17,
+          "acc": 0.5882352941176471,
+          "nll": 0.8909463631971223,
+          "brier": 0.5844481046015214,
+          "ece": 0.323506167851663,
+          "mean_conf": 0.8855414450299884,
+          "confidence_bias": 0.29730615091234136,
+          "confident_error_rate": 0.23529411764705882,
+          "coverage_at_0_9": 0.7058823529411765,
+          "accuracy_at_0_9": 0.6666666666666666,
+          "coverage_at_5pct_error": 0.17647058823529413,
+          "coverage_at_1pct_error": 0.17647058823529413
         },
         "long_policy": {
           "n": 19,
           "acc": 0.21052631578947367,
+          "nll": 1.6198424118360804,
+          "brier": 0.9501318928553117,
+          "ece": 0.40032160864841904,
+          "mean_conf": 0.6108479244378927,
+          "confidence_bias": 0.40032160864841904,
+          "confident_error_rate": 0.05263157894736842,
+          "coverage_at_0_9": 0.05263157894736842,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
+          "score_mae": 0.4105976174660464,
+          "ranked_probability_score": 0.13040975449243591
         },
         "multi_hop": {
           "n": 18,
+          "acc": 0.4444444444444444,
+          "nll": 1.0756646284750686,
+          "brier": 0.6212047882547728,
+          "ece": 0.17079526618894453,
+          "mean_conf": 0.5902725537487883,
+          "confidence_bias": 0.14582810930434387,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.05555555555555555,
           "accuracy_at_0_9": 1.0,
+          "coverage_at_5pct_error": 0.16666666666666666,
+          "coverage_at_1pct_error": 0.16666666666666666,
+          "score_mae": 0.10929065461430265,
+          "ranked_probability_score": 0.08843563517050587
         },
         "probability": {
           "n": 10,
           "acc": 0.5,
+          "nll": 0.7781094688183425,
+          "brier": 0.5066911105112001,
+          "ece": 0.12302537166641263,
+          "mean_conf": 0.5782217047826121,
+          "confidence_bias": 0.07822170478261214,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.0,
+          "accuracy_at_0_9": null,
           "coverage_at_5pct_error": 0.1,
           "coverage_at_1pct_error": 0.1
         },
         "routing_hard": {
           "n": 5,
           "acc": 1.0,
+          "nll": 0.11790766167390429,
+          "brier": 0.029803866265500058,
+          "ece": 0.10733065826074913,
+          "mean_conf": 0.892669341739251,
+          "confidence_bias": -0.10733065826074895,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.6,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "temporal_numeric": {
           "n": 15,
           "acc": 0.13333333333333333,
+          "nll": 1.872570080540508,
+          "brier": 1.0507975537151109,
+          "ece": 0.46625775536857367,
+          "mean_conf": 0.5772131644777224,
+          "confidence_bias": 0.4438798311443891,
+          "confident_error_rate": 0.06666666666666667,
+          "coverage_at_0_9": 0.06666666666666667,
           "accuracy_at_0_9": 0.0,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0,
+          "score_mae": 1.8155008348426862,
+          "ranked_probability_score": 0.5551970640131222
         },
         "tradeoff": {
           "n": 6,
+          "acc": 0.3333333333333333,
+          "nll": 1.8645472555110578,
+          "brier": 0.9793224099311413,
+          "ece": 0.5772121379330721,
+          "mean_conf": 0.7302834506219541,
+          "confidence_bias": 0.3969501172886208,
+          "confident_error_rate": 0.16666666666666666,
+          "coverage_at_0_9": 0.3333333333333333,
+          "accuracy_at_0_9": 0.5,
           "coverage_at_5pct_error": 0.0,
           "coverage_at_1pct_error": 0.0
         },
         "trap": {
           "n": 8,
+          "acc": 1.0,
+          "nll": 0.22414039721355677,
+          "brier": 0.10286538936066428,
+          "ece": 0.17351376874535934,
+          "mean_conf": 0.8264862312546406,
+          "confidence_bias": -0.1735137687453594,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.5,
           "accuracy_at_0_9": 1.0,
+          "coverage_at_5pct_error": 1.0,
+          "coverage_at_1pct_error": 1.0,
+          "score_mae": 0.2948937312312564,
+          "ranked_probability_score": 0.01322314676905235
         }
       },
       "variants": {
         "clean": {
           "n": 111,
+          "acc": 0.46846846846846846,
+          "nll": 1.1337637630688588,
+          "brier": 0.6575415552787955,
+          "ece": 0.23352159510752035,
+          "mean_conf": 0.6867711010782002,
+          "confidence_bias": 0.2183026326097317,
+          "confident_error_rate": 0.06306306306306306,
+          "coverage_at_0_9": 0.25225225225225223,
+          "accuracy_at_0_9": 0.75,
+          "coverage_at_5pct_error": 0.07207207207207207,
+          "coverage_at_1pct_error": 0.07207207207207207,
+          "score_mae": 0.5988975177927606,
+          "ranked_probability_score": 0.16734159401413287
         }
       },
+      "macro_task_acc": 0.5471777482922994,
+      "objective": -1.0094683007115177,
       "paired_flip": {
         "pairs": 0,
         "incomplete_pairs": 0,
       },
       "reference_distribution": {
         "n": 10,
+        "squared_l2": 0.08885015478793142,
+        "total_variation": 0.185838091217169,
+        "kl_reference_to_model": 0.10584525813582088,
         "note": "Exact reference-distribution fidelity, separate from hard-label Brier/ECE."
       },
       "unknowable": null,
       },
       "latency_ms": {
         "n": 111,
+        "mean": 46.03076461117362,
+        "p50": 38.96701900521293,
+        "p95": 108.61240749363787,
+        "p99": 123.60410959809093
       }
     }
   },

evidence/served-nosys-original-report.json CHANGED Viewed

@@ -8,22 +8,22 @@
   "served_models": [
     "decision-model"
   ],
-  "elapsed_measurement_s": 2.666326795006171,
-  "requests_per_second": 27.00344163920588,
-  "questions_per_second": 27.00344163920588,
   "latency_ms": {
     "n": 72,
-    "mean": 36.9241978882605,
-    "p50": 37.165592511883005,
-    "p95": 39.12460421561264,
-    "p99": 43.66849939222452
   },
   "adapter_ms": {
     "n": 72,
-    "mean": 35.50634430495039,
-    "p50": 35.74040098465048,
-    "p95": 37.74651304702275,
-    "p99": 41.85468893789225
   },
   "input_tokens_per_request": {
     "n": 72,
@@ -37,44 +37,44 @@
       "clean": {
         "n": 72,
         "acc": 0.8888888888888888,
-        "nll": 0.2445291027435668,
-        "brier": 0.14146770822826904,
-        "ece": 0.07089677407755061,
-        "mean_conf": 0.9290339623031046,
-        "confidence_bias": 0.040145073414215804,
         "confident_error_rate": 0.013888888888888888,
-        "coverage_at_0_9": 0.7777777777777778,
-        "accuracy_at_0_9": 0.9821428571428571,
-        "coverage_at_5pct_error": 0.875,
-        "coverage_at_1pct_error": 0.4722222222222222,
-        "score_mae": 0.14699746098780417,
-        "ranked_probability_score": 0.027507417748960535
       },
       "tasks": {
         "adequacy": {
           "n": 12,
           "acc": 0.6666666666666666,
-          "nll": 0.7362793168402896,
-          "brier": 0.4277953207024657,
-          "ece": 0.2804673152635425,
-          "mean_conf": 0.8273459270172787,
-          "confidence_bias": 0.16067926035061209,
           "confident_error_rate": 0.08333333333333333,
-          "coverage_at_0_9": 0.3333333333333333,
-          "accuracy_at_0_9": 0.75,
           "coverage_at_5pct_error": 0.16666666666666666,
           "coverage_at_1pct_error": 0.16666666666666666
         },
         "extraction": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.029664413922215655,
-          "brier": 0.00422488499266305,
-          "ece": 0.028473599180563025,
-          "mean_conf": 0.971526400819437,
-          "confidence_bias": -0.028473599180563025,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -82,13 +82,13 @@
         "intent": {
           "n": 12,
           "acc": 0.8333333333333334,
-          "nll": 0.3530283282604048,
-          "brier": 0.24567434750699083,
-          "ece": 0.1528447829790891,
-          "mean_conf": 0.9619515814755606,
-          "confidence_bias": 0.12861824814222722,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.8333333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.8333333333333334,
           "coverage_at_1pct_error": 0.8333333333333334
@@ -96,29 +96,29 @@
         "ordinal": {
           "n": 12,
           "acc": 0.9166666666666666,
-          "nll": 0.20100958799798233,
-          "brier": 0.10103710326805294,
-          "ece": 0.0759267994222482,
-          "mean_conf": 0.9162006485755824,
-          "confidence_bias": -0.00046601809108426373,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.75,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666,
-          "score_mae": 0.14699746098780417,
-          "ranked_probability_score": 0.027507417748960535
         },
         "policy": {
           "n": 12,
           "acc": 0.9166666666666666,
-          "nll": 0.10621874252814244,
-          "brier": 0.06285648604607869,
-          "ece": 0.079613773421764,
-          "mean_conf": 0.9358306599039143,
-          "confidence_bias": 0.019163993237247712,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.8333333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666
@@ -126,13 +126,13 @@
         "routing": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.04097422691236581,
-          "brier": 0.007218106853362921,
-          "ece": 0.03865144397314558,
-          "mean_conf": 0.9613485560268544,
-          "confidence_bias": -0.03865144397314557,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.9166666666666666,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -142,27 +142,27 @@
         "adequacy": {
           "n": 12,
           "acc": 0.6666666666666666,
-          "nll": 0.7362793168402896,
-          "brier": 0.4277953207024657,
-          "ece": 0.2804673152635425,
-          "mean_conf": 0.8273459270172787,
-          "confidence_bias": 0.16067926035061209,
           "confident_error_rate": 0.08333333333333333,
-          "coverage_at_0_9": 0.3333333333333333,
-          "accuracy_at_0_9": 0.75,
           "coverage_at_5pct_error": 0.16666666666666666,
           "coverage_at_1pct_error": 0.16666666666666666
         },
         "extraction": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.029664413922215655,
-          "brier": 0.00422488499266305,
-          "ece": 0.028473599180563025,
-          "mean_conf": 0.971526400819437,
-          "confidence_bias": -0.028473599180563025,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 1.0,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -170,13 +170,13 @@
         "intent": {
           "n": 12,
           "acc": 0.8333333333333334,
-          "nll": 0.3530283282604048,
-          "brier": 0.24567434750699083,
-          "ece": 0.1528447829790891,
-          "mean_conf": 0.9619515814755606,
-          "confidence_bias": 0.12861824814222722,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.8333333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.8333333333333334,
           "coverage_at_1pct_error": 0.8333333333333334
@@ -184,29 +184,29 @@
         "ordinal": {
           "n": 12,
           "acc": 0.9166666666666666,
-          "nll": 0.20100958799798233,
-          "brier": 0.10103710326805294,
-          "ece": 0.0759267994222482,
-          "mean_conf": 0.9162006485755824,
-          "confidence_bias": -0.00046601809108426373,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.75,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666,
-          "score_mae": 0.14699746098780417,
-          "ranked_probability_score": 0.027507417748960535
         },
         "policy": {
           "n": 12,
           "acc": 0.9166666666666666,
-          "nll": 0.10621874252814244,
-          "brier": 0.06285648604607869,
-          "ece": 0.079613773421764,
-          "mean_conf": 0.9358306599039143,
-          "confidence_bias": 0.019163993237247712,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.8333333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666
@@ -214,13 +214,13 @@
         "routing": {
           "n": 12,
           "acc": 1.0,
-          "nll": 0.04097422691236581,
-          "brier": 0.007218106853362921,
-          "ece": 0.03865144397314558,
-          "mean_conf": 0.9613485560268544,
-          "confidence_bias": -0.03865144397314557,
           "confident_error_rate": 0.0,
-          "coverage_at_0_9": 0.9166666666666666,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
@@ -230,22 +230,22 @@
         "clean": {
           "n": 72,
           "acc": 0.8888888888888888,
-          "nll": 0.2445291027435668,
-          "brier": 0.14146770822826904,
-          "ece": 0.07089677407755061,
-          "mean_conf": 0.9290339623031046,
-          "confidence_bias": 0.040145073414215804,
           "confident_error_rate": 0.013888888888888888,
-          "coverage_at_0_9": 0.7777777777777778,
-          "accuracy_at_0_9": 0.9821428571428571,
-          "coverage_at_5pct_error": 0.875,
-          "coverage_at_1pct_error": 0.4722222222222222,
-          "score_mae": 0.14699746098780417,
-          "ranked_probability_score": 0.027507417748960535
         }
       },
       "macro_task_acc": 0.8888888888888888,
-      "objective": -0.24452910274356676,
       "paired_flip": {
         "pairs": 0,
         "incomplete_pairs": 0,
@@ -275,10 +275,10 @@
       },
       "latency_ms": {
         "n": 72,
-        "mean": 36.9241978882605,
-        "p50": 37.165592511883005,
-        "p95": 39.12460421561264,
-        "p99": 43.66849939222452
       }
     }
   },

   "served_models": [
     "decision-model"
   ],
+  "elapsed_measurement_s": 2.017445308039896,
+  "requests_per_second": 35.68869981905659,
+  "questions_per_second": 35.68869981905659,
   "latency_ms": {
     "n": 72,
+    "mean": 27.915438723640463,
+    "p50": 22.776359983254224,
+    "p95": 37.59642824297771,
+    "p99": 51.80573260760867
   },
   "adapter_ms": {
     "n": 72,
+    "mean": 26.59171291452367,
+    "p50": 21.543186478083953,
+    "p95": 36.17154256789945,
+    "p99": 50.218841551686644
   },
   "input_tokens_per_request": {
     "n": 72,
       "clean": {
         "n": 72,
         "acc": 0.8888888888888888,
+        "nll": 0.3167528651931516,
+        "brier": 0.17124937445495855,
+        "ece": 0.09403401161899587,
+        "mean_conf": 0.8421306839082972,
+        "confidence_bias": -0.04675820498059169,
         "confident_error_rate": 0.013888888888888888,
+        "coverage_at_0_9": 0.4305555555555556,
+        "accuracy_at_0_9": 0.967741935483871,
+        "coverage_at_5pct_error": 0.8888888888888888,
+        "coverage_at_1pct_error": 0.2777777777777778,
+        "score_mae": 0.19513049786330436,
+        "ranked_probability_score": 0.026410730125522457
       },
       "tasks": {
         "adequacy": {
           "n": 12,
           "acc": 0.6666666666666666,
+          "nll": 0.683123061526575,
+          "brier": 0.45385406932289785,
+          "ece": 0.13453225962896861,
+          "mean_conf": 0.7765243923506177,
+          "confidence_bias": 0.10985772568395102,
           "confident_error_rate": 0.08333333333333333,
+          "coverage_at_0_9": 0.25,
+          "accuracy_at_0_9": 0.6666666666666666,
           "coverage_at_5pct_error": 0.16666666666666666,
           "coverage_at_1pct_error": 0.16666666666666666
         },
         "extraction": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.24122705142808795,
+          "brier": 0.11136659001151566,
+          "ece": 0.19344591228522273,
+          "mean_conf": 0.8065540877147773,
+          "confidence_bias": -0.19344591228522268,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.3333333333333333,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "intent": {
           "n": 12,
           "acc": 0.8333333333333334,
+          "nll": 0.3484790691487531,
+          "brier": 0.18052583058427374,
+          "ece": 0.21071098357468712,
+          "mean_conf": 0.8248146273993879,
+          "confidence_bias": -0.008518705933945436,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.25,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.8333333333333334,
           "coverage_at_1pct_error": 0.8333333333333334
         "ordinal": {
           "n": 12,
           "acc": 0.9166666666666666,
+          "nll": 0.28015326678006397,
+          "brier": 0.1296383102293185,
+          "ece": 0.15925303903164556,
+          "mean_conf": 0.8634828861172629,
+          "confidence_bias": -0.05318378054940376,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.5833333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666,
+          "score_mae": 0.19513049786330436,
+          "ranked_probability_score": 0.026410730125522457
         },
         "policy": {
           "n": 12,
           "acc": 0.9166666666666666,
+          "nll": 0.18700125094153675,
+          "brier": 0.1128412336883513,
+          "ece": 0.12327599960708283,
+          "mean_conf": 0.9229406430622116,
+          "confidence_bias": 0.006273976395544945,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.75,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666
         "routing": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.16053349133389275,
+          "brier": 0.039270212893394316,
+          "ece": 0.14153253319447423,
+          "mean_conf": 0.8584674668055258,
+          "confidence_bias": -0.14153253319447423,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.4166666666666667,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "adequacy": {
           "n": 12,
           "acc": 0.6666666666666666,
+          "nll": 0.683123061526575,
+          "brier": 0.45385406932289785,
+          "ece": 0.13453225962896861,
+          "mean_conf": 0.7765243923506177,
+          "confidence_bias": 0.10985772568395102,
           "confident_error_rate": 0.08333333333333333,
+          "coverage_at_0_9": 0.25,
+          "accuracy_at_0_9": 0.6666666666666666,
           "coverage_at_5pct_error": 0.16666666666666666,
           "coverage_at_1pct_error": 0.16666666666666666
         },
         "extraction": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.24122705142808795,
+          "brier": 0.11136659001151566,
+          "ece": 0.19344591228522273,
+          "mean_conf": 0.8065540877147773,
+          "confidence_bias": -0.19344591228522268,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.3333333333333333,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "intent": {
           "n": 12,
           "acc": 0.8333333333333334,
+          "nll": 0.3484790691487531,
+          "brier": 0.18052583058427374,
+          "ece": 0.21071098357468712,
+          "mean_conf": 0.8248146273993879,
+          "confidence_bias": -0.008518705933945436,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.25,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.8333333333333334,
           "coverage_at_1pct_error": 0.8333333333333334
         "ordinal": {
           "n": 12,
           "acc": 0.9166666666666666,
+          "nll": 0.28015326678006397,
+          "brier": 0.1296383102293185,
+          "ece": 0.15925303903164556,
+          "mean_conf": 0.8634828861172629,
+          "confidence_bias": -0.05318378054940376,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.5833333333333334,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666,
+          "score_mae": 0.19513049786330436,
+          "ranked_probability_score": 0.026410730125522457
         },
         "policy": {
           "n": 12,
           "acc": 0.9166666666666666,
+          "nll": 0.18700125094153675,
+          "brier": 0.1128412336883513,
+          "ece": 0.12327599960708283,
+          "mean_conf": 0.9229406430622116,
+          "confidence_bias": 0.006273976395544945,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.75,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 0.9166666666666666,
           "coverage_at_1pct_error": 0.9166666666666666
         "routing": {
           "n": 12,
           "acc": 1.0,
+          "nll": 0.16053349133389275,
+          "brier": 0.039270212893394316,
+          "ece": 0.14153253319447423,
+          "mean_conf": 0.8584674668055258,
+          "confidence_bias": -0.14153253319447423,
           "confident_error_rate": 0.0,
+          "coverage_at_0_9": 0.4166666666666667,
           "accuracy_at_0_9": 1.0,
           "coverage_at_5pct_error": 1.0,
           "coverage_at_1pct_error": 1.0
         "clean": {
           "n": 72,
           "acc": 0.8888888888888888,
+          "nll": 0.3167528651931516,
+          "brier": 0.17124937445495855,
+          "ece": 0.09403401161899587,
+          "mean_conf": 0.8421306839082972,
+          "confidence_bias": -0.04675820498059169,
           "confident_error_rate": 0.013888888888888888,
+          "coverage_at_0_9": 0.4305555555555556,
+          "accuracy_at_0_9": 0.967741935483871,
+          "coverage_at_5pct_error": 0.8888888888888888,
+          "coverage_at_1pct_error": 0.2777777777777778,
+          "score_mae": 0.19513049786330436,
+          "ranked_probability_score": 0.026410730125522457
         }
       },
       "macro_task_acc": 0.8888888888888888,
+      "objective": -0.31675286519315154,
       "paired_flip": {
         "pairs": 0,
         "incomplete_pairs": 0,
       },
       "latency_ms": {
         "n": 72,
+        "mean": 27.915438723640463,
+        "p50": 22.776359983254224,
+        "p95": 37.59642824297771,
+        "p99": 51.80573260760867
       }
     }
   },

release-manifest.json CHANGED Viewed

@@ -18,9 +18,23 @@
   "serving": {
     "wording": "native",
     "system_prompt": null,
-    "temperature": 1.55,
     "served_model_name": "standard-one-3b",
     "engine": "SGLang 0.5.20",
-    "note": "serving settings shown are those of v1.1; they are being re-fitted for v2"
   }
 }

   "serving": {
     "wording": "native",
     "system_prompt": null,
     "served_model_name": "standard-one-3b",
     "engine": "SGLang 0.5.20",
+    "temperature_by_type": {
+      "choice": 0.95,
+      "noul": 1.05,
+      "score": 0.85
+    },
+    "default_temperature": 0.95,
+    "selection": "wording and temperatures chosen on non-JevBench data (evidence/prompt-wording-choice.json, evidence/calibration-per-type-served.json)",
+    "alternative_wording": {
+      "wording": "served",
+      "default_temperature": 0.9,
+      "temperature_by_type": {
+        "choice": 0.9,
+        "noul": 1.05,
+        "score": 0.95
+      }
+    }
   }
 }

server/README.md CHANGED Viewed

@@ -15,15 +15,19 @@ CUDA_VISIBLE_DEVICES=0 SGLANG_VLM_CACHE_SIZE_MB=0 python -m sglang.launch_server
   --chunked-prefill-size -1 --disable-radix-cache --mm-preprocess-cache-size-mb 0 \
   --model-config-parser hf --load-format safetensors
-# terminal 2: adapter (no system prompt; temperatures: see the model card)
 jev-adapter --engine-url http://127.0.0.1:30000 --model standard-one-3b --alias jev-latest \
   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-3B-Instruct-2512-BF16 \
   --tokenizer-revision b6d637bef2393152b3da2b2fde72eecdee30557e \
-  --prompt-wording native --native-system-prompt none \
-  --default-temperature <T> --temperature-by-type choice=<T>,noul=<T>,score=<T>
 ```
 `--temperature-by-type` sets one default temperature per answer type (`choice`, `noul`, `score`); a request's own
 `options.temperature` still wins. Let only this adapter talk to the engine port. The recommended values and the full
 request format are in the model card (`../README.md`).

   --chunked-prefill-size -1 --disable-radix-cache --mm-preprocess-cache-size-mb 0 \
   --model-config-parser hf --load-format safetensors
+# terminal 2: adapter (no system prompt)
 jev-adapter --engine-url http://127.0.0.1:30000 --model standard-one-3b --alias jev-latest \
   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-3B-Instruct-2512-BF16 \
   --tokenizer-revision b6d637bef2393152b3da2b2fde72eecdee30557e \
+  --prompt-wording native --native-system-prompt none --default-temperature 0.95 --temperature-by-type choice=0.95,noul=1.05,score=0.85
 ```
+The adapter supports two prompt wordings; each has its own fitted temperatures (details on the model card):
+- `native` (recommended default): `--prompt-wording native --native-system-prompt none --default-temperature 0.95 --temperature-by-type choice=0.95,noul=1.05,score=0.85`
+- `served`: `--prompt-wording served --default-temperature 0.90 --temperature-by-type choice=0.90,noul=1.05,score=0.95`
 `--temperature-by-type` sets one default temperature per answer type (`choice`, `noul`, `score`); a request's own
 `options.temperature` still wins. Let only this adapter talk to the engine port. The recommended values and the full
 request format are in the model card (`../README.md`).