AUDIT.md — Independent Verification of ACO Claims

Date: 2026-06-15 · Auditor: ML Intern (independent re-evaluation, not training logs)

This document records an independent audit of the claims in the ACO project. Method: re-load artifacts from the Hub, recreate test splits from scratch (train_test_split(test_size=0.15, seed=42) on the original source datasets), run fresh inference, and recompute metrics with sklearn — deliberately not trusting any logged training metric.

Part A — Specialist classifier metrics: VERIFIED ✅

All three v1 DistilBERT specialists were re-evaluated on freshly recreated held-out splits. GPU verification job: 6a1f651eb291489980136ba6 (COMPLETED). Tool-gater re-run cross-checked against v2_verification_results.json (exact match).

Model	Metric	Claimed (old)	Verified (re-run)	Verdict
Tool-Use Gater	Accuracy	0.9503	0.9823	✅ Better than claimed
Tool-Use Gater	F1 Macro	0.8649	0.9195	✅ Better than claimed
Tier Router	Accuracy	0.7008	0.7016	✅ Matches
Tier Router	F1 Macro	0.6730	0.6738	✅ Matches
Verifier Gater	Accuracy	0.6559	0.6547	✅ Matches
Verifier Gater	F1 Macro	0.6551	0.6539	✅ Matches

Test set sizes: Tool-Use Gater N=1695, Tier Router N=1260, Verifier Gater N=863.

Majority-class baselines (proves real learning, not majority voting):

Model	Majority baseline	Verified accuracy	Lift
Tool-Use Gater	93.9%	98.2%	+4.3 pp
Tier Router	47.1%	70.2%	+23.1 pp
Verifier Gater	55.4%	65.5%	+10.1 pp

Correction applied: the tool-gater model card and project summary previously quoted ~0.935 acc / 0.60 F1-macro (an early epoch-1 checkpoint, pre-oversampling). The pushed model is the final checkpoint and is better than documented. Cards updated 2026-06-15.

Architecture claims: confirmed via config.json — all three are DistilBertForSequenceClassification, n_layers=6, dim=768, vocab=30522, 67.0M params, model.safetensors = 255.4 MB each. ✅

v2 ModernBERT: regression confirmed ❌

v2_verification_results.json (job 6a207319) shows the v2 upgrade did not help:

Model	v1 F1	v2 F1	Delta	Verdict
tier-router	0.670	0.677	+0.007	flat (noise)
tool-gater	0.920	0.724	−0.196	regression
verifier-gater	0.654	0.667	+0.013	flat (noise)

Recommendation stands: use v1.

Part B — Cost-savings, cascade, and Django claims

B1. "Cascade ≈ 56% cost reduction at iso-quality" — ⚠️ SIMULATED, not test-verified

Source of the number: a trace-based simulation. Costs come from a hardcoded pricing table and per-tier success-rate model (aco/benchmark.py), and from replaying SWE-Router dataset traces. There is no live Docker test execution behind the headline figure.
The repo already corrected its own overclaim. CORRECTED_REPORT.md (Fix #1) shows that against a fair baseline (frontier-with-retry, equal retry budget), frontier-retry actually solves 4 MORE instances (420 vs 416), not fewer. The solve-rate difference is not statistically significant ([−2.8pp, +1.0pp], 1000-bootstrap). Cascade's real, defensible win is cost (~56% cheaper at statistically-equivalent quality), not higher solve rate.
A prior "$585 provider-routing savings" figure was a unit error, corrected to $18.16 in CORRECTED_REPORT.md (Fix #3).
Verdict: The cost-reduction claim is internally consistent and honestly caveated within the repo's own docs, but it is a simulation result. It should not be cited as a measured, live-evaluated SWE-bench result. The README has been updated to label it "simulated."

B2. "4/4 Django instances solved by free-tier Llama-3.1-8B" — ⚠️ MISLEADING (true but weaker than it sounds)

From BATCH_VALIDATION_RESULTS.md (job 6a04d3a3, status ERROR — crashed at instance 5).
What is true: 4 Django instances each produced a patch that passed git apply --check (i.e. syntactically valid diffs) from T1 in 3–6 turns.
What is not true: these were not test-verified. The same doc states plainly: "No test verification... The patches are syntactically valid diffs but may not fix the bug correctly." FAIL_TO_PASS pytest was never run on them.
Verdict: "4/4 valid patches" = 4/4 apply-able diffs, not 4/4 resolved SWE-bench instances. Do not read it as a 100% solve rate.

B3. Live SWE-bench resolution by the cascade agent — ❌ NOT DEMONSTRATED

The validation scripts (validate_cascade.py, quick_validate.py) contain correct end-to-end logic: clone repo → conda env → cascade agent (T1→T2) → apply model patch + test_patch → run FAIL_TO_PASS and PASS_TO_PASS via pytest.
However, no completed results artifact with test-verified resolved=True exists in the repo (no populated validation_results.jsonl / batch_results.jsonl). CORRECTED_REPORT.md (Fix #4) confirms Docker/test verification was only "PARTIAL" and blocked by lack of a Docker daemon on HF infra.
Verdict: The cascade is proven in simulation and the agent produces patches, but a genuine live SWE-bench resolution rate is not established anywhere in the repo.

B4. Dataset existence — ✅ VERIFIED

narcolepticchicken/aco-traces exists with tool_gater, tier_router, verifier_gater configs (train/test parquet). Preprocessing in build_traces.py is identical to the training/verification loaders. The earlier "aco-traces is empty" claim is false now.

Bottom line

Classifier metrics (Part A): all TRUE and verified. Tool-gater was understated and has been corrected upward. v2 ModernBERT is a confirmed regression — v1 is the production set.
Cost-savings (Part B): honest but must be labelled "simulated." The 56% figure is a trace simulation, the "4/4 Django" is apply-check-only (not test-verified), and no live SWE-bench resolution rate exists. The repo's own TRUTH.md / CORRECTED_REPORT.md already disclose these caveats; the stale overclaims were in the README files, now corrected.

Nothing fabricated. The main risk was stale/over-optimistic README framing, now fixed.