---
license: apache-2.0
library_name: flashback
tags:
- anomaly-detection
- distributed-training
- observability
- debugging
- machine-learning
- flashback
metrics:
- mae
model-index:
- name: flashback-first-bad-step
  results:
  - task:
      type: anomaly-detection
      name: First-bad-step localisation
    dataset:
      name: Flashback Forensics
      type: NagaYu/flashback-forensics
      split: test
    metrics:
    - type: mae
      name: Mean absolute step error
      value: 0.00
    - type: accuracy
      name: Exact-step hit rate
      value: 1.000
---

# Flashback: first-bad-step detector

Given a run's **sketch index** -- a few hundred bytes of summary statistics per
training step -- this model reports the step at which the run first went wrong.

It is small on purpose: 10 logistic weights plus 9 fitted
consensus hyperparameters. The interesting part is not the model, it is that
the *input* is cheap enough to record for every step of every run.

## Results on held-out runs

| detector | mean abs. error (steps) | exact | within 2 | never detected |
|---|---:|---:|---:|---:|
| loss curve alone | 0.50 | 2/6 | 4/6 | 2 |
| **this model** | **0.00** | **6/6** | **6/6** | **0** |

Train split: 12 runs, MAE 0.00.
Splits are by run: no step of a test run appears in training.

## Usage

```python
from flashback.detector import FirstBadStepDetector
from flashback.sketch import StepSketchIndex
from huggingface_hub import snapshot_download

det = FirstBadStepDetector.load(snapshot_download("<user>/flashback-first-bad-step"))
report = det.predict(StepSketchIndex("runs/my-run/sketch", mode="r"))
print(report.first_bad_step, report.confidence, report.top_metrics)
```

To produce the sketch index in the first place, add three lines to your
training loop:

```python
from flashback.integrations import FlashbackCallback
trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")])
```

## Fitted configuration

```json
{
  "win": 32,
  "gap": 1,
  "mode": "diff",
  "k_cal": 8.0,
  "k_min": 8.0,
  "min_votes": 3,
  "vote_window": 2,
  "exclude": [
    "lr",
    "step_wall_ms",
    "batch_id_mean",
    "batch_uniq_frac"
  ],
  "use_logistic": true,
  "logistic_threshold": 0.5,
  "version": "1"
}
```

## Feature weights

| feature | weight |
|---|---:|
| `bias` | -9.8951 |
| `log1p_max_z` | +0.1471 |
| `log1p_top5_mean_z` | +0.1950 |
| `frac_metrics_firing` | +18.4084 |
| `hard_signal` | +1.2226 |
| `log1p_z_grad_norm` | +0.3008 |
| `log1p_z_update_max` | +0.1584 |
| `log1p_z_loss` | +0.0378 |
| `log1p_z_hist_tail` | +0.2497 |
| `rel_position` | +15.0875 |

## Limitations

- It needs a **healthy stretch** of the run to calibrate against; a run that is
  broken from step 0 has no baseline and will not be localised.
- It reports *regime changes*, so a fault that ramps in smoothly over hundreds
  of steps has no single correct answer and the reported step will be early or
  late by roughly the ramp length.
- It was fitted on small Transformers (a few hundred thousand to a few million
  parameters). The features are scale-free by construction, but transfer to
  very large models is **not** demonstrated here.
- No claim of bitwise determinism is made anywhere in Flashback; this is a
  statistical detector over recorded statistics.

Generated by Flashback v0.1.0.