# TRoB-1 Main Analysis Data

This directory contains version 1.0.0 of the deidentified data package for the main TRoB-1 external-validation analyses. It includes all 161 assessment targets, including invalid base-model outputs, for both clinician-assisted runs. It is not a training corpus and cannot reconstruct the source articles.

The package covers main performance estimates, clinician judgment changes, between-clinician agreement, descriptive workflow times, and aggregate training distributions. Supplementary first-pass and retry audits and cross-hardware reproducibility assessments are not included.

## Files

| File | Purpose |
|---|---|
| `validation_records.csv` | Deidentified target-level reference judgements and outputs from both independent clinician-assisted runs. |
| `workflow_times.csv` | Aggregate recorded times for the two reference reviewers and both clinician-assisted runs. |
| `training_label_distribution.csv` | Aggregate judgement distribution for 4,321 training records. |
| `training_domain_distribution.csv` | Aggregate RoB 1 domain distribution for 4,321 training records. |
| `expected_results.json` | Machine-readable reference values used by the reproduction script. |
| `reproduce_results.py` | Standard-library Python script that recomputes and checks the released results. |
| `TRoB-1_minimal_reproducibility_data.xlsx` | Human-readable workbook containing the released data and summaries. |
| `DATA_DICTIONARY.md` | Variable definitions. |
| `MANIFEST.json` | Version, scope, licensing, and exclusion metadata. |
| `LICENSE` | CC BY 4.0 notice for the deidentified data. |

## Reproduce results

From the repository root, run:

```bash
python data/reproduce_results.py
```

The script uses only the Python standard library. It recomputes accuracy, macro-F1, MCC, Cohen kappa, Gwet AC1, clinician label-change counts, between-clinician agreement, and aggregate workflow times. It exits with an error if a checked value differs from `expected_results.json`. It reproduces point estimates and descriptive totals; it does not run the LLM or calculate bootstrap confidence intervals or permutation-test P values.

Headline values are:

| Clinician | Workflow | Correct, n/161 | Accuracy | Macro-F1 | MCC |
|---|---|---:|---:|---:|---:|
| 1 | Gemma 2 base, strict | 97 | 0.6025 | 0.5178 | 0.3244 |
| 1 | TRoB-1 raw | 119 | 0.7391 | 0.6033 | 0.4608 |
| 1 | Clinician-verified TRoB-1 | 132 | 0.8199 | 0.7086 | 0.6153 |
| 2 | Gemma 2 base, strict | 84 | 0.5217 | 0.4972 | 0.2857 |
| 2 | TRoB-1 raw | 118 | 0.7329 | 0.6065 | 0.4659 |
| 2 | Clinician-verified TRoB-1 | 136 | 0.8447 | 0.7763 | 0.6877 |

Final judgements from the two clinician-assisted runs agreed for 124 of 161 targets (0.7702; Cohen kappa=0.5220; Gwet AC1=0.6977).

## Deidentification and exclusions

Synthetic `record_id` and `rct_id` values link rows only within this package. The `outcome_type` column follows the reference-standard assessment target. The release excludes titles, authors, DOIs, registry identifiers, source passages, supporting quotations, model explanations, clinician comments, source-row numbers, internal job identifiers, file paths, and timestamps.

Verbatim training examples and source passages are not distributed because they may be protected by publisher copyright. Only aggregate training-set distributions are released. The training source included 4,321 RoB 1 assessment records from 1,753 unique RCTs identified across 199 systematic reviews.

## License

The deidentified files in this directory are licensed under CC BY 4.0. The license does not apply to omitted source passages or to the TRoB-1 adapter. See [`LICENSE`](LICENSE).