TRoB-1 Main Analysis Data
This directory contains version 1.0.0 of the deidentified data package for the main TRoB-1 external-validation analyses. It includes all 161 assessment targets, including invalid base-model outputs, for both clinician-assisted runs. It is not a training corpus and cannot reconstruct the source articles.
The package covers main performance estimates, clinician judgment changes, between-clinician agreement, descriptive workflow times, and aggregate training distributions. Supplementary first-pass and retry audits and cross-hardware reproducibility assessments are not included.
Files
| File | Purpose |
|---|---|
validation_records.csv |
Deidentified target-level reference judgements and outputs from both independent clinician-assisted runs. |
workflow_times.csv |
Aggregate recorded times for the two reference reviewers and both clinician-assisted runs. |
training_label_distribution.csv |
Aggregate judgement distribution for 4,321 training records. |
training_domain_distribution.csv |
Aggregate RoB 1 domain distribution for 4,321 training records. |
expected_results.json |
Machine-readable reference values used by the reproduction script. |
reproduce_results.py |
Standard-library Python script that recomputes and checks the released results. |
TRoB-1_minimal_reproducibility_data.xlsx |
Human-readable workbook containing the released data and summaries. |
DATA_DICTIONARY.md |
Variable definitions. |
MANIFEST.json |
Version, scope, licensing, and exclusion metadata. |
LICENSE |
CC BY 4.0 notice for the deidentified data. |
Reproduce results
From the repository root, run:
python data/reproduce_results.py
The script uses only the Python standard library. It recomputes accuracy, macro-F1, MCC, Cohen kappa, Gwet AC1, clinician label-change counts, between-clinician agreement, and aggregate workflow times. It exits with an error if a checked value differs from expected_results.json. It reproduces point estimates and descriptive totals; it does not run the LLM or calculate bootstrap confidence intervals or permutation-test P values.
Headline values are:
| Clinician | Workflow | Correct, n/161 | Accuracy | Macro-F1 | MCC |
|---|---|---|---|---|---|
| 1 | Gemma 2 base, strict | 97 | 0.6025 | 0.5178 | 0.3244 |
| 1 | TRoB-1 raw | 119 | 0.7391 | 0.6033 | 0.4608 |
| 1 | Clinician-verified TRoB-1 | 132 | 0.8199 | 0.7086 | 0.6153 |
| 2 | Gemma 2 base, strict | 84 | 0.5217 | 0.4972 | 0.2857 |
| 2 | TRoB-1 raw | 118 | 0.7329 | 0.6065 | 0.4659 |
| 2 | Clinician-verified TRoB-1 | 136 | 0.8447 | 0.7763 | 0.6877 |
Final judgements from the two clinician-assisted runs agreed for 124 of 161 targets (0.7702; Cohen kappa=0.5220; Gwet AC1=0.6977).
Deidentification and exclusions
Synthetic record_id and rct_id values link rows only within this package. The outcome_type column follows the reference-standard assessment target. The release excludes titles, authors, DOIs, registry identifiers, source passages, supporting quotations, model explanations, clinician comments, source-row numbers, internal job identifiers, file paths, and timestamps.
Verbatim training examples and source passages are not distributed because they may be protected by publisher copyright. Only aggregate training-set distributions are released. The training source included 4,321 RoB 1 assessment records from 1,753 unique RCTs identified across 199 systematic reviews.
License
The deidentified files in this directory are licensed under CC BY 4.0. The license does not apply to omitted source passages or to the TRoB-1 adapter. See LICENSE.