Model 2 v2 — fresh 500-update FineWeb-Edu test
22 layers, hidden 1152, intermediate 3168; tied embedding/LM head. Backbone 550,888,224 parameters plus a 1,800,199,552-parameter width-64 bigram/trigram table. Existing 60,800-token vocabulary; context 2,560, four rows per GPU, two GPUs, 32 accumulation passes = 655,360 input tokens/update.
Fused per-head QKV and GDN depthwise convolution. Distributed Muon (8 Polar Express steps, peak LR 0.024) and AdamW8bit (peak LR 0.001). BF16 backbone/table communication; FP32 accumulation and clipping. Embedding/head decay zero.
Fresh initialization and fresh optimizers. Stops at update 500 (327,680,000 input
tokens); warmup 50, decay from 400. Validation at zero/every 50/final; full
checkpoints every 1B tokens/final, latest two retained, uploaded in the background.
The original model-2 run is preserved at step 107 in
Ba2han/mini-qwen-next-model2-fineweb-500.
Pinned FineWeb-Edu revision 66b78cd134ab7b0545a1a618a1cc029f3091eb20.
Global document shuffle across four seeded source shards, exact-text deduplication,
content-hash document holdout and document-aware BOS/EOS packing. Training and
held-out documents are separate. Document/state/attention boundaries and row cuts
are preserved. RAM source cache removed after packing.
See MODEL2_REVIEW_RESTART.md for implementation and optional clipping, table LR and canonical-token ablations. This model requires the included custom code; it is not a Transformers AutoModel checkpoint.