CharlesDDDD/looped_window_600M_full_512_256_128_weighted_loss

Updated Feb 4

Note looped_window_600M full->SWA=512->SWA=256->SWA=128, weighted loss. 9 ckpts: 10B-90B tokens.

CharlesDDDD/looped_600M_bookend_2_looped4_window_128

Updated Feb 4

Note looped_window 600M bookend:2 looped4 window=128. 10 ckpts: 10B-100B tokens.

CharlesDDDD/looped_600M_4to1_looped1_window_128

Updated Feb 4

Note looped_window 600M 4:1 ratio looped1 window=128. 10 ckpts: 10B-100B tokens.

CharlesDDDD/looped_transformer_loop_count_4

Updated Feb 5 • 6

Note looped_transformer 600M pure full looped4 (loop_count=4). 10 ckpts: 10B-100B tokens.