crumb

/

Gale-medium-init

like 3

this model is an init from hand selected layers of mistral! its untrained so it cant generate text v well, i have a trained one coming soon though!! apache license, ill put it in the final repo

Thanks for the info! Are you planning to open source your distillation code?

its not actually a distillation! i just deleted layers, updated the config, and then finetuned on a text dataset w/ the huggingface trainer, I can probablt clean up the code and release it anyway though!