Description
Pretraining details mostly stay inside companies, leaving researchers guessing how data mixes and training choices matter. daVinci-LLM is an open pretraining research project that trains models from scratch and releases everything: data, training process and ablations.
It advances the science of pretraining, open sourced by SJTU's GAIR lab.
From scratch:Full pretraining.
Fully open:Data and process.
Ablations:Reproducible results.
It advances the science of pretraining, open sourced by SJTU's GAIR lab.
Features
From scratch:Full pretraining.
Fully open:Data and process.
Ablations:Reproducible results.
Tags:llm