DictaLM 2.0
Multi-node pretraining of a Hebrew LLM with distributed data and model parallelism.
Dicta-LM 2.0 is a powerful, open-source generative large language model (LLM) for Hebrew. It is freely available for both research and commercial use under the Apache 2.0 license. The model is based on Mistral-7B-v0.1 and is considered one of the best Hebrew LLMs in its size category (up to 13 billion parameters). My main contributions were orchestrating and running multi-node pretraining. In collaboration with DICTA, MAFAT and IAHLT.