OpenLM Awesomes

An up-to-date awesome list of open-source large language models, instruction & RLHF datasets and high-performance codebases.

• Open Large Language Models
• Open Instruction-following Large Language Models
• Open Instruction Tuning and RLHF Datasets
• High-performance Open Instruction Tuning Codebase

Open Large Language Models

Model	Stars	Organization	Language1	Checkpoints
LLaMa2		Meta AI	Multi-Lang	7B & 13B & 30B & 65B
OPT		Meta AI	Eng.	125M & 350M & 1.3B & 2.7B & 6.7B & 13B & 30B & 66B & 175B3
BLOOM	--	BigScience	Multi-Lang	560m & 1.1B & 1.7B & 3B & 7.1B & 176B
GLM4		Tsinghua	CHI. & Eng.	2B & 10B & 130B5

Open Instruction-following Large Language Models

Model	Stars	Organization	Based Model	Checkpoints
BLOOMZ	--	BigScience	BLOOM	560m & 1.1B & 1.7B & 3B & 7.1B & 176B
Alpaca		Stanford	LLaMa	7B
Vicuna		lm-sys	LLaMa	7B & 13B
Chinese-Vicuna		Facico	LLaMa	7B

Open Instruction Tuning and RLHF Datasets

Dataset	Language	#Samples	Annotation Type	Online Link
alpaca_chinese_dataset	Chinese	52K	GPT-generated & Human-annotated	hikariming/alpaca_chinese_dataset
BELLE/data	Chinese	1.5M	GPT-generated	BELLE/data/1.5M
pCLUE	Chinese	1.2M	Formated from Existing Datasets	CLUEbenchmark/pCLUE
Med-ChatGLM/data	Chinese	7K	GPT-generated	SCIR-HI/Med-ChatGLM
COIG	Chinese	181K	Follow this paper	BAAI/COIG

High-performance Open Instruction Tuning Codebase

• DeepSpeed Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
• ColossalChat is a project to implement LLM with RLHF, powered by the Colossal-AI project.
• LMFlow: An extensible, convenient, and efficient toolbox for finetuning large machine learning models, designed to be user-friendly, speedy and reliable, and accessible to the entire community.

Footnotes

1. Languages used in pretraining texts. CHI is short for Chinese. Eng is short for English. ↩

2. Direct access to the LLaMa models is currently not possible due to Meta AI's policies. However, interested individuals can apply to access the models by filling out a form, which can be found at here. Alternatively, the checkpoints for the LLaMa models can be accessed via the following link on the Decapoda Research page of Huggingface. ↩

3. Direct access to the OPT models with 176B params is currently not possible due to Meta AI's policies. However, interested individuals can apply to access the models by filling out a form, which can be found at here. ↩

4. There are other GLM models (with different pretraining corpus) that are available at the THUDM page of Huggingface. GLM models include different size: 10B/2B/515M/410M/335M/110M (English), 10B/335M (Chinese). In above Table, GLM models with Engilish are provided. ↩

5. Direct access to the GLM model with 130B params is currently not possible due to THUDM's policies. However, interested individuals can apply to access the models by filling out a form, which can be found at here. ↩