AReaL (Ant Reasoning RL) is a fully open-sourced, scalable, and efficient reinforcement learning training system for large language models developed at the RL Lab, Ant Research, built upon the open-source project RealHF. We fully commit to open-source by releasing training details, data, and infra required to reproduce all the models with the desired performances. AReaL aims to help everyone build their own AI agents easily and affordably. Our team loves milk tea as it is delicious, customizable, and affordable. We hope you all enjoy our project just like how you enjoy A-ReaL-milk-tea.🧋
AReaL Highlights
- 🛠️ Open & Reproducible: We will continuously release all code, datasets, and training recipes for RL training LLMs .
- 🚀 Scalability: AReaL can seamlessly adapt to different computational resource settings, ranging from 1 single node to 1K GPUs.
- 🔪 Cutting-Edge Performances: AReaL can produce models with cutting-edge reasoning capabilities. We are actively working on other domains, such as coding and agent, as well.
[2025/03/31] (v0.2, Boba) Our milestone release Boba! Please call it A-ReaL-Boba! This release includes much accelerated training with SGLang support and SOTA 7B and 32B models on math reasoning.
[2025/02/24] (v0.1) Our initial release includes reproducible results for 1.5B and 7B LRMs. Check our v0.1 technical blog.
In our boba release, we highlight the 3 most important milestones:
- Full SGLang support and a collection of efficiency improvements
- A SOTA 7B math reasoning model AReaL-boba-RL-7B and the corresponing training data AReaL-boba-106k.
- A particularly competitive 32B model AReaL-boba-SFT-32B that can be trained with extremely low cost. (Training Data: AReaL-boba-SFT-200)
For the complete training and model details, please check our v0.2 technical blog.
Thanks to a series of system-level optimizations, AReaL v0.2 improves its end-to-end training performance by up to 73%.
In the following table, we show the convergence time under different resource settings:
| Model Size | 1.5B | 1.5B | 1.5B | 7B | 7B | 32B (SFT) |
|---|---|---|---|---|---|---|
| #GPU | 8 | 32 | 128 | 32 | 128 | 64 |
| Step | 250 | 250 | 250 | 400 | 400 | 300 |
| Time (h) | ~240 | ~69 | ~27 | ~252 | ~90 | ~3.5 |
| Model | AIME 2024 | AIME 2025 | GPQA-Diamond |
|---|---|---|---|
| O1-Preview | 56.7 | - | |
| R1-Distill-Qwen-7B | 55.0 | 39.7 | 47.1 |
| Light-R1-7B-DS | 56.7 | 44.9 | 40.9 |
| AReaL-boba-RL-7B 🤗 | 61.9 | 48.3 | 47.6 |
We use R1-Distill-Qwen-7B as our base model. After RL training, the pass@1 scores on AIME 2024 and AIME 2025 improve by 6.9 and 8.6 points, respectively, achieving SOTA performance among 7B models in mathematical reasoning. We have released the training data at AReaL-boba-106k.
Although our dataset primarily consists of math and logic problems, we observed that RL training led to measurable improvements on the challenging STEM benchmark GPQA. We plan to open-source more datasets in the future, including code, STEM, and other domains. All the reported numbers are re-evaluated using our evaluation code with more details in our blog.
| Model | AIME 2024 |
|---|---|
| R1-Distill-Qwen-32B | 72.6 |
| QwQ-32B | 78.9 |
| AReaL-boba-SFT-32B 🤗 | 78.8 |
Building upon R1-Distill-Qwen-32B, we replicate QwQ-32B's inference performance on AIME 2024 using just 200 data points via Supervised Fine-Tuning (SFT). We have released the training data at AReaL-boba-SFT-200.
# Train the distilled 7B model
python3 -m realhf.apps.quickstart ppo-math \
--config examples/configs/7B-distill/ppo-7B-distill-gpus-128.yaml
# Evaluate the 7B model
python evaluation/eval_and_aggregate.py \
--model_path ${MODEL_PATH} \
--output_path ${OUTPUT_PATH} \
--data_names aime24,aime25 \
--prompt_type AReaL-boba \
--output_path outputs --temperature 1.0AReaL is under active development. We will have major releases in a weekly manner. We also highly appreciate efforts from the community as well. Here we highlight our future research and development plan.
- Support for SGLang.
- RL training with coding problems.
- Asynchronous generation and RL training.
- Optimizations for distributed training: expert parallel and zero-bubble pipelining.
- RL for vision-language models (VLM).
- Function calling and agent capabilities.
- RL training receipes for 1.5B and 7B models.
- A complete RL training receipe for 32B models.
- Sample-efficient multi-task RL algorithms.
- Agentic capabilities with end-to-end RL.
- Stable RL training for larger MOE models.
We would like to remark that major contributors are from RL Lab at Ant Research and Institute for Interdisciplinary Information Sciences, Tsinghua University.
Our team has also received invaluable assistance from the Super Computing Technology (SCT) team at Ant Group, particularly in large-scale cluster operations and maintenance.
We also appreciate all the pioneer works from the community, particularly the ReaLHF project from OpenPsi Inc. and many other projects, including but not limited to, DeepScaleR, Open-Reasoner-Zero, OpenRLHF, verl, SGLang, QwQ, Light-R1, and DAPO.
@inproceedings{mei2025real,
author = {Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},
title = {ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},
booktitle = {Proceedings of the Eighth Conference on Machine Learning and Systems,
MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},
publisher = {mlsys.org},
year = {2025},
}
@misc{areal2025,
author = {RL Lab, Ant Research},
title = {AReaL: Ant Reasoning RL},
year = {2025},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/inclusionAI/AReaL}},
}
