Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models

Advancing LLM pre-training efficiency through multi-dimensional data quality assessment

🎯 Overview

The composition of pre-training datasets for large language models (LLMs) remains largely undisclosed, hindering transparency and efforts to optimize data quality—a critical driver of model performance. Meta-rater introduces a groundbreaking multi-dimensional data selection framework that doubles convergence speed and improves downstream task performance by 3.23% compared to random selection.

🏆 Key Achievements

• 📈 2x Faster Convergence: Meta-rater achieves equivalent performance using only 15B tokens compared to 30B tokens with random selection
• 🎯 3.23% Performance Gain: Significant improvement over random sampling on downstream tasks
• 🔍 Multi-dimensional Quality Assessment: Novel PRRC framework (Professionalism, Readability, Reasoning, Cleanliness)
• 📊 Scalable Framework: Benefits persist and increase from 1.3B to 7.2B parameter models
• 🏗️ Comprehensive Dataset: First fully annotated 627B-token SlimPajama with 25 quality metrics

🧠 PRRC Framework

We introduce four novel evaluation dimensions to comprehensively assess data quality:

Dimension	Description	F1 Score
🎓 Professionalism	Degree of expertise and technical knowledge required	91.57%
📖 Readability	Ease of understanding and text clarity	87.47%
🧮 Reasoning	Complexity of logical thinking and analysis	89.59%
✨ Cleanliness	Format quality and noise-free content	87.88%

🔬 Meta-rater Methodology

Our framework integrates 25 quality scores across three categories:

1. Natural Language Quality Signals (11): Rule-based measures from RedPajama
2. Data Importance Scores (3): DSIR similarity to Books, Wikipedia, and AutoMathText
3. Model-based Ratings (11): PRRC + QuRating + FineWeb-Edu + WanjuanCC

Algorithm Overview

# Simplified Meta-rater workflow
for i in range(N_proxy_models):
    weights = generate_random_weights(25)  # Random combination weights
    selected_data = select_top_k(data, weights @ quality_scores)
    proxy_model = train_small_model(selected_data)
    validation_loss = evaluate(proxy_model, validation_set)
    
regression_model = fit_regression(weights, validation_losses)
optimal_weights = find_minimum(regression_model)
final_data = select_top_k(data, optimal_weights @ quality_scores)

📊 Results

Main Results (1.3B Models, 30B Tokens)

Method	General Knowledge	Commonsense Reasoning	Reading Comprehension	Average
Random Baseline	52.79	43.94	30.02	43.78
QuRating-Educational Value	57.66	46.72	28.10	46.16
Meta-rater (All 25)	58.90	45.41	31.55	47.01

Scaling Results

Model Size	Method	Average Performance	Improvement
3.3B	Random	52.98%	-
3.3B	Meta-rater	54.71%	+1.73%
7.2B	Random	52.12%	-
7.2B	Meta-rater	55.24%	+3.12%

💡 Key Insight: Meta-rater benefits increase with model scale, demonstrating that quality data selection becomes more valuable for larger models.

📦 Available Resources

🤖 Pre-trained Models

Model	Size	Training Tokens	Selection Method	Performance on Downstream Tasks	HF Link
Meta-rater-1.3B	1.3B	30B	All 25 scores	47.01%
Meta-rater-3.3B	3.3B	100B	All 25 scores	54.71%
Meta-rater-7.2B	7.2B	150B	All 25 scores	55.24%

🎯 PRRC Rating Models

Model	Dimension	F1 Score on Test set	HF Link
Professionalism	Expertise assessment	91.57%
Readability	Text clarity rating	87.47%
Reasoning	Logic complexity assessment	89.59%
Cleanliness	Format quality evaluation	87.88%

📊 Datasets

• Annotated SlimPajama-627B:

  • 627B tokens with 25 quality scores per document
  • First fully annotated large-scale pre-training dataset
  • Ready for research and production use

• Top 30B token SlimPajama Subset selected by the Professionalism rater:

• Top 30B token SlimPajama Subset selected by the Readability rater:

• Top 30B token SlimPajama Subset selected by the Reasoning rater:

• Top 30B token SlimPajama Subset selected by the Cleanliness rater:

📝 PRRC Rating Prompts

All prompts for rating PRRC dimensions (Professionalism, Readability, Reasoning, Cleanliness) are provided in the prompts/ directory:

• prompts/professionalism.txt
• prompts/readability.txt
• prompts/reason.txt
• prompts/cleanliness.txt

🛠️ Training & Evaluation Scripts

The scripts/ directory contains shell scripts for training and evaluating PRRC raters:

• scripts/prrc.sh: Training script for PRRC raters
• scripts/evaluation.sh: Evaluation script for PRRC raters

🧑‍💻 Source Code for PRRC Raters

The src/ directory contains Python code for training and evaluating PRRC raters:

• src/train_singletask.py: Training script for a single PRRC dimension
• src/test_singletask.py: Evaluation script for a single PRRC dimension
• src/utils.py: Utility functions for data processing and model management

These resources enable full reproducibility of PRRC rater training, evaluation, and prompt-based annotation.

📈 Computational Efficiency

Meta-rater is designed for efficiency:

Process	FLOPs (×10¹⁹)	Percentage of 1.3B Training
Quality Score Rating	33.0	141%
Meta-rater Construction	0.18	0.8%
Total Overhead	33.2	142%

💡 Note: Quality scores are computed once and reused across multiple experiments. For larger models (3.3B+), the overhead becomes negligible (17% for 3.3B training).

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

📚 Citation

If you use Meta-rater in your research, please cite our paper:

@article{zhuang2025meta,
  title={Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models},
  author={Zhuang, Xinlin and Peng, Jiahui and Ma, Ren and Wang, Yinfan and Bai, Tianyi and Wei, Xingjian and Qiu, Jiantao and Zhang, Chi and Qian, Ying and He, Conghui},
  journal={arXiv preprint arXiv:2504.14194},
  year={2025}
}

🤝 Acknowledgments

• Shanghai Artificial Intelligence Laboratory for computational resources
• InternTrain Team for pre-training infrastructure support
• Community contributors for valuable feedback and improvements

📞 Contact

• Project Lead: Ren Ma ([email protected])
• Corresponding Author: Conghui He ([email protected])
• Issues: Please use GitHub Issues for bug reports and feature requests

---

⭐ Star us on GitHub if you find Meta-rater useful! ⭐

Made with ❤️ by the OpenDataLab team