Question about task difficulty in the dataset

[chunhuizhang]

Hi, I noticed that the difficulty in the dataset have different range (e.g. MATH in 1~5, and AIME in 0~10). Does the difficulty in MATH, AIMC and AMC share the same standard ?

[michaelzhiluo]

Ultimately depends on the LLM as a difficulty judge, see the prompt in system_prompts.py and the difficulty judge in deepscaler/data/preprocess. We average the difficulty score over pass 8.