Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

292 Open 379 Closed
292 Open 379 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Bug] Pass@K测评报错：TypeError: '>' not supported between instances of 'list' and 'int'
#1991 opened Apr 1, 2025 by bruno686
2 tasks done
[Bug] F1-score file found error
#1989 opened Mar 31, 2025 by GenerallyCovetous
2 tasks done
@bittersweet1999
5
[Bug] 最新版本的默认summarizer 会在平均分上产生冗余结果
#1984 opened Mar 29, 2025 by linbeyoung
2 tasks done
@tonysy
[Bug] The ifeval evaluation on 541 cases returns unexpected low score.
#1983 opened Mar 29, 2025 by lebronjamesking
2 tasks done
@tonysy
[Feature] How about add bfcl evaluation to the repo
#1981 opened Mar 28, 2025 by lebronjamesking
1 task
@tonysy
2
[Bug] [Bug] LiveStemBench数据集评测时，推理结果正确。但是启动打分程序时报错 mmengine KeyError: 'cfg or default_args must contain the key "type", but got {}\nNone'
#1980 opened Mar 28, 2025 by BigFishLi
2 tasks done
@tonysy
1
[Bug] 多卡测试llama-3-8b-vllm，精度为0
#1979 opened Mar 27, 2025 by normlpl
2 tasks done
[Bug] How generic datasets can be evaluated using llm_judge?
#1977 opened Mar 27, 2025 by GenerallyCovetous
2 tasks done
@Zhudongsheng75
6
[Bug] opencompass install error on python 3.11
#1976 opened Mar 27, 2025 by wanghaitaoofzjlab
2 tasks done
1
@Zhudongsheng75
2
[Feature] 增加 benbench 评测集
#1975 opened Mar 25, 2025 by linbeyoung
1 task
@Zhudongsheng75
1
[Feature] Does OpenCompass support multiple machines with multiple gpus (e.g. 2 machines with 8 gpus total 16 gpus)?
#1972 opened Mar 25, 2025 by GenerallyCovetous
1 task
@Zhudongsheng75
3
[Bug] configs/datasets/GaokaoBench/README.md 文档数据问题
#1965 opened Mar 24, 2025 by yodhcn
2 tasks done
[Bug] extract_non_reasoning_content error when predict is null
#1964 opened Mar 20, 2025 by simplew2011
2 tasks done
@acylam
2
[Feature] Can base model start the service via openai for ppl evaluation?
#1961 opened Mar 19, 2025 by GenerallyCovetous
1 task
@MaiziXiao
[Bug] Unexpected output when using "attn_implementation=eager"
#1955 opened Mar 18, 2025 by The-kamisato
2 tasks done
@liushz
2
[Feature] 希望增加多机多卡测试的功能
#1950 opened Mar 17, 2025 by HenryBao91
1 task
@bittersweet1999
4
[Bug] If-eval deepseek-r1 score may not corret
#1949 opened Mar 17, 2025 by MeJerry215
2 tasks done
@bittersweet1999
3
opencompass/tasks/outer_eval/alpacaeval.py中 example.json 内容是什么格式？框架中没有看到样例
#1948 opened Mar 16, 2025 by liuchunming033
@tonysy
[Feature] aime2025的llm测评
#1946 opened Mar 14, 2025 by GaoJieCN
1 task done
@tonysy
3
[Bug] huamneval测试Deepseek-R1结果过低
#1943 opened Mar 13, 2025 by Gannn12138
2 tasks done
@acylam
[Bug] commonsense_qa和strategyqa的results为空
#1941 opened Mar 13, 2025 by gxlover0625
2 tasks done
@acylam
2
[Bug] VLLM推理时参数不匹配
#1935 opened Mar 12, 2025 by c-box
2 tasks done
@MaiziXiao
1
[Bug] mmlu_pro结果正则提取出错
#1933 opened Mar 11, 2025 by wangzhaode
2 tasks done
@liushz
4
[Bug] LiveCodeBench评测报错
#1930 opened Mar 10, 2025 by AllenShow
2 tasks done
[Bug] math_500_gen评测报错
#1929 opened Mar 10, 2025 by AllenShow
2 tasks done
2
ProTip! Add no:assignee to see everything that’s not assigned.