Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

190 Open 82 Closed
190 Open 82 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Tasks were not found: activitynetqa
#239 opened Sep 10, 2024 by jby20180901
12
Is multi-node evaluation possible?
#234 opened Sep 8, 2024 by orrzohar
@choiszt
8
Encountering task missing issue even when specifying task from list mentioned at current_tasks.md
#233 opened Sep 6, 2024 by rodosingh
task_manager.list_all_tasks() is not showing outputs properly.
#232 opened Sep 6, 2024 by rodosingh
Cannot find a valid ref in kc/worldqa
#230 opened Sep 6, 2024 by tychen-SJTU
2 tasks
1
[logging] The program is not responding during multi-gpu evaluation
#222 opened Sep 3, 2024 by ssmisya
1
AttributeError: 'LlavaQwenConfig' object has no attribute 'mm_newline_position'
#219 opened Sep 2, 2024 by linxid
2 tasks
name 'get_model_name_from_path' is not defined
#214 opened Aug 27, 2024 by jby20180901
4
LiveBench doesn't work.
#211 opened Aug 26, 2024 by jyC23333
question about RefCOCO|+|g. Is it a region caption task?
#210 opened Aug 25, 2024 by yuecao0119
1
PiPy / Github version number mismatch
#208 opened Aug 24, 2024 by tyler-romero
[Request] Loosen dependency version constraints, such as for transformers
#207 opened Aug 24, 2024 by tyler-romero
Wrongly treating mvbench as an image task
#205 opened Aug 23, 2024 by hulianyuyy
3
Llava-v1.5 and Llava-v1.5-hf Performes on MMMU differs a lot!
#203 opened Aug 22, 2024 by hxhcreate
2
KeyError: 'Missing task mathvista_testmini' on phi3v
#202 opened Aug 19, 2024 by adnanit035
Why subtitles are set to False for LongVideoBench
#200 opened Aug 16, 2024 by aniki-ly
[Performance] Too slow MMBench evaluation
#199 opened Aug 15, 2024 by minjunsz
2
[FEATURE REQUEST] MMIU dataset
#195 opened Aug 14, 2024 by wamiq-reyaz
[BUG] Potential bug in process_docs() function pipeline
#194 opened Aug 14, 2024 by wykang
1
DocVQA Eval: IndexError: Invalid key: 5205 is out of bounds for size 5188
#193 opened Aug 14, 2024 by SachinG007
5
RefCOCO|+|g based on REC
#190 opened Aug 12, 2024 by yuecao0119
4
[ERROR] Failed to run the mathvista_test tasks
#189 opened Aug 11, 2024 by liguopeng0923
8
[Common Issue] Common issues you might encounter when using lmms-eval
#186 opened Aug 9, 2024 by kcz358
[BUG] science_qa dataset evaluation might be wrong
#185 opened Aug 8, 2024 by hxhcreate
pip install broken on Mac because of decord
#183 opened Aug 7, 2024 by philmod-h
ProTip! Add no:assignee to see everything that’s not assigned.