Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

190 Open 82 Closed
190 Open 82 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Bug: Unable to calculate metrics from saved predictions using --predict_only and --from_log bug Something isn't working help wanted Extra attention is needed
#337 opened Oct 21, 2024 by zjwu0522
1
VideoMME CUDA out of memory
#336 opened Oct 21, 2024 by yhy-2000
2
[Feature Request] Update Datasets Version, so that lmms-eval can be used in Offline Environment enhancement New feature or request
#335 opened Oct 18, 2024 by jungle-gym-ac
4
How to evaluate LLaVA on TextVQA test dataset using lmms-eval?
#330 opened Oct 17, 2024 by lqxass
How to accelerate multimodal data(image, video) loading?
#324 opened Oct 15, 2024 by zhangyuygss
1
incorrect RANK (?) causing issues in 'gather', lmms_eval.evaluator
#322 opened Oct 15, 2024 by orrzohar
2
Llava-1.5-7B model replicates the results on the mmbench_cn_dev task
#313 opened Oct 10, 2024 by fmm170
1
huggingface_hub.errors.LocalTokenNotFoundError: Token is required (token=True), but no token found. You need to provide a token or be logged in to Hugging Face with huggingface-cli login or huggingface_hub.login. See https://huggingface.co/settings/tokens.
#309 opened Oct 9, 2024 by luohao123
1
[Language Tasks]Could i use llava_hf.py and Llava(Multimodal model) to run language tasks?
#307 opened Oct 8, 2024 by Violettttee
@pufanyi @kcz358
2
[New Model] Llama-3.2-11B-Vision-Instruct and Molmo
#300 opened Oct 7, 2024 by raki-1203
2
New Task Guide discussion help wanted Extra attention is needed
#299 opened Oct 6, 2024 by Luodian
Bug: MVBench-video
#298 opened Oct 5, 2024 by orrzohar
2
Discussion: Update GPT eval models discussion
#294 opened Oct 4, 2024 by zhijian-liu
2
Calculate metrics after saving output
#288 opened Sep 29, 2024 by Mohamed-Dhouib
2
Try on qwen2vl-72b, out of memory
#285 opened Sep 28, 2024 by ChinChyi
2
Bug on Qwen2-VL over videomme bug Something isn't working
#283 opened Sep 27, 2024 by ThisisBillhe
2
Bug on Qwen2-VL
#282 opened Sep 25, 2024 by yusalei
2
Batch GPT is not working help wanted Extra attention is needed
#279 opened Sep 24, 2024 by NimrodShabtay
2
[new model] qwen2-vl support
#264 opened Sep 19, 2024 by jxgu1016
5
How to set model precision when evaluating and what's the default precision? Such as "fp16" or "loat_in_8bit"
#263 opened Sep 18, 2024 by transcend-0
1
Issue of GPT4o evaluation with dense frame sampling strategy
#256 opened Sep 15, 2024 by HYUNJS
video-mme (short/medium/long) enhancement New feature or request
#254 opened Sep 14, 2024 by orrzohar
@choiszt
5
[Result] Can you provide some results on benchmarks?
#248 opened Sep 13, 2024 by Chenfeng1271
4
[BUG] Commit #0d02bad may break srt_api by mis-use of asyncio
#243 opened Sep 11, 2024 by Maxwell-Lyu
2 tasks done
4
No llavavid module in the latest LLaVA-NeXT
#242 opened Sep 11, 2024 by YitaoLiu1996
2
ProTip! Exclude everything labeled bug with -label:bug.