Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

190 Open 82 Closed
190 Open 82 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Enhancement] custom logits_processings
#181 opened Aug 4, 2024 by atawari
2
How to use local dataset(avoid downloading from huggingface)
#179 opened Jul 31, 2024 by xsgldhy
5
LLAVA-next-vid errorThe size of tensor a (2) must match the size of tensor b (7) at non-singleton dimension 0
#175 opened Jul 30, 2024 by guoyanan1g
3
Adding InfiniBench: A Comprehensive Benchmark for Large Multimodal Models in Very Long Video Understanding
#172 opened Jul 27, 2024 by KerolosAtef
3
How to combine multiple images within the prompt?
#170 opened Jul 27, 2024 by Ofir408
1
[Feature] Break down response generation and response validation that requires external API help wanted Extra attention is needed
#163 opened Jul 24, 2024 by zwcolin
6
How to evaluate the offline video datasets. help wanted Extra attention is needed
#160 opened Jul 23, 2024 by liguopeng0923
1
ConnectionError: Couldn't reach 'lmms-lab/ScienceQA' on the Hub (ProxyError) help wanted Extra attention is needed
#154 opened Jul 19, 2024 by xmu-xiaoma666
2
Results on video
#151 opened Jul 18, 2024 by liguopeng0923
TypeError: 'NoneType' object is not subscriptable
#144 opened Jul 12, 2024 by rixejzvdl649
1
Reproduction results of videollava in VideoMMe
#141 opened Jul 11, 2024 by adrianwestmoon
Reproduction results of llava v1.5 in MMBench CN
#139 opened Jul 11, 2024 by xzhouzeng
2
Error when evaluate llava-v1.5-7b[importing reka,flashattention error]
#135 opened Jul 5, 2024 by zhizhou57
1
The score is zero for seedbench/seed_video [Bug]
#132 opened Jul 1, 2024 by jungle-gym-ac
3
latency and throughput (tokens/sec) in evaluation enhancement New feature or request help wanted Extra attention is needed
#131 opened Jun 30, 2024 by hasanar1f
6
Adding MobileCaptureVQA to the benchmark
#127 opened Jun 20, 2024 by arnaudstiegler
1
[FAQ] about caching the zipped file from HF to local folder faq Questions and Answers
#124 opened Jun 20, 2024 by Luodian
Propose to print error information when import models
#123 opened Jun 20, 2024 by SDaoer
1
[Reproduce] Unable to reproduce AI2D, ChartQA and InfoVQA results for llava-1.6-mistral-7b
#122 opened Jun 20, 2024 by GoGoJoestar
1
Cannot find any code / link to worldqa dataset worldqa-related Any issue that is related to worldqa
#121 opened Jun 19, 2024 by Hasnat79
1
Can not load llava_hf models since the new updates!
#119 opened Jun 18, 2024 by hasanar1f
4
There is an inconsistency in the number of mmbench images help wanted Extra attention is needed
#116 opened Jun 15, 2024 by MuskAI
1
Unable to reproduce SQA results for llava-1.5 help wanted Extra attention is needed
#115 opened Jun 15, 2024 by clairez-cerebras
3
Error during videomme help wanted Extra attention is needed
#112 opened Jun 13, 2024 by yukang2017
5
building context takes too long question Further information is requested
#110 opened Jun 13, 2024 by simplelifetime
1
ProTip! Add no:assignee to see everything that’s not assigned.