-
Notifications
You must be signed in to change notification settings - Fork 288
add process sync in evaluation metric computation via a temp file in lmms_eval/evaluator.py #142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Luodian
pushed a commit
that referenced
this pull request
Sep 1, 2024
Luodian
added a commit
that referenced
this pull request
Sep 1, 2024
* chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd78 Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980 Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f3 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd18585 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86 Merge: e6844db a5c1869 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c1869 Merge: 2ebec77 557083a Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77 Merge: 211bfed b23d349 Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfed Merge: 7c208b7 79514ee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514ee Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e1 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed38173 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b7 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40de Merge: e19b43a ba7081c Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a Merge: 11fd7e3 a0de897 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3 Merge: 383e7fe a752259 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a752259 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fe Merge: 06fa000 ed2e7f7 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f7 Merge: 03947e1 06fa000 Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de897 Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c Merge: fce85f1 903b042 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042 Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86 Merge: ebe7217 fce85f1 Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f4806 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e1 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef2474 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1 Merge: dbe6329 d94f83c Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe6329 Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83c Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 4587665 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f4 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877b Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd6 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c52 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb5295 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651f Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b625 Merge: 22a4958 050b2c3 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c3 Merge: 74facb4 ef30651 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef30651 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab677 Merge: dbfb238 74facb4 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb4 Merge: 8ba192f d5df72d Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72d Merge: 5bf59ed 8ba192f Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 98b3955 Merge: a056f11 be9dada Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit a056f11 Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit 8ba192f Merge: 7cc2890 be9dada Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit be9dada Merge: 62ea8ce 7cc2890 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit 62ea8ce Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit 7cc2890 Merge: 4bc7224 ea14cd4 Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit dbfb238 Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit ea14cd4 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit 4bc7224 Merge: 2797987 bf14cb8 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit bf14cb8 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit 6248113 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 2797987 Merge: 63d82f1 66d4bb2 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit 66d4bb2 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 63d82f1 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit 44a3379 Merge: 5ed0035 0ce46d0 Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 0ce46d0 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 46a88d8 Merge: 47b13b9 5ed0035 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit 47b13b9 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 5ed0035 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit ed88068 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit fea3806 Merge: d99a24a 05dc8e8 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit 05dc8e8 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit cbeee20 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f00d549 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 3415633 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit 50575a9 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit c9b2252 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit 465bd42 Merge: e43bd84 d99a24a Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit e43bd84 Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d99a24a Merge: 374590b a66003b Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit a66003b Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit ee91f27 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 326b969 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit cd050d4 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 205721e Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit db8e718 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 427dabb Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 043b483 Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit e1f04db Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 96e8d98 Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit 374590b Merge: 504685e cb3b9ce Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit 504685e Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit cb3b9ce Merge: c9793b3 67b64ea Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit c9793b3 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit 67b64ea Merge: 8ee7848 5fd6845 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit 5fd6845 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 8ee7848 Merge: 747e197 6fefaf7 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit 747e197 Merge: 4854a34 0584307 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit 6fefaf7 Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit 5f4fe36 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit 0584307 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit 0581ab3 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 4854a34 Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit d224794 Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 453e793 Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 909edd6 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 7c1ac97 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 811301c Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 71401ba Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 24dc435 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 616edf4 Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit 4c5a99e Merge: 45c05b2 b05c3e2 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit b05c3e2 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit c200897 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 7f9fb6b Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 45c05b2 Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit 53f013e Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit 22520a9 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit d1eefb1 Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit b2b4dbd Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 662f05c Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 0932932 Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit 557a6a3 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 6aeb550 Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit aea80e6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 3c12a08 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit 82317a6 Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit a8bba1c Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit caa5893 Merge: c094448 423b006 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit c094448 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 64f07e4 Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 8aaa828 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 7847dc4 Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit 3e56b4f Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit fa3ff92 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 423b006 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit b7fd7a9 Merge: 986139a c5a130b Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 986139a Merge: b46239c 8d3526c Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit b46239c Merge: bc69a74 373265f Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit bc69a74 Merge: eef3aeb 626e8a9 Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 626e8a9 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit eef3aeb Merge: c4e9dd9 9bca441 Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 9bca441 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit 7687495 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 4eebd3e Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 35fe80b Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 955bd06 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit c4e9dd9 Merge: d8a3a99 319afcc Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit 319afcc Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 2f3811c Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit 28962cb Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit e457cfb Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit d8a3a99 Merge: 3dcd015 ed17129 Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit ed17129 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit cd87420 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit 8557367 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit 3dcd015 Merge: 95df9fe 743673a Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit 743673a Merge: c1a5472 95df9fe Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit c1a5472 Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit 373265f Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit d853051 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit 22a4958 Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c) commit 8d3526c Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 557083a) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]>
Luodian
added a commit
that referenced
this pull request
Sep 1, 2024
* fix doc * [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) * Create README.md * Add files via upload * Add MathVerse * Fix typo in qwen_vl that was causing "reference before assignment" * convert contexts to list if necessary and remove unnecessary construction of `questions` * refactor query construction for clarity * Create ScreenSpot on clean branch * Update README to reflect new tasks * Add README file specific to ScreenSpot * slight update * Init webSRC * Draft README for WebSRC * Update main README with new task names * Draft and validate websrc eval on dev split * Add code to enable compilation of submission for WebSRC test split * Bugfix: WebSRC should be token-level F1 NOT character-level * Add qwen vl api * Fix llava conv template for llama3 * Fix llava_hf generation for 1.6 * Parse result for llava_hf 1.6 * Add model_name parameter to Llava constructor * Fix endless warning for llava_hf generation * Fix llava_hf image tokens number issue * Create LICENSE * Update LICENSE * Update LICENSE * Better task list_with_num * Fix idefics2 llava in the wild bugs * Remove redundant code in fuyu * Fix instructblip qformer size mismatch and multi-images problem * Comment out parse result in xcomposer * Comment out Spice in caption task so that don't need to download stanford nlp model * Update gitignore * Add separated pope tasks by category * Fix pope random name in pope full * Set printing info for llava_hf to debug level * Adding Phi3v model. * Adding prompt arguments for Phi3v on MathVista-TestMini * Adding documentation of Phi3v class. * [Fix] import issues of multilingual llava and olympiadbench * fix compatibility issue of older version llava * add upd * add upd * add upd * add upd * add upd * add upd * Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> * merge model_specific_prompt_kwargs and dataset_name into each task yaml * Add MathVerse in README.md * slightly change query_prompt for the reproduction * update utils.py for leaderboard submission * add conbench * update README * Update README.md * init include vcr * modify the form of VCR * switch logic * add crossed_text to vcr_wiki output * include the try-except logic for spacy * update vcr_wiki tasks * update vcr_wiki tasks in README.md * include std and confidence interval * update gpt-3.5-turbo version * update gpt-3.5-turbo version * chore: Remove unnecessary files and code related to live_bench and sft_eval tasks * Bump version to 0.2.0.dev0 * chore: Update lmms-eval to support video evaluations for LLaVA models * Update llava conv_template in lmms_eval/models/llava.py * Update image alignment in README.md * chore: Update lmms-eval to support video evaluations for LLaVA models * chore: Update lmms-eval to support video evaluations for LLaVA models * Update README.md * Update README.md * update aggregation function for vcr_wiki * update README.md * Update README.md * update version * add II-Bench * fix dataset_path * Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image * add tinyllava * LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) * fix #117, allow auto download with tar format videos * fix #117, allow auto download with tar format videos * fix typo * feat: Add support for auto downloading tar format videos * Release llava-wilder * chore: Update dependencies to fix potential risks and improve compatibility * tutorial * docs * update preparation * small fix * small fix * lint * to sh script * update readme * Remove handling non-visual loop in llava * Add llava_hf back to registry * Update README.md * Update README.md * update ablation for videomme datasets * chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. * chore: Remove unused models from lmms_eval package * feat: Allow loading model configurations from other packages * feat: Allow including external tasks from plugins * chore: Add loguru for logging in lmms_eval package * Remove unnecessary lines since use batched visuals now in llava * Add longva * Revise model registry for llava_hf and longva * Delete unnecessary lines * Remove unnecessary lines for video llava * Update pyproject.toml * Update activitynetqa_generation.yaml * Fix vid mme post prompt issue * new task gqa-ru * add mmbench_ru_dev * change prompt to ru * create new task vitatecs * Update README.md * Add wild vision 0617 * Hardcode to keep image for wild vision * Fixing scoring logic * Fixing dataset name * Fixing handling None filtered score * Add detailcaps * Add install capture_metric in env * Add files via upload * feat: Add tie_weights parameter to Llava model initialization * Upgrade lmms-eval to support more models and evaluation tasks * Upgrade lmms-eval to version 0.2.1 * Rename xcomposer 4KHD * chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py * Update utils.py * Update _default_template_vcr_yaml * add process sync via temp file in lmms_eval/evaluator.py * Update utils.py * Update _default_template_vcr_yaml * Add muirbench * Squashed commit of the following: commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83cb3f08b61a2c75cc4326e58792100605b3 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159ff35db330536c0b6dfb4b0a3b24142209 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 45876652a877a8006b828f32f5cc4660629f9190 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651b8c54d36cd94169e3d376f5ed225a195a Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f49b72790b081f981e0e6147131542f7f68 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877bfe5ac590cdbe92dd74d18f83a2091f748 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b8557d5894976448a5c559ed7a66a6cf16 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851d42ba31f5da2a7a65e91499db45174dbc Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9718f287ad433afc90e340b5e17a89c1ed Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a1fe04cfb783dc31d4fbdef3bda0ce19e4 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd621c76a483ff98f8c7eb78f64753da683a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889ba02d819979c7922f6336cf4f1f718f65 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c520c2a3dad26d2b2ca5c4ed4db05a493c73 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651fc9daf97e2f57831ed6e6e7ee7ead7d555 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b6259828212fcba954cdeb8cf90d6e5daabf8 Merge: 22a4958e 050b2c37 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c370017e9b97475dd6cf01fd051b5ca5c86 Merge: 74facb41 ef306512 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef306512e5135f76dffa383f600b8733015836e8 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab67732a4238097725deddf867fb1946ffee40 Merge: dbfb2387 74facb41 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb41a826691dfce4458cf1d8659b34fc5bf5 Merge: 8ba192f9 d5df72de Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72de2d03108d6b365818ecc3551ac9aa6302 Merge: 5bf59ed2 8ba192f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed250da98a408a94e214a73caa400cba842 Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 98b3955cb808e36303c030aea78eb037d1ec59ce Merge: a056f118 be9dada8 Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit a056f118704eccec86ce32ab86981ce4bc1e1deb Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit 8ba192f94edf5d99598983445d5faa4f8807c49f Merge: 7cc28907 be9dada8 Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit be9dada8b4189c53c08e1674ab273242cf2f80a0 Merge: 62ea8ceb 7cc28907 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit 62ea8ceb223ef2b51ebab2bcd50d5cf339c35cfe Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit 7cc28907edbb4eb58ee1398772a48110ea35dd96 Merge: 4bc7224d ea14cd4b Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit dbfb23873979f789477f4797ee2d6071e0fd921e Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit ea14cd4b361f4c95b3665cbdb95bc51754090eb5 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit 4bc7224dcd27fe8b288bfc3fed4d7a9da9635658 Merge: 2797987f bf14cb85 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit bf14cb8527b2b7ac438a36567a875168bc02d294 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit 6248113f4e11a0ac396d31fa1b032a142fea8cb4 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 2797987f5b88b87bd172714b678a75a1d8051826 Merge: 63d82f1f 66d4bb2d Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit 66d4bb2d9c9afbbdea40196d4ad80e214d0b14b6 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 63d82f1ff11eb430d91a15d6788a1f0b4d596850 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit 44a33799671cb668f55366d5e5a4ddb051a3a1b4 Merge: 5ed00356 0ce46d08 Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 0ce46d088e473d12d63de44f17c67dceab25658c Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 46a88d8b0199ed44d2ff459fb372f2e006960cea Merge: 47b13b9b 5ed00356 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit 47b13b9b320d36ac53b3622557e31239f7c22621 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 5ed00356676cf5d0ff056cf27d1b519b8e303ff7 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit ed8806839db5988ced672bd162b7b046edb4863a Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit fea3806026932a6e2bd6e538bcc413e33abdf245 Merge: d99a24ab 05dc8e85 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit 05dc8e853eab7c6bc782a1e2662d2efe7422f767 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit cbeee20bc4ffb510a2b23d96cdaf4077be7c2a9e Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f00d5498b69dd4f7e54c907ac906abc7c128f000 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 34156335db74cef9e3f0915d7172fd6b22456c15 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit 50575a950736bc8fc1e191310314cbb5fdff5720 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit c9b2252fb8a15dd04252af5e6b4613855afd6ada Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit 465bd4205e8097e9c037b24a3ed08dd6a7694efa Merge: e43bd840 d99a24ab Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit e43bd840b63eb499856e36d9d2ba45c924abcead Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d99a24abd06df10d07e5a4d0ad5030613f92f2e7 Merge: 374590be a66003be Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit a66003befe4175824a1be6ed59f5f5b88c15f792 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit ee91f272985f32eeb9cd6faa41afdd8eb49cac30 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 326b9694fc77398592b8caf3ba0bc2e2bb903813 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit cd050d4a721d01a2ace0cd030cf7f8dc67eb8c4d Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 205721e0aad76dde30255e56149bbed121883356 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit db8e718b502469e8536ee359c5559de87635ffc7 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 427dabb790118f538b64e4e5bf6a7aab9689b3d9 Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 043b483eb55f7be4fea75c9bc0b9b03d251b109b Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit e1f04db8f58dd10591fde335ea13f74cda7c79bd Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 96e8d9867c9549ab7490f4b12cfeb6a06238e0aa Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit 374590be62f988a76cf6704cfe394cd8ae7d4cb6 Merge: 504685e2 cb3b9ce7 Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit 504685e20b17659b913cf46f3012c16bf429e09d Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit cb3b9ce71411da862ff01342a9122a3c656ffbd1 Merge: c9793b38 67b64ea4 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit c9793b3883714f254a700230b7bee781d6110e73 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit 67b64ea44a5a39d96c7a196a8a8345a7486bd912 Merge: 8ee7848a 5fd68451 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit 5fd684515c55ef643726c1b6c720c7cbd2183ba1 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 8ee7848aaa6383aa1f919c3f21199c81db3fff89 Merge: 747e1978 6fefaf7c Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit 747e19782996065cdce7157ee8c5e15beb5b6c59 Merge: 4854a34d 05843072 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit 6fefaf7cea504e35583ee7217449da290295a7a4 Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit 5f4fe360def1c48ea0cb1da6409d192784882308 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit 05843072d608b970bcada1cd0db65a3c80864060 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit 0581ab3cfb362e2024988b46fbbb00324f1233c9 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 4854a34d4d37efb5e201f2691ecdb054590cf20b Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit d224794c49520f4d28a31862cf977198cd6cbc5e Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 453e7936424220f02b99517059ca71babfbe5f5a Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 909edd6769ddcf8a546be4fdd129416687516878 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 7c1ac9706cafc4801fa4da181d2f610b7838c7b8 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 811301c5280ddd74986645086f026ab730c8848c Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 71401bafd1d515f704f86ab4817a758542bc4672 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 24dc435908d921e9f1a5706e3141b12e5d838d18 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 616edf43731415b35f0f5e97748ed2e017a2891d Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit 4c5a99e21a63fb0ee1c7d15546d18066e1d9894b Merge: 45c05b2b b05c3e22 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit b05c3e222fabd308dd7af4e04c1c6a0812962fe6 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit c2008971308ce8168d57c24d00b725832f099244 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 7f9fb6bcc6cd24a7b8011b8753d0ea98cc2451fd Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 45c05b2b2bece76e06849a52a0d034f9c0ac2367 Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit 53f013ed8278776551ca992562253387cc9968d2 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit 22520a95f13334b75eee0cf0387151067a6bf516 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit d1eefb1565014b47287ffa6b350229062f8f602f Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit b2b4dbd2dc13432c79208db35abf7f55c97f1790 Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 662f05ce4c62a46a83f819d3a5925a9bd20059b5 Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 09329322916bfbb604d72ddaf50441a0947f8805 Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit 557a6a3b15e07e506bc05e2cc76ff6a2f8c93964 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 6aeb5504e74ed1980b53700d8e4d4dcf7d1b38fc Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit aea80e6a71f716951353e1e5d68380243396b4d6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 3c12a080d66b9c38f615b961befca7c30f82fa39 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit 82317a635a4978b32e095a06cc295d0ae23661c2 Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit a8bba1cdb51061a0d27bf9a98cca1505b5c58ea5 Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit caa5893b5fd2c1d32c72b97f371ccd9a8d9ec3a0 Merge: c0944486 423b0060 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit c09444860362a136f17641f8b2a1f91c2bbc3715 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 64f07e497f53e5bcbe9e8fb5830cc7a1daaf7ff1 Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 8aaa828108da8514dd9cd23a9d6d83a8b67f2d65 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 7847dc4d8efe60605102414bb071b1da9851228e Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit 3e56b4f92db39a2ce92903b0c43a34f1d14d59ec Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit fa3ff92b07ea5aaa633a2039818c310744f84d07 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 423b00606aa77fd6b324c19e3d480b73ab852db6 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit b7fd7a9f7aa3c0e1e50374047dfffc46a7462b90 Merge: 986139a9 c5a130b6 Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 986139a9a31154679bdea029b09639f84712db27 Merge: b46239ca 8d3526c0 Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit b46239cabab7b545ec99d9eae6c851e531b18374 Merge: bc69a744 373265f2 Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit bc69a744d2cffeb06eba62e843bcc7869e27613a Merge: eef3aeb6 626e8a91 Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 626e8a91a4af2dd5dd774fc130cc2f4d74b2bc37 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit eef3aeb6ab589bb1d5045af5b5c1984a69402d19 Merge: c4e9dd9f 9bca4413 Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 9bca441376325173128e5c50087f068e519c48da Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit 7687495b1ed552eeba088cb9ad5aaf1170e7fff9 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 4eebd3e5d7ab3b8c3116eea57318db72d2ce32bb Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 35fe80b67656114a8824eb59574089663bdc4c9a Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 955bd0635cc6c14a96ad869f1002e6dbefdc5071 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit c4e9dd9f6e40e8586587c4a75987aa109a37f14b Merge: d8a3a99f 319afccb Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit 319afccbe713ddf40a8a6fa28501e64c0ad34725 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 2f3811ca1bbad6a441016b05fde09a571900fca8 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit 28962cbe83631ec5d6481aaea4907a7c96fec848 Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit e457cfb4f2d6869e8367d6d5b03ad25ee4acc363 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit d8a3a99ff6142fe101fa3c188cc7f29593c44345 Merge: 3dcd0158 ed171293 Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit ed171293d1e82075c5c6a847fc91ecbfd45cf89f Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit cd874201c46f32a2903ddffae85f9db73e14adfd Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit 85573674e90c8d505312ba18c5102e0051255078 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit 3dcd01582b719555bcf8eb25d91cc5e42abd2c5f Merge: 95df9fee 743673a1 Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit 743673a1419b6e729e18c96f148745cc739d4c71 Merge: c1a54721 95df9fee Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit c1a5472135c3b84061b64d997ab50dda0412ba4f Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit 373265f24e7a89cbd49ab724a2e388cc0930be78 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit d8530514a5ef9378d2adeaceb228b60ec25a6718 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit 22a4958e993463edff352ac033014f9a485706cc Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) commit 8d3526c0869f0ad7747ff6bb02441140792b461c Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]> commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench commit 4f8db1d37b1f824432927e74d6d82e06bb5aaed1 Author: Pu Fanyi <[email protected]> Date: Fri Jul 12 17:26:50 2024 -0700 Upload live_bench results (#140) * upload results * add a readme * chore: Update upload_results.py script to use shell syntax * Update upload_results.py * Update upload_results.py commit 18f3812c4f9af2e49af6b50e8afe7f607b8a75d6 Author: Pu Fanyi <[email protected]> Date: Wed Jul 10 18:13:43 2024 -0700 Load tasks only one time (#139) * chore: Initialize tasks only once to avoid re-initialization * chore: Initialize tasks only once to avoid re-initialization * chore: Refactor task initialization to avoid re-initialization * chore: Update task initialization to fix include_path issue * chore: Update task initialization to fix include_path issue * chore: Remove unnecessary line in muirbench.yaml * chore: Remove unnecessary line in muirbench.yaml and update gitignore * chore: Update lmms_eval to use correct variable name for world size * Update mmvet * chore: Update lmms_eval to use correct variable name for world size * chore: Remove unused lmms_eval configuration file * refactor: Update lmms_eval to handle both image and video tasks This commit updates the `Llava_OneVision` class in `llava_onevision.py` to handle both image and video tasks. It introduces conditional logic to differentiate between the two types of tasks and process the input accordingly. Additionally, it sets the image aspect ratio based on the number of visual inputs and the configuration settings. Closes #123 * Fix llava onevision loglikelihood video bug (cherry picked from commit f96e3e69fe86dcd9cb33d2bc18cc4ff2003de8be) * refactor: Update mm_spatial_pool_mode to use bilinear interpolation This commit updates the `mm_spatial_pool_mode` parameter in the `Llava_OneVision` class of `llava_onevision.py` to use bilinear interpolation instead of the previous average pooling mode. This change improves the spatial pooling process for the model. Closes #456 * chore: Update pyproject.toml with protobuf dependency version 3.20 * Squashed commit of the following: commit e106f49ceeb295fd4c89a0877073bc01b4b77c5f Author: Fanyi Pu <[email protected]> Date: Thu Jul 25 08:14:03 2024 +0800 livebench_july commit a16295653fdda20d5e8c41c549d731ec422013e3 Author: Fanyi Pu <[email protected]> Date: Mon Jul 22 15:09:58 2024 +0800 websites commit 2cdc06ffe6ba53a4c707c1acf9fc5f2e7886b2b8 Author: Fanyi Pu <[email protected]> Date: Sun Jul 21 15:34:39 2024 +0800 everything use gpt-4o commit e67538d65526c58903d9e62d1914ebd39924ab67 Author: Fanyi Pu <[email protected]> Date: Sun Jul 21 14:29:55 2024 +0800 chore: Update dataset capture settings in create_dataset.py commit 0a3bb33d37cda05bb7bfba4ecf873c2860092a03 Author: Fanyi Pu <[email protected]> Date: Sun Jul 21 01:58:14 2024 +0800 gpt-4-turbo => gpt-4o commit 837f8b0400f04f4367f8f8f954afd64666d62fc6 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 16:48:04 2024 +0800 chore: Update dataset name and version for live_bench task commit fa58e730978b5536005c8bd0291abbeddd761205 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 15:05:13 2024 +0800 generate data commit faa96227a7af7bd6546578b2db68dce2acbc2c0c Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 13:15:18 2024 +0800 fix commit 60ea7ddb4fcd9f08013cd0d5b9dd8090f7e6b83e Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 13:12:31 2024 +0800 fix bugs commit 827d69d0bf967f5d69bfbee9848b4d568ca853b1 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 08:39:41 2024 +0800 use claude to generate commit b7e2619d1a51144cd434861ac151187aed82c8c4 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 07:36:59 2024 +0800 extract information commit f87d55d47cb0d6653765e9e3f988f4bc186f7d4c Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 07:24:07 2024 +0800 claude auto detect json mode commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c…
kcz358
added a commit
that referenced
this pull request
Sep 1, 2024
…s. (#218) * Load tasks only one time (#139) * chore: Initialize tasks only once to avoid re-initialization * chore: Initialize tasks only once to avoid re-initialization * chore: Refactor task initialization to avoid re-initialization * chore: Update task initialization to fix include_path issue * chore: Update task initialization to fix include_path issue * Upload live_bench results (#140) * upload results * add a readme * chore: Update upload_results.py script to use shell syntax * Update upload_results.py * Update upload_results.py * Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * if no response directly return 0 (#142) * merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83cb3f08b61a2c75cc4326e58792100605b3 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159ff35db330536c0b6dfb4b0a3b24142209 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 45876652a877a8006b828f32f5cc4660629f9190 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651b8c54d36cd94169e3d376f5ed225a195a Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f49b72790b081f981e0e6147131542f7f68 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877bfe5ac590cdbe92dd74d18f83a2091f748 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b8557d5894976448a5c559ed7a66a6cf16 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851d42ba31f5da2a7a65e91499db45174dbc Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9718f287ad433afc90e340b5e17a89c1ed Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a1fe04cfb783dc31d4fbdef3bda0ce19e4 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd621c76a483ff98f8c7eb78f64753da683a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889ba02d819979c7922f6336cf4f1f718f65 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c520c2a3dad26d2b2ca5c4ed4db05a493c73 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651fc9daf97e2f57831ed6e6e7ee7ead7d555 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b6259828212fcba954cdeb8cf90d6e5daabf8 Merge: 22a4958e 050b2c37 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c370017e9b97475dd6cf01fd051b5ca5c86 Merge: 74facb41 ef306512 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef306512e5135f76dffa383f600b8733015836e8 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab67732a4238097725deddf867fb1946ffee40 Merge: dbfb2387 74facb41 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb41a826691dfce4458cf1d8659b34fc5bf5 Merge: 8ba192f9 d5df72de Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72de2d03108d6b365818ecc3551ac9aa6302 Merge: 5bf59ed2 8ba192f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed250da98a408a94e214a73caa400cba842 Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 98b3955cb808e36303c030aea78eb037d1ec59ce Merge: a056f118 be9dada8 Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit a056f118704eccec86ce32ab86981ce4bc1e1deb Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit 8ba192f94edf5d99598983445d5faa4f8807c49f Merge: 7cc28907 be9dada8 Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit be9dada8b4189c53c08e1674ab273242cf2f80a0 Merge: 62ea8ceb 7cc28907 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit 62ea8ceb223ef2b51ebab2bcd50d5cf339c35cfe Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit 7cc28907edbb4eb58ee1398772a48110ea35dd96 Merge: 4bc7224d ea14cd4b Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit dbfb23873979f789477f4797ee2d6071e0fd921e Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit ea14cd4b361f4c95b3665cbdb95bc51754090eb5 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit 4bc7224dcd27fe8b288bfc3fed4d7a9da9635658 Merge: 2797987f bf14cb85 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit bf14cb8527b2b7ac438a36567a875168bc02d294 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit 6248113f4e11a0ac396d31fa1b032a142fea8cb4 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 2797987f5b88b87bd172714b678a75a1d8051826 Merge: 63d82f1f 66d4bb2d Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit 66d4bb2d9c9afbbdea40196d4ad80e214d0b14b6 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 63d82f1ff11eb430d91a15d6788a1f0b4d596850 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit 44a33799671cb668f55366d5e5a4ddb051a3a1b4 Merge: 5ed00356 0ce46d08 Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 0ce46d088e473d12d63de44f17c67dceab25658c Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 46a88d8b0199ed44d2ff459fb372f2e006960cea Merge: 47b13b9b 5ed00356 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit 47b13b9b320d36ac53b3622557e31239f7c22621 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 5ed00356676cf5d0ff056cf27d1b519b8e303ff7 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit ed8806839db5988ced672bd162b7b046edb4863a Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit fea3806026932a6e2bd6e538bcc413e33abdf245 Merge: d99a24ab 05dc8e85 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit 05dc8e853eab7c6bc782a1e2662d2efe7422f767 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit cbeee20bc4ffb510a2b23d96cdaf4077be7c2a9e Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f00d5498b69dd4f7e54c907ac906abc7c128f000 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 34156335db74cef9e3f0915d7172fd6b22456c15 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit 50575a950736bc8fc1e191310314cbb5fdff5720 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit c9b2252fb8a15dd04252af5e6b4613855afd6ada Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit 465bd4205e8097e9c037b24a3ed08dd6a7694efa Merge: e43bd840 d99a24ab Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit e43bd840b63eb499856e36d9d2ba45c924abcead Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d99a24abd06df10d07e5a4d0ad5030613f92f2e7 Merge: 374590be a66003be Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit a66003befe4175824a1be6ed59f5f5b88c15f792 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit ee91f272985f32eeb9cd6faa41afdd8eb49cac30 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 326b9694fc77398592b8caf3ba0bc2e2bb903813 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit cd050d4a721d01a2ace0cd030cf7f8dc67eb8c4d Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 205721e0aad76dde30255e56149bbed121883356 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit db8e718b502469e8536ee359c5559de87635ffc7 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 427dabb790118f538b64e4e5bf6a7aab9689b3d9 Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 043b483eb55f7be4fea75c9bc0b9b03d251b109b Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit e1f04db8f58dd10591fde335ea13f74cda7c79bd Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 96e8d9867c9549ab7490f4b12cfeb6a06238e0aa Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit 374590be62f988a76cf6704cfe394cd8ae7d4cb6 Merge: 504685e2 cb3b9ce7 Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit 504685e20b17659b913cf46f3012c16bf429e09d Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit cb3b9ce71411da862ff01342a9122a3c656ffbd1 Merge: c9793b38 67b64ea4 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit c9793b3883714f254a700230b7bee781d6110e73 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit 67b64ea44a5a39d96c7a196a8a8345a7486bd912 Merge: 8ee7848a 5fd68451 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit 5fd684515c55ef643726c1b6c720c7cbd2183ba1 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 8ee7848aaa6383aa1f919c3f21199c81db3fff89 Merge: 747e1978 6fefaf7c Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit 747e19782996065cdce7157ee8c5e15beb5b6c59 Merge: 4854a34d 05843072 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit 6fefaf7cea504e35583ee7217449da290295a7a4 Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit 5f4fe360def1c48ea0cb1da6409d192784882308 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit 05843072d608b970bcada1cd0db65a3c80864060 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit 0581ab3cfb362e2024988b46fbbb00324f1233c9 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 4854a34d4d37efb5e201f2691ecdb054590cf20b Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit d224794c49520f4d28a31862cf977198cd6cbc5e Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 453e7936424220f02b99517059ca71babfbe5f5a Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 909edd6769ddcf8a546be4fdd129416687516878 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 7c1ac9706cafc4801fa4da181d2f610b7838c7b8 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 811301c5280ddd74986645086f026ab730c8848c Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 71401bafd1d515f704f86ab4817a758542bc4672 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 24dc435908d921e9f1a5706e3141b12e5d838d18 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 616edf43731415b35f0f5e97748ed2e017a2891d Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit 4c5a99e21a63fb0ee1c7d15546d18066e1d9894b Merge: 45c05b2b b05c3e22 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit b05c3e222fabd308dd7af4e04c1c6a0812962fe6 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit c2008971308ce8168d57c24d00b725832f099244 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 7f9fb6bcc6cd24a7b8011b8753d0ea98cc2451fd Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 45c05b2b2bece76e06849a52a0d034f9c0ac2367 Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit 53f013ed8278776551ca992562253387cc9968d2 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit 22520a95f13334b75eee0cf0387151067a6bf516 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit d1eefb1565014b47287ffa6b350229062f8f602f Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit b2b4dbd2dc13432c79208db35abf7f55c97f1790 Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 662f05ce4c62a46a83f819d3a5925a9bd20059b5 Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 09329322916bfbb604d72ddaf50441a0947f8805 Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit 557a6a3b15e07e506bc05e2cc76ff6a2f8c93964 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 6aeb5504e74ed1980b53700d8e4d4dcf7d1b38fc Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit aea80e6a71f716951353e1e5d68380243396b4d6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 3c12a080d66b9c38f615b961befca7c30f82fa39 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit 82317a635a4978b32e095a06cc295d0ae23661c2 Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit a8bba1cdb51061a0d27bf9a98cca1505b5c58ea5 Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit caa5893b5fd2c1d32c72b97f371ccd9a8d9ec3a0 Merge: c0944486 423b0060 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit c09444860362a136f17641f8b2a1f91c2bbc3715 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 64f07e497f53e5bcbe9e8fb5830cc7a1daaf7ff1 Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 8aaa828108da8514dd9cd23a9d6d83a8b67f2d65 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 7847dc4d8efe60605102414bb071b1da9851228e Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit 3e56b4f92db39a2ce92903b0c43a34f1d14d59ec Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit fa3ff92b07ea5aaa633a2039818c310744f84d07 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 423b00606aa77fd6b324c19e3d480b73ab852db6 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit b7fd7a9f7aa3c0e1e50374047dfffc46a7462b90 Merge: 986139a9 c5a130b6 Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 986139a9a31154679bdea029b09639f84712db27 Merge: b46239ca 8d3526c0 Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit b46239cabab7b545ec99d9eae6c851e531b18374 Merge: bc69a744 373265f2 Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit bc69a744d2cffeb06eba62e843bcc7869e27613a Merge: eef3aeb6 626e8a91 Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 626e8a91a4af2dd5dd774fc130cc2f4d74b2bc37 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit eef3aeb6ab589bb1d5045af5b5c1984a69402d19 Merge: c4e9dd9f 9bca4413 Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 9bca441376325173128e5c50087f068e519c48da Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit 7687495b1ed552eeba088cb9ad5aaf1170e7fff9 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 4eebd3e5d7ab3b8c3116eea57318db72d2ce32bb Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 35fe80b67656114a8824eb59574089663bdc4c9a Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 955bd0635cc6c14a96ad869f1002e6dbefdc5071 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit c4e9dd9f6e40e8586587c4a75987aa109a37f14b Merge: d8a3a99f 319afccb Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit 319afccbe713ddf40a8a6fa28501e64c0ad34725 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 2f3811ca1bbad6a441016b05fde09a571900fca8 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit 28962cbe83631ec5d6481aaea4907a7c96fec848 Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit e457cfb4f2d6869e8367d6d5b03ad25ee4acc363 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit d8a3a99ff6142fe101fa3c188cc7f29593c44345 Merge: 3dcd0158 ed171293 Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit ed171293d1e82075c5c6a847fc91ecbfd45cf89f Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit cd874201c46f32a2903ddffae85f9db73e14adfd Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit 85573674e90c8d505312ba18c5102e0051255078 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit 3dcd01582b719555bcf8eb25d91cc5e42abd2c5f Merge: 95df9fee 743673a1 Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit 743673a1419b6e729e18c96f148745cc739d4c71 Merge: c1a54721 95df9fee Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit c1a5472135c3b84061b64d997ab50dda0412ba4f Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit 373265f24e7a89cbd49ab724a2e388cc0930be78 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit d8530514a5ef9378d2adeaceb228b60ec25a6718 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit 22a4958e993463edff352ac033014f9a485706cc Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) commit 8d3526c0869f0ad7747ff6bb02441140792b461c Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]> * Fix llava onevision loglikelihood video bug * LiveBench July (#146) * claude auto detect json mode * extract information * use claude to generate * fix bugs * fix * generate data * chore: Update dataset name and version for live_bench task * gpt-4-turbo => gpt-4o * chore: Update dataset capture settings in create_dataset.py * everything use gpt-4o * websites * livebench_july * Refactor code to simplify data assignment in example.ipynb * chore: Update dataset name for live_bench task * Add xcomposer2d5 from fanyi, revise something for better usage (#145) * internvl2 * fix some bugs * fix * lint * feat: Add XComposer2D5 model to AVAILABLE_MODELS * xcomposer * Fix llava vid error when using public * Fix xcomposer2d5 * Add generation tokens --------- Co-authored-by: Fanyi Pu <[email protected]> * Dev/ov evals (#147) * fix doc * [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) * Create README.md * Add files via upload * Add MathVerse * Fix typo in qwen_vl that was causing "reference before assignment" * convert contexts to list if necessary and remove unnecessary construction of `questions` * refactor query construction for clarity * Create ScreenSpot on clean branch * Update README to reflect new tasks * Add README file specific to ScreenSpot * slight update * Init webSRC * Draft README for WebSRC * Update main README with new task names * Draft and validate websrc eval on dev split * Add code to enable compilation of submission for WebSRC test split * Bugfix: WebSRC should be token-level F1 NOT character-level * Add qwen vl api * Fix llava conv template for llama3 * Fix llava_hf generation for 1.6 * Parse result for llava_hf 1.6 * Add model_name parameter to Llava constructor * Fix endless warning for llava_hf generation * Fix llava_hf image tokens number issue * Create LICENSE * Update LICENSE * Update LICENSE * Better task list_with_num * Fix idefics2 llava in the wild bugs * Remove redundant code in fuyu * Fix instructblip qformer size mismatch and multi-images problem * Comment out parse result in xcomposer * Comment out Spice in caption task so that don't need to download stanford nlp model * Update gitignore * Add separated pope tasks by category * Fix pope random name in pope full * Set printing info for llava_hf to debug level * Adding Phi3v model. * Adding prompt arguments for Phi3v on MathVista-TestMini * Adding documentation of Phi3v class. * [Fix] import issues of multilingual llava and olympiadbench * fix compatibility issue of older version llava * add upd * add upd * add upd * add upd * add upd * add upd * Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> * merge model_specific_prompt_kwargs and dataset_name into each task yaml * Add MathVerse in README.md * slightly change query_prompt for the reproduction * update utils.py for leaderboard submission * add conbench * update README * Update README.md * init include vcr * modify the form of VCR * switch logic * add crossed_text to vcr_wiki output * include the try-except logic for spacy * update vcr_wiki tasks * update vcr_wiki tasks in README.md * include std and confidence interval * update gpt-3.5-turbo version * update gpt-3.5-turbo version * chore: Remove unnecessary files and code related to live_bench and sft_eval tasks * Bump version to 0.2.0.dev0 * chore: Update lmms-eval to support video evaluations for LLaVA models * Update llava conv_template in lmms_eval/models/llava.py * Update image alignment in README.md * chore: Update lmms-eval to support video evaluations for LLaVA models * chore: Update lmms-eval to support video evaluations for LLaVA models * Update README.md * Update README.md * update aggregation function for vcr_wiki * update README.md * Update README.md * update version * add II-Bench * fix dataset_path * Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image * add tinyllava * LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) * fix #117, allow auto download with tar format videos * fix #117, allow auto download with tar format videos * fix typo * feat: Add support for auto downloading tar format videos * Release llava-wilder * chore: Update dependencies to fix potential risks and improve compatibility * tutorial * docs * update preparation * small fix * small fix * lint * to sh script * update readme * Remove handling non-visual loop in llava * Add llava_hf back to registry * Update README.md * Update README.md * update ablation for videomme datasets * chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. * chore: Remove unused models from lmms_eval package * feat: Allow loading model configurations from other packages * feat: Allow including external tasks from plugins * chore: Add loguru for logging in lmms_eval package * Remove unnecessary lines since use batched visuals now in llava * Add longva * Revise model registry for llava_hf and longva * Delete unnecessary lines * Remove unnecessary lines for video llava * Update pyproject.toml * Update activitynetqa_generation.yaml * Fix vid mme post prompt issue * new task gqa-ru * add mmbench_ru_dev * change prompt to ru * create new task vitatecs * Update README.md * Add wild vision 0617 * Hardcode to keep image for wild vision * Fixing scoring logic * Fixing dataset name * Fixing handling None filtered score * Add detailcaps * Add install capture_metric in env * Add files via upload * feat: Add tie_weights parameter to Llava model initialization * Upgrade lmms-eval to support more models and evaluation tasks * Upgrade lmms-eval to version 0.2.1 * Rename xcomposer 4KHD * chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py * Update utils.py * Update _default_template_vcr_yaml * add process sync via temp file in lmms_eval/evaluator.py * Update utils.py * Update _default_template_vcr_yaml * Add muirbench * Squashed commit of the following: commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83cb3f08b61a2c75cc4326e58792100605b3 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159ff35db330536c0b6dfb4b0a3b24142209 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 45876652a877a8006b828f32f5cc4660629f9190 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651b8c54d36cd94169e3d376f5ed225a195a Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f49b72790b081f981e0e6147131542f7f68 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877bfe5ac590cdbe92dd74d18f83a2091f748 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b8557d5894976448a5c559ed7a66a6cf16 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851d42ba31f5da2a7a65e91499db45174dbc Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9718f287ad433afc90e340b5e17a89c1ed Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a1fe04cfb783dc31d4fbdef3bda0ce19e4 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd621c76a483ff98f8c7eb78f64753da683a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889ba02d819979c7922f6336cf4f1f718f65 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c520c2a3dad26d2b2ca5c4ed4db05a493c73 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651fc9daf97e2f57831ed6e6e7ee7ead7d555 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b6259828212fcba954cdeb8cf90d6e5daabf8 Merge: 22a4958e 050b2c37 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c370017e9b97475dd6cf01fd051b5ca5c86 Merge: 74facb41 ef306512 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef306512e5135f76dffa383f600b8733015836e8 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab67732a4238097725deddf867fb1946ffee40 Merge: dbfb2387 74facb41 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb41a826691dfce4458cf1d8659b34fc5bf5 Merge: 8ba192f9 d5df72de Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72de2d03108d6b365818ecc3551ac9aa6302 Merge: 5bf59ed2 8ba192f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed250da98a408a94e214a73caa400cba842 Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos comm…
kcz358
pushed a commit
that referenced
this pull request
Sep 5, 2024
kcz358
pushed a commit
that referenced
this pull request
Sep 5, 2024
* chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd78 Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980 Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f3 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd18585 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86 Merge: e6844db a5c1869 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c1869 Merge: 2ebec77 557083a Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77 Merge: 211bfed b23d349 Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfed Merge: 7c208b7 79514ee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514ee Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e1 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed38173 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b7 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40de Merge: e19b43a ba7081c Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a Merge: 11fd7e3 a0de897 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3 Merge: 383e7fe a752259 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a752259 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fe Merge: 06fa000 ed2e7f7 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f7 Merge: 03947e1 06fa000 Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de897 Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c Merge: fce85f1 903b042 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042 Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86 Merge: ebe7217 fce85f1 Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f4806 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e1 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef2474 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1 Merge: dbe6329 d94f83c Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe6329 Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83c Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 4587665 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f4 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877b Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd6 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c52 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb5295 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651f Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b625 Merge: 22a4958 050b2c3 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c3 Merge: 74facb4 ef30651 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef30651 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab677 Merge: dbfb238 74facb4 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb4 Merge: 8ba192f d5df72d Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72d Merge: 5bf59ed 8ba192f Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 98b3955 Merge: a056f11 be9dada Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit a056f11 Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit 8ba192f Merge: 7cc2890 be9dada Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit be9dada Merge: 62ea8ce 7cc2890 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit 62ea8ce Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit 7cc2890 Merge: 4bc7224 ea14cd4 Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit dbfb238 Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit ea14cd4 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit 4bc7224 Merge: 2797987 bf14cb8 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit bf14cb8 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit 6248113 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 2797987 Merge: 63d82f1 66d4bb2 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit 66d4bb2 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 63d82f1 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit 44a3379 Merge: 5ed0035 0ce46d0 Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 0ce46d0 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 46a88d8 Merge: 47b13b9 5ed0035 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit 47b13b9 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 5ed0035 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit ed88068 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit fea3806 Merge: d99a24a 05dc8e8 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit 05dc8e8 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit cbeee20 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f00d549 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 3415633 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit 50575a9 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit c9b2252 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit 465bd42 Merge: e43bd84 d99a24a Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit e43bd84 Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d99a24a Merge: 374590b a66003b Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit a66003b Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit ee91f27 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 326b969 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit cd050d4 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 205721e Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit db8e718 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 427dabb Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 043b483 Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit e1f04db Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 96e8d98 Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit 374590b Merge: 504685e cb3b9ce Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit 504685e Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit cb3b9ce Merge: c9793b3 67b64ea Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit c9793b3 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit 67b64ea Merge: 8ee7848 5fd6845 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit 5fd6845 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 8ee7848 Merge: 747e197 6fefaf7 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit 747e197 Merge: 4854a34 0584307 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit 6fefaf7 Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit 5f4fe36 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit 0584307 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit 0581ab3 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 4854a34 Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit d224794 Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 453e793 Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 909edd6 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 7c1ac97 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 811301c Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 71401ba Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 24dc435 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 616edf4 Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit 4c5a99e Merge: 45c05b2 b05c3e2 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit b05c3e2 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit c200897 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 7f9fb6b Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 45c05b2 Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit 53f013e Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit 22520a9 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit d1eefb1 Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit b2b4dbd Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 662f05c Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 0932932 Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit 557a6a3 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 6aeb550 Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit aea80e6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 3c12a08 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit 82317a6 Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit a8bba1c Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit caa5893 Merge: c094448 423b006 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit c094448 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 64f07e4 Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 8aaa828 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 7847dc4 Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit 3e56b4f Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit fa3ff92 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 423b006 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit b7fd7a9 Merge: 986139a c5a130b Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 986139a Merge: b46239c 8d3526c Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit b46239c Merge: bc69a74 373265f Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit bc69a74 Merge: eef3aeb 626e8a9 Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 626e8a9 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit eef3aeb Merge: c4e9dd9 9bca441 Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 9bca441 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit 7687495 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 4eebd3e Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 35fe80b Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 955bd06 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit c4e9dd9 Merge: d8a3a99 319afcc Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit 319afcc Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 2f3811c Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit 28962cb Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit e457cfb Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit d8a3a99 Merge: 3dcd015 ed17129 Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit ed17129 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit cd87420 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit 8557367 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit 3dcd015 Merge: 95df9fe 743673a Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit 743673a Merge: c1a5472 95df9fe Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit c1a5472 Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit 373265f Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit d853051 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit 22a4958 Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c) commit 8d3526c Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 557083a) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]>
kcz358
added a commit
that referenced
this pull request
Sep 5, 2024
* fix doc * [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) * Create README.md * Add files via upload * Add MathVerse * Fix typo in qwen_vl that was causing "reference before assignment" * convert contexts to list if necessary and remove unnecessary construction of `questions` * refactor query construction for clarity * Create ScreenSpot on clean branch * Update README to reflect new tasks * Add README file specific to ScreenSpot * slight update * Init webSRC * Draft README for WebSRC * Update main README with new task names * Draft and validate websrc eval on dev split * Add code to enable compilation of submission for WebSRC test split * Bugfix: WebSRC should be token-level F1 NOT character-level * Add qwen vl api * Fix llava conv template for llama3 * Fix llava_hf generation for 1.6 * Parse result for llava_hf 1.6 * Add model_name parameter to Llava constructor * Fix endless warning for llava_hf generation * Fix llava_hf image tokens number issue * Create LICENSE * Update LICENSE * Update LICENSE * Better task list_with_num * Fix idefics2 llava in the wild bugs * Remove redundant code in fuyu * Fix instructblip qformer size mismatch and multi-images problem * Comment out parse result in xcomposer * Comment out Spice in caption task so that don't need to download stanford nlp model * Update gitignore * Add separated pope tasks by category * Fix pope random name in pope full * Set printing info for llava_hf to debug level * Adding Phi3v model. * Adding prompt arguments for Phi3v on MathVista-TestMini * Adding documentation of Phi3v class. * [Fix] import issues of multilingual llava and olympiadbench * fix compatibility issue of older version llava * add upd * add upd * add upd * add upd * add upd * add upd * Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> * merge model_specific_prompt_kwargs and dataset_name into each task yaml * Add MathVerse in README.md * slightly change query_prompt for the reproduction * update utils.py for leaderboard submission * add conbench * update README * Update README.md * init include vcr * modify the form of VCR * switch logic * add crossed_text to vcr_wiki output * include the try-except logic for spacy * update vcr_wiki tasks * update vcr_wiki tasks in README.md * include std and confidence interval * update gpt-3.5-turbo version * update gpt-3.5-turbo version * chore: Remove unnecessary files and code related to live_bench and sft_eval tasks * Bump version to 0.2.0.dev0 * chore: Update lmms-eval to support video evaluations for LLaVA models * Update llava conv_template in lmms_eval/models/llava.py * Update image alignment in README.md * chore: Update lmms-eval to support video evaluations for LLaVA models * chore: Update lmms-eval to support video evaluations for LLaVA models * Update README.md * Update README.md * update aggregation function for vcr_wiki * update README.md * Update README.md * update version * add II-Bench * fix dataset_path * Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image * add tinyllava * LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) * fix #117, allow auto download with tar format videos * fix #117, allow auto download with tar format videos * fix typo * feat: Add support for auto downloading tar format videos * Release llava-wilder * chore: Update dependencies to fix potential risks and improve compatibility * tutorial * docs * update preparation * small fix * small fix * lint * to sh script * update readme * Remove handling non-visual loop in llava * Add llava_hf back to registry * Update README.md * Update README.md * update ablation for videomme datasets * chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. * chore: Remove unused models from lmms_eval package * feat: Allow loading model configurations from other packages * feat: Allow including external tasks from plugins * chore: Add loguru for logging in lmms_eval package * Remove unnecessary lines since use batched visuals now in llava * Add longva * Revise model registry for llava_hf and longva * Delete unnecessary lines * Remove unnecessary lines for video llava * Update pyproject.toml * Update activitynetqa_generation.yaml * Fix vid mme post prompt issue * new task gqa-ru * add mmbench_ru_dev * change prompt to ru * create new task vitatecs * Update README.md * Add wild vision 0617 * Hardcode to keep image for wild vision * Fixing scoring logic * Fixing dataset name * Fixing handling None filtered score * Add detailcaps * Add install capture_metric in env * Add files via upload * feat: Add tie_weights parameter to Llava model initialization * Upgrade lmms-eval to support more models and evaluation tasks * Upgrade lmms-eval to version 0.2.1 * Rename xcomposer 4KHD * chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py * Update utils.py * Update _default_template_vcr_yaml * add process sync via temp file in lmms_eval/evaluator.py * Update utils.py * Update _default_template_vcr_yaml * Add muirbench * Squashed commit of the following: commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83cb3f08b61a2c75cc4326e58792100605b3 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159ff35db330536c0b6dfb4b0a3b24142209 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 45876652a877a8006b828f32f5cc4660629f9190 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651b8c54d36cd94169e3d376f5ed225a195a Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f49b72790b081f981e0e6147131542f7f68 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877bfe5ac590cdbe92dd74d18f83a2091f748 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b8557d5894976448a5c559ed7a66a6cf16 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851d42ba31f5da2a7a65e91499db45174dbc Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9718f287ad433afc90e340b5e17a89c1ed Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a1fe04cfb783dc31d4fbdef3bda0ce19e4 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd621c76a483ff98f8c7eb78f64753da683a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889ba02d819979c7922f6336cf4f1f718f65 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c520c2a3dad26d2b2ca5c4ed4db05a493c73 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651fc9daf97e2f57831ed6e6e7ee7ead7d555 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b6259828212fcba954cdeb8cf90d6e5daabf8 Merge: 22a4958e 050b2c37 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c370017e9b97475dd6cf01fd051b5ca5c86 Merge: 74facb41 ef306512 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef306512e5135f76dffa383f600b8733015836e8 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab67732a4238097725deddf867fb1946ffee40 Merge: dbfb2387 74facb41 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb41a826691dfce4458cf1d8659b34fc5bf5 Merge: 8ba192f9 d5df72de Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72de2d03108d6b365818ecc3551ac9aa6302 Merge: 5bf59ed2 8ba192f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed250da98a408a94e214a73caa400cba842 Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 98b3955cb808e36303c030aea78eb037d1ec59ce Merge: a056f118 be9dada8 Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit a056f118704eccec86ce32ab86981ce4bc1e1deb Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit 8ba192f94edf5d99598983445d5faa4f8807c49f Merge: 7cc28907 be9dada8 Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit be9dada8b4189c53c08e1674ab273242cf2f80a0 Merge: 62ea8ceb 7cc28907 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit 62ea8ceb223ef2b51ebab2bcd50d5cf339c35cfe Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit 7cc28907edbb4eb58ee1398772a48110ea35dd96 Merge: 4bc7224d ea14cd4b Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit dbfb23873979f789477f4797ee2d6071e0fd921e Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit ea14cd4b361f4c95b3665cbdb95bc51754090eb5 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit 4bc7224dcd27fe8b288bfc3fed4d7a9da9635658 Merge: 2797987f bf14cb85 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit bf14cb8527b2b7ac438a36567a875168bc02d294 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit 6248113f4e11a0ac396d31fa1b032a142fea8cb4 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 2797987f5b88b87bd172714b678a75a1d8051826 Merge: 63d82f1f 66d4bb2d Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit 66d4bb2d9c9afbbdea40196d4ad80e214d0b14b6 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 63d82f1ff11eb430d91a15d6788a1f0b4d596850 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit 44a33799671cb668f55366d5e5a4ddb051a3a1b4 Merge: 5ed00356 0ce46d08 Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 0ce46d088e473d12d63de44f17c67dceab25658c Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 46a88d8b0199ed44d2ff459fb372f2e006960cea Merge: 47b13b9b 5ed00356 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit 47b13b9b320d36ac53b3622557e31239f7c22621 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 5ed00356676cf5d0ff056cf27d1b519b8e303ff7 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit ed8806839db5988ced672bd162b7b046edb4863a Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit fea3806026932a6e2bd6e538bcc413e33abdf245 Merge: d99a24ab 05dc8e85 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit 05dc8e853eab7c6bc782a1e2662d2efe7422f767 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit cbeee20bc4ffb510a2b23d96cdaf4077be7c2a9e Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f00d5498b69dd4f7e54c907ac906abc7c128f000 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 34156335db74cef9e3f0915d7172fd6b22456c15 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit 50575a950736bc8fc1e191310314cbb5fdff5720 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit c9b2252fb8a15dd04252af5e6b4613855afd6ada Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit 465bd4205e8097e9c037b24a3ed08dd6a7694efa Merge: e43bd840 d99a24ab Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit e43bd840b63eb499856e36d9d2ba45c924abcead Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d99a24abd06df10d07e5a4d0ad5030613f92f2e7 Merge: 374590be a66003be Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit a66003befe4175824a1be6ed59f5f5b88c15f792 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit ee91f272985f32eeb9cd6faa41afdd8eb49cac30 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 326b9694fc77398592b8caf3ba0bc2e2bb903813 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit cd050d4a721d01a2ace0cd030cf7f8dc67eb8c4d Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 205721e0aad76dde30255e56149bbed121883356 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit db8e718b502469e8536ee359c5559de87635ffc7 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 427dabb790118f538b64e4e5bf6a7aab9689b3d9 Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 043b483eb55f7be4fea75c9bc0b9b03d251b109b Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit e1f04db8f58dd10591fde335ea13f74cda7c79bd Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 96e8d9867c9549ab7490f4b12cfeb6a06238e0aa Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit 374590be62f988a76cf6704cfe394cd8ae7d4cb6 Merge: 504685e2 cb3b9ce7 Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit 504685e20b17659b913cf46f3012c16bf429e09d Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit cb3b9ce71411da862ff01342a9122a3c656ffbd1 Merge: c9793b38 67b64ea4 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit c9793b3883714f254a700230b7bee781d6110e73 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit 67b64ea44a5a39d96c7a196a8a8345a7486bd912 Merge: 8ee7848a 5fd68451 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit 5fd684515c55ef643726c1b6c720c7cbd2183ba1 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 8ee7848aaa6383aa1f919c3f21199c81db3fff89 Merge: 747e1978 6fefaf7c Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit 747e19782996065cdce7157ee8c5e15beb5b6c59 Merge: 4854a34d 05843072 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit 6fefaf7cea504e35583ee7217449da290295a7a4 Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit 5f4fe360def1c48ea0cb1da6409d192784882308 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit 05843072d608b970bcada1cd0db65a3c80864060 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit 0581ab3cfb362e2024988b46fbbb00324f1233c9 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 4854a34d4d37efb5e201f2691ecdb054590cf20b Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit d224794c49520f4d28a31862cf977198cd6cbc5e Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 453e7936424220f02b99517059ca71babfbe5f5a Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 909edd6769ddcf8a546be4fdd129416687516878 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 7c1ac9706cafc4801fa4da181d2f610b7838c7b8 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 811301c5280ddd74986645086f026ab730c8848c Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 71401bafd1d515f704f86ab4817a758542bc4672 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 24dc435908d921e9f1a5706e3141b12e5d838d18 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 616edf43731415b35f0f5e97748ed2e017a2891d Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit 4c5a99e21a63fb0ee1c7d15546d18066e1d9894b Merge: 45c05b2b b05c3e22 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit b05c3e222fabd308dd7af4e04c1c6a0812962fe6 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit c2008971308ce8168d57c24d00b725832f099244 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 7f9fb6bcc6cd24a7b8011b8753d0ea98cc2451fd Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 45c05b2b2bece76e06849a52a0d034f9c0ac2367 Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit 53f013ed8278776551ca992562253387cc9968d2 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit 22520a95f13334b75eee0cf0387151067a6bf516 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit d1eefb1565014b47287ffa6b350229062f8f602f Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit b2b4dbd2dc13432c79208db35abf7f55c97f1790 Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 662f05ce4c62a46a83f819d3a5925a9bd20059b5 Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 09329322916bfbb604d72ddaf50441a0947f8805 Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit 557a6a3b15e07e506bc05e2cc76ff6a2f8c93964 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 6aeb5504e74ed1980b53700d8e4d4dcf7d1b38fc Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit aea80e6a71f716951353e1e5d68380243396b4d6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 3c12a080d66b9c38f615b961befca7c30f82fa39 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit 82317a635a4978b32e095a06cc295d0ae23661c2 Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit a8bba1cdb51061a0d27bf9a98cca1505b5c58ea5 Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit caa5893b5fd2c1d32c72b97f371ccd9a8d9ec3a0 Merge: c0944486 423b0060 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit c09444860362a136f17641f8b2a1f91c2bbc3715 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 64f07e497f53e5bcbe9e8fb5830cc7a1daaf7ff1 Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 8aaa828108da8514dd9cd23a9d6d83a8b67f2d65 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 7847dc4d8efe60605102414bb071b1da9851228e Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit 3e56b4f92db39a2ce92903b0c43a34f1d14d59ec Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit fa3ff92b07ea5aaa633a2039818c310744f84d07 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 423b00606aa77fd6b324c19e3d480b73ab852db6 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit b7fd7a9f7aa3c0e1e50374047dfffc46a7462b90 Merge: 986139a9 c5a130b6 Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 986139a9a31154679bdea029b09639f84712db27 Merge: b46239ca 8d3526c0 Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit b46239cabab7b545ec99d9eae6c851e531b18374 Merge: bc69a744 373265f2 Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit bc69a744d2cffeb06eba62e843bcc7869e27613a Merge: eef3aeb6 626e8a91 Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 626e8a91a4af2dd5dd774fc130cc2f4d74b2bc37 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit eef3aeb6ab589bb1d5045af5b5c1984a69402d19 Merge: c4e9dd9f 9bca4413 Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 9bca441376325173128e5c50087f068e519c48da Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit 7687495b1ed552eeba088cb9ad5aaf1170e7fff9 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 4eebd3e5d7ab3b8c3116eea57318db72d2ce32bb Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 35fe80b67656114a8824eb59574089663bdc4c9a Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 955bd0635cc6c14a96ad869f1002e6dbefdc5071 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit c4e9dd9f6e40e8586587c4a75987aa109a37f14b Merge: d8a3a99f 319afccb Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit 319afccbe713ddf40a8a6fa28501e64c0ad34725 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 2f3811ca1bbad6a441016b05fde09a571900fca8 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit 28962cbe83631ec5d6481aaea4907a7c96fec848 Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit e457cfb4f2d6869e8367d6d5b03ad25ee4acc363 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit d8a3a99ff6142fe101fa3c188cc7f29593c44345 Merge: 3dcd0158 ed171293 Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit ed171293d1e82075c5c6a847fc91ecbfd45cf89f Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit cd874201c46f32a2903ddffae85f9db73e14adfd Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit 85573674e90c8d505312ba18c5102e0051255078 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit 3dcd01582b719555bcf8eb25d91cc5e42abd2c5f Merge: 95df9fee 743673a1 Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit 743673a1419b6e729e18c96f148745cc739d4c71 Merge: c1a54721 95df9fee Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit c1a5472135c3b84061b64d997ab50dda0412ba4f Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit 373265f24e7a89cbd49ab724a2e388cc0930be78 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit d8530514a5ef9378d2adeaceb228b60ec25a6718 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit 22a4958e993463edff352ac033014f9a485706cc Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) commit 8d3526c0869f0ad7747ff6bb02441140792b461c Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]> commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench commit 4f8db1d37b1f824432927e74d6d82e06bb5aaed1 Author: Pu Fanyi <[email protected]> Date: Fri Jul 12 17:26:50 2024 -0700 Upload live_bench results (#140) * upload results * add a readme * chore: Update upload_results.py script to use shell syntax * Update upload_results.py * Update upload_results.py commit 18f3812c4f9af2e49af6b50e8afe7f607b8a75d6 Author: Pu Fanyi <[email protected]> Date: Wed Jul 10 18:13:43 2024 -0700 Load tasks only one time (#139) * chore: Initialize tasks only once to avoid re-initialization * chore: Initialize tasks only once to avoid re-initialization * chore: Refactor task initialization to avoid re-initialization * chore: Update task initialization to fix include_path issue * chore: Update task initialization to fix include_path issue * chore: Remove unnecessary line in muirbench.yaml * chore: Remove unnecessary line in muirbench.yaml and update gitignore * chore: Update lmms_eval to use correct variable name for world size * Update mmvet * chore: Update lmms_eval to use correct variable name for world size * chore: Remove unused lmms_eval configuration file * refactor: Update lmms_eval to handle both image and video tasks This commit updates the `Llava_OneVision` class in `llava_onevision.py` to handle both image and video tasks. It introduces conditional logic to differentiate between the two types of tasks and process the input accordingly. Additionally, it sets the image aspect ratio based on the number of visual inputs and the configuration settings. Closes #123 * Fix llava onevision loglikelihood video bug (cherry picked from commit f96e3e69fe86dcd9cb33d2bc18cc4ff2003de8be) * refactor: Update mm_spatial_pool_mode to use bilinear interpolation This commit updates the `mm_spatial_pool_mode` parameter in the `Llava_OneVision` class of `llava_onevision.py` to use bilinear interpolation instead of the previous average pooling mode. This change improves the spatial pooling process for the model. Closes #456 * chore: Update pyproject.toml with protobuf dependency version 3.20 * Squashed commit of the following: commit e106f49ceeb295fd4c89a0877073bc01b4b77c5f Author: Fanyi Pu <[email protected]> Date: Thu Jul 25 08:14:03 2024 +0800 livebench_july commit a16295653fdda20d5e8c41c549d731ec422013e3 Author: Fanyi Pu <[email protected]> Date: Mon Jul 22 15:09:58 2024 +0800 websites commit 2cdc06ffe6ba53a4c707c1acf9fc5f2e7886b2b8 Author: Fanyi Pu <[email protected]> Date: Sun Jul 21 15:34:39 2024 +0800 everything use gpt-4o commit e67538d65526c58903d9e62d1914ebd39924ab67 Author: Fanyi Pu <[email protected]> Date: Sun Jul 21 14:29:55 2024 +0800 chore: Update dataset capture settings in create_dataset.py commit 0a3bb33d37cda05bb7bfba4ecf873c2860092a03 Author: Fanyi Pu <[email protected]> Date: Sun Jul 21 01:58:14 2024 +0800 gpt-4-turbo => gpt-4o commit 837f8b0400f04f4367f8f8f954afd64666d62fc6 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 16:48:04 2024 +0800 chore: Update dataset name and version for live_bench task commit fa58e730978b5536005c8bd0291abbeddd761205 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 15:05:13 2024 +0800 generate data commit faa96227a7af7bd6546578b2db68dce2acbc2c0c Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 13:15:18 2024 +0800 fix commit 60ea7ddb4fcd9f08013cd0d5b9dd8090f7e6b83e Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 13:12:31 2024 +0800 fix bugs commit 827d69d0bf967f5d69bfbee9848b4d568ca853b1 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 08:39:41 2024 +0800 use claude to generate commit b7e2619d1a51144cd434861ac151187aed82c8c4 Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 07:36:59 2024 +0800 extract information commit f87d55d47cb0d6653765e9e3f988f4bc186f7d4c Author: Fanyi Pu <[email protected]> Date: Sat Jul 20 07:24:07 2024 +0800 claude auto detect json mode commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 …
MichalCiesiolka
pushed a commit
to MichalCiesiolka/lmms-eval-llmzszl
that referenced
this pull request
Apr 3, 2025
…s. (EvolvingLMMs-Lab#218) * Load tasks only one time (#139) * chore: Initialize tasks only once to avoid re-initialization * chore: Initialize tasks only once to avoid re-initialization * chore: Refactor task initialization to avoid re-initialization * chore: Update task initialization to fix include_path issue * chore: Update task initialization to fix include_path issue * Upload live_bench results (#140) * upload results * add a readme * chore: Update upload_results.py script to use shell syntax * Update upload_results.py * Update upload_results.py * Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * if no response directly return 0 (#142) * merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83cb3f08b61a2c75cc4326e58792100605b3 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159ff35db330536c0b6dfb4b0a3b24142209 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 45876652a877a8006b828f32f5cc4660629f9190 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651b8c54d36cd94169e3d376f5ed225a195a Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f49b72790b081f981e0e6147131542f7f68 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877bfe5ac590cdbe92dd74d18f83a2091f748 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b8557d5894976448a5c559ed7a66a6cf16 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851d42ba31f5da2a7a65e91499db45174dbc Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9718f287ad433afc90e340b5e17a89c1ed Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a1fe04cfb783dc31d4fbdef3bda0ce19e4 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd621c76a483ff98f8c7eb78f64753da683a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889ba02d819979c7922f6336cf4f1f718f65 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c520c2a3dad26d2b2ca5c4ed4db05a493c73 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651fc9daf97e2f57831ed6e6e7ee7ead7d555 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b6259828212fcba954cdeb8cf90d6e5daabf8 Merge: 22a4958e 050b2c37 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c370017e9b97475dd6cf01fd051b5ca5c86 Merge: 74facb41 ef306512 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef306512e5135f76dffa383f600b8733015836e8 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab67732a4238097725deddf867fb1946ffee40 Merge: dbfb2387 74facb41 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb41a826691dfce4458cf1d8659b34fc5bf5 Merge: 8ba192f9 d5df72de Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72de2d03108d6b365818ecc3551ac9aa6302 Merge: 5bf59ed2 8ba192f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed250da98a408a94e214a73caa400cba842 Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 98b3955cb808e36303c030aea78eb037d1ec59ce Merge: a056f118 be9dada8 Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit a056f118704eccec86ce32ab86981ce4bc1e1deb Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit 8ba192f94edf5d99598983445d5faa4f8807c49f Merge: 7cc28907 be9dada8 Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit be9dada8b4189c53c08e1674ab273242cf2f80a0 Merge: 62ea8ceb 7cc28907 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit 62ea8ceb223ef2b51ebab2bcd50d5cf339c35cfe Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit 7cc28907edbb4eb58ee1398772a48110ea35dd96 Merge: 4bc7224d ea14cd4b Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit dbfb23873979f789477f4797ee2d6071e0fd921e Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit ea14cd4b361f4c95b3665cbdb95bc51754090eb5 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit 4bc7224dcd27fe8b288bfc3fed4d7a9da9635658 Merge: 2797987f bf14cb85 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit bf14cb8527b2b7ac438a36567a875168bc02d294 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit 6248113f4e11a0ac396d31fa1b032a142fea8cb4 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 2797987f5b88b87bd172714b678a75a1d8051826 Merge: 63d82f1f 66d4bb2d Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit 66d4bb2d9c9afbbdea40196d4ad80e214d0b14b6 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 63d82f1ff11eb430d91a15d6788a1f0b4d596850 Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit 44a33799671cb668f55366d5e5a4ddb051a3a1b4 Merge: 5ed00356 0ce46d08 Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 0ce46d088e473d12d63de44f17c67dceab25658c Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 46a88d8b0199ed44d2ff459fb372f2e006960cea Merge: 47b13b9b 5ed00356 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit 47b13b9b320d36ac53b3622557e31239f7c22621 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 5ed00356676cf5d0ff056cf27d1b519b8e303ff7 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit ed8806839db5988ced672bd162b7b046edb4863a Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit fea3806026932a6e2bd6e538bcc413e33abdf245 Merge: d99a24ab 05dc8e85 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit 05dc8e853eab7c6bc782a1e2662d2efe7422f767 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit cbeee20bc4ffb510a2b23d96cdaf4077be7c2a9e Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f00d5498b69dd4f7e54c907ac906abc7c128f000 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 34156335db74cef9e3f0915d7172fd6b22456c15 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit 50575a950736bc8fc1e191310314cbb5fdff5720 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit c9b2252fb8a15dd04252af5e6b4613855afd6ada Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit 465bd4205e8097e9c037b24a3ed08dd6a7694efa Merge: e43bd840 d99a24ab Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit e43bd840b63eb499856e36d9d2ba45c924abcead Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d99a24abd06df10d07e5a4d0ad5030613f92f2e7 Merge: 374590be a66003be Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit a66003befe4175824a1be6ed59f5f5b88c15f792 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit ee91f272985f32eeb9cd6faa41afdd8eb49cac30 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 326b9694fc77398592b8caf3ba0bc2e2bb903813 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit cd050d4a721d01a2ace0cd030cf7f8dc67eb8c4d Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 205721e0aad76dde30255e56149bbed121883356 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit db8e718b502469e8536ee359c5559de87635ffc7 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 427dabb790118f538b64e4e5bf6a7aab9689b3d9 Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 043b483eb55f7be4fea75c9bc0b9b03d251b109b Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit e1f04db8f58dd10591fde335ea13f74cda7c79bd Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 96e8d9867c9549ab7490f4b12cfeb6a06238e0aa Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit 374590be62f988a76cf6704cfe394cd8ae7d4cb6 Merge: 504685e2 cb3b9ce7 Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit 504685e20b17659b913cf46f3012c16bf429e09d Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit cb3b9ce71411da862ff01342a9122a3c656ffbd1 Merge: c9793b38 67b64ea4 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit c9793b3883714f254a700230b7bee781d6110e73 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit 67b64ea44a5a39d96c7a196a8a8345a7486bd912 Merge: 8ee7848a 5fd68451 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit 5fd684515c55ef643726c1b6c720c7cbd2183ba1 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 8ee7848aaa6383aa1f919c3f21199c81db3fff89 Merge: 747e1978 6fefaf7c Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit 747e19782996065cdce7157ee8c5e15beb5b6c59 Merge: 4854a34d 05843072 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit 6fefaf7cea504e35583ee7217449da290295a7a4 Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit 5f4fe360def1c48ea0cb1da6409d192784882308 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit 05843072d608b970bcada1cd0db65a3c80864060 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit 0581ab3cfb362e2024988b46fbbb00324f1233c9 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 4854a34d4d37efb5e201f2691ecdb054590cf20b Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit d224794c49520f4d28a31862cf977198cd6cbc5e Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 453e7936424220f02b99517059ca71babfbe5f5a Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 909edd6769ddcf8a546be4fdd129416687516878 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 7c1ac9706cafc4801fa4da181d2f610b7838c7b8 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 811301c5280ddd74986645086f026ab730c8848c Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 71401bafd1d515f704f86ab4817a758542bc4672 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 24dc435908d921e9f1a5706e3141b12e5d838d18 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 616edf43731415b35f0f5e97748ed2e017a2891d Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit 4c5a99e21a63fb0ee1c7d15546d18066e1d9894b Merge: 45c05b2b b05c3e22 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit b05c3e222fabd308dd7af4e04c1c6a0812962fe6 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit c2008971308ce8168d57c24d00b725832f099244 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 7f9fb6bcc6cd24a7b8011b8753d0ea98cc2451fd Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 45c05b2b2bece76e06849a52a0d034f9c0ac2367 Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit 53f013ed8278776551ca992562253387cc9968d2 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit 22520a95f13334b75eee0cf0387151067a6bf516 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit d1eefb1565014b47287ffa6b350229062f8f602f Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit b2b4dbd2dc13432c79208db35abf7f55c97f1790 Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 662f05ce4c62a46a83f819d3a5925a9bd20059b5 Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 09329322916bfbb604d72ddaf50441a0947f8805 Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit 557a6a3b15e07e506bc05e2cc76ff6a2f8c93964 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 6aeb5504e74ed1980b53700d8e4d4dcf7d1b38fc Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit aea80e6a71f716951353e1e5d68380243396b4d6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 3c12a080d66b9c38f615b961befca7c30f82fa39 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit 82317a635a4978b32e095a06cc295d0ae23661c2 Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit a8bba1cdb51061a0d27bf9a98cca1505b5c58ea5 Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit caa5893b5fd2c1d32c72b97f371ccd9a8d9ec3a0 Merge: c0944486 423b0060 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit c09444860362a136f17641f8b2a1f91c2bbc3715 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 64f07e497f53e5bcbe9e8fb5830cc7a1daaf7ff1 Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 8aaa828108da8514dd9cd23a9d6d83a8b67f2d65 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 7847dc4d8efe60605102414bb071b1da9851228e Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit 3e56b4f92db39a2ce92903b0c43a34f1d14d59ec Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit fa3ff92b07ea5aaa633a2039818c310744f84d07 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 423b00606aa77fd6b324c19e3d480b73ab852db6 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit b7fd7a9f7aa3c0e1e50374047dfffc46a7462b90 Merge: 986139a9 c5a130b6 Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 986139a9a31154679bdea029b09639f84712db27 Merge: b46239ca 8d3526c0 Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit b46239cabab7b545ec99d9eae6c851e531b18374 Merge: bc69a744 373265f2 Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit bc69a744d2cffeb06eba62e843bcc7869e27613a Merge: eef3aeb6 626e8a91 Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 626e8a91a4af2dd5dd774fc130cc2f4d74b2bc37 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit eef3aeb6ab589bb1d5045af5b5c1984a69402d19 Merge: c4e9dd9f 9bca4413 Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 9bca441376325173128e5c50087f068e519c48da Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit 7687495b1ed552eeba088cb9ad5aaf1170e7fff9 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 4eebd3e5d7ab3b8c3116eea57318db72d2ce32bb Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 35fe80b67656114a8824eb59574089663bdc4c9a Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 955bd0635cc6c14a96ad869f1002e6dbefdc5071 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit c4e9dd9f6e40e8586587c4a75987aa109a37f14b Merge: d8a3a99f 319afccb Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit 319afccbe713ddf40a8a6fa28501e64c0ad34725 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 2f3811ca1bbad6a441016b05fde09a571900fca8 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit 28962cbe83631ec5d6481aaea4907a7c96fec848 Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit e457cfb4f2d6869e8367d6d5b03ad25ee4acc363 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit d8a3a99ff6142fe101fa3c188cc7f29593c44345 Merge: 3dcd0158 ed171293 Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit ed171293d1e82075c5c6a847fc91ecbfd45cf89f Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit cd874201c46f32a2903ddffae85f9db73e14adfd Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit 85573674e90c8d505312ba18c5102e0051255078 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit 3dcd01582b719555bcf8eb25d91cc5e42abd2c5f Merge: 95df9fee 743673a1 Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit 743673a1419b6e729e18c96f148745cc739d4c71 Merge: c1a54721 95df9fee Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit c1a5472135c3b84061b64d997ab50dda0412ba4f Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit 373265f24e7a89cbd49ab724a2e388cc0930be78 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit d8530514a5ef9378d2adeaceb228b60ec25a6718 Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit 22a4958e993463edff352ac033014f9a485706cc Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) commit 8d3526c0869f0ad7747ff6bb02441140792b461c Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]> * Fix llava onevision loglikelihood video bug * LiveBench July (#146) * claude auto detect json mode * extract information * use claude to generate * fix bugs * fix * generate data * chore: Update dataset name and version for live_bench task * gpt-4-turbo => gpt-4o * chore: Update dataset capture settings in create_dataset.py * everything use gpt-4o * websites * livebench_july * Refactor code to simplify data assignment in example.ipynb * chore: Update dataset name for live_bench task * Add xcomposer2d5 from fanyi, revise something for better usage (#145) * internvl2 * fix some bugs * fix * lint * feat: Add XComposer2D5 model to AVAILABLE_MODELS * xcomposer * Fix llava vid error when using public * Fix xcomposer2d5 * Add generation tokens --------- Co-authored-by: Fanyi Pu <[email protected]> * Dev/ov evals (#147) * fix doc * [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) * Create README.md * Add files via upload * Add MathVerse * Fix typo in qwen_vl that was causing "reference before assignment" * convert contexts to list if necessary and remove unnecessary construction of `questions` * refactor query construction for clarity * Create ScreenSpot on clean branch * Update README to reflect new tasks * Add README file specific to ScreenSpot * slight update * Init webSRC * Draft README for WebSRC * Update main README with new task names * Draft and validate websrc eval on dev split * Add code to enable compilation of submission for WebSRC test split * Bugfix: WebSRC should be token-level F1 NOT character-level * Add qwen vl api * Fix llava conv template for llama3 * Fix llava_hf generation for 1.6 * Parse result for llava_hf 1.6 * Add model_name parameter to Llava constructor * Fix endless warning for llava_hf generation * Fix llava_hf image tokens number issue * Create LICENSE * Update LICENSE * Update LICENSE * Better task list_with_num * Fix idefics2 llava in the wild bugs * Remove redundant code in fuyu * Fix instructblip qformer size mismatch and multi-images problem * Comment out parse result in xcomposer * Comment out Spice in caption task so that don't need to download stanford nlp model * Update gitignore * Add separated pope tasks by category * Fix pope random name in pope full * Set printing info for llava_hf to debug level * Adding Phi3v model. * Adding prompt arguments for Phi3v on MathVista-TestMini * Adding documentation of Phi3v class. * [Fix] import issues of multilingual llava and olympiadbench * fix compatibility issue of older version llava * add upd * add upd * add upd * add upd * add upd * add upd * Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> * merge model_specific_prompt_kwargs and dataset_name into each task yaml * Add MathVerse in README.md * slightly change query_prompt for the reproduction * update utils.py for leaderboard submission * add conbench * update README * Update README.md * init include vcr * modify the form of VCR * switch logic * add crossed_text to vcr_wiki output * include the try-except logic for spacy * update vcr_wiki tasks * update vcr_wiki tasks in README.md * include std and confidence interval * update gpt-3.5-turbo version * update gpt-3.5-turbo version * chore: Remove unnecessary files and code related to live_bench and sft_eval tasks * Bump version to 0.2.0.dev0 * chore: Update lmms-eval to support video evaluations for LLaVA models * Update llava conv_template in lmms_eval/models/llava.py * Update image alignment in README.md * chore: Update lmms-eval to support video evaluations for LLaVA models * chore: Update lmms-eval to support video evaluations for LLaVA models * Update README.md * Update README.md * update aggregation function for vcr_wiki * update README.md * Update README.md * update version * add II-Bench * fix dataset_path * Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image * add tinyllava * LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) * fix #117, allow auto download with tar format videos * fix #117, allow auto download with tar format videos * fix typo * feat: Add support for auto downloading tar format videos * Release llava-wilder * chore: Update dependencies to fix potential risks and improve compatibility * tutorial * docs * update preparation * small fix * small fix * lint * to sh script * update readme * Remove handling non-visual loop in llava * Add llava_hf back to registry * Update README.md * Update README.md * update ablation for videomme datasets * chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. * chore: Remove unused models from lmms_eval package * feat: Allow loading model configurations from other packages * feat: Allow including external tasks from plugins * chore: Add loguru for logging in lmms_eval package * Remove unnecessary lines since use batched visuals now in llava * Add longva * Revise model registry for llava_hf and longva * Delete unnecessary lines * Remove unnecessary lines for video llava * Update pyproject.toml * Update activitynetqa_generation.yaml * Fix vid mme post prompt issue * new task gqa-ru * add mmbench_ru_dev * change prompt to ru * create new task vitatecs * Update README.md * Add wild vision 0617 * Hardcode to keep image for wild vision * Fixing scoring logic * Fixing dataset name * Fixing handling None filtered score * Add detailcaps * Add install capture_metric in env * Add files via upload * feat: Add tie_weights parameter to Llava model initialization * Upgrade lmms-eval to support more models and evaluation tasks * Upgrade lmms-eval to version 0.2.1 * Rename xcomposer 4KHD * chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py * Update utils.py * Update _default_template_vcr_yaml * add process sync via temp file in lmms_eval/evaluator.py * Update utils.py * Update _default_template_vcr_yaml * Add muirbench * Squashed commit of the following: commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit e31cd7883d4555c7530795c7f102b8d78cbd372f Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 1d8c980d1089f9d7702c3b92d5c85039f2809c6d Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 6da76f36ecf5f9aa73057e767a4fcb60c99ff896 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit cd1858523fcd8630082cbefba8710e0de3ee8805 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit 672d7e5bb49dcb34e1b2fdeb09f3f4588dc583a6 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 2037a86261b55fa42b8ba3a04eab192b3e69d6ea Merge: e6844db1 a5c18692 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a5c186925de989b616f58a35ece36065a32b4594 Merge: 2ebec77f 557083a1 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 557083a156c3dd67ac79e22b4202e9b69b6b00f4 Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 2ebec77f5606d79e9a7b995970e32792050606a1 Merge: 211bfede b23d349e Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit b23d349e46d60dc149ffaa54d6e019f4996ed92d Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit c6e211d5f9dbb7572d3a141b6504cb1ca2007c33 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 211bfedebad243ef82a8b0be36c3b5a9b9cb2f72 Merge: 7c208b76 79514eee Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit 79514eeebcfd6f655be2a10c776037d12a7b7214 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit 725fac2781446958b905e1e6c6eb3c0a8e582e49 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 8d963e132ac03fc0d835d480cfcfcabe72af143c Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit e2990d0a69e876721256fdf946c68ba7ae0cbdc1 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit ed381736730d8fb785b4ee919fdb751734ecef25 Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit 7c208b76640c986cfe94233dce735c3ca4ad4319 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit 39d40dea47bc59ff04e8b0cbc445345098debc9a Merge: e19b43a3 ba7081c0 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit e19b43a3a1e7212e623061b164b0419cc0dda689 Merge: 11fd7e3f a0de8970 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 11fd7e3fc05908aeb01e4a6161a7b55cd38b3122 Merge: 383e7fea a7522592 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit a75225926e5954f85466d257f99acf0163fde596 Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit ba7081c0abac840002d320e30733e891298dfa11 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 27ea9c0055a8abf3a8198829b8617018479918e2 Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 383e7fead3138aedf62e9c0ec48303835ef26e2a Merge: 06fa000f ed2e7f79 Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ed2e7f792151d21bce8f1c498270b9391e1d5c85 Merge: 03947e14 06fa000f Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit a0de89708d5e6f259bb17f0eaace3c5b901b275c Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit 06fa000f60d3e4d160fac8ceb9959ae92a98f752 Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit b388d79e0df6f60068196cb7047453ebd22d6ef1 Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit 8f9d620fcd9d0a0742ee6bcf51ea63bd6b088a36 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit 6341b7c15ce9fb28eb06b067ddb299d6cf2e16c3 Merge: fce85f1b 903b042b Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit 903b042be016016d4ebeecb07701f3076a2d323c Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit d78ec86407b729a964906a8c2e50704b4bc74d06 Merge: ebe7217a fce85f1b Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit ebe7217a486c1e754e42c2cbdb834e09fbbcc9b0 Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 120c474b056f9177c74e1fd9691d59e2f234b785 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit 7d6201f921088afd3f52a35076e3c6fcc9aa518c Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 12f480699c71a12a24d4349d9b0681933201a3a6 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit 12cea76f1f0f14b1fd1007c9d39a9b0557368637 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 03947e14a46fd25b412931f7c9c25f4a2971d0b4 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b80a91f73e15ddd0b0ce1322d7d121fa14030eed Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit 8ef24740dd48a11c97eb627f2fff4aca107fef0d Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit af38885fc2e066f5ea44388f33e07176f836fe28 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit fce85f1b03ff7043b29dee787c5d17a08dd2687a Merge: dbe63293 d94f83cb Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit dbe63293245a5141fdfd80bda7657c304f6bd32f Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit d94f83cb3f08b61a2c75cc4326e58792100605b3 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit cab8159ff35db330536c0b6dfb4b0a3b24142209 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit 45876652a877a8006b828f32f5cc4660629f9190 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 3463651b8c54d36cd94169e3d376f5ed225a195a Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit cb0d3f49b72790b081f981e0e6147131542f7f68 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 813877bfe5ac590cdbe92dd74d18f83a2091f748 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit a14684b8557d5894976448a5c559ed7a66a6cf16 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit d0f8851d42ba31f5da2a7a65e91499db45174dbc Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 63748e9718f287ad433afc90e340b5e17a89c1ed Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 7f1159a1fe04cfb783dc31d4fbdef3bda0ce19e4 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 19f9bd621c76a483ff98f8c7eb78f64753da683a Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit ce6f889ba02d819979c7922f6336cf4f1f718f65 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit f513c520c2a3dad26d2b2ca5c4ed4db05a493c73 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit efb529552c5e4ba039a4cba8e9aa5cb7ba65bf90 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 742651fc9daf97e2f57831ed6e6e7ee7ead7d555 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 511b6259828212fcba954cdeb8cf90d6e5daabf8 Merge: 22a4958e 050b2c37 Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit 050b2c370017e9b97475dd6cf01fd051b5ca5c86 Merge: 74facb41 ef306512 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit ef306512e5135f76dffa383f600b8733015836e8 Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 9bab67732a4238097725deddf867fb1946ffee40 Merge: dbfb2387 74facb41 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 74facb41a826691dfce4458cf1d8659b34fc5bf5 Merge: 8ba192f9 d5df72de Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit d5df72de2d03108d6b365818ecc3551ac9aa6302 Merge: 5bf59ed2 8ba192f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 5bf59ed250da98a408a94e214a73caa400cba842 Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos comm…
dadwadw233
pushed a commit
to dadwadw233/lmms-eval
that referenced
this pull request
Apr 28, 2025
…s. (EvolvingLMMs-Lab#218) * Load tasks only one time (#139) * chore: Initialize tasks only once to avoid re-initialization * chore: Initialize tasks only once to avoid re-initialization * chore: Refactor task initialization to avoid re-initialization * chore: Update task initialization to fix include_path issue * chore: Update task initialization to fix include_path issue * Upload live_bench results (#140) * upload results * add a readme * chore: Update upload_results.py script to use shell syntax * Update upload_results.py * Update upload_results.py * Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * if no response directly return 0 (#142) * merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit ec9904af65ac296756db05e958a31c36eb3d9058 Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 0ab768c8bad11da505c9acc884a54136cf5980ce Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 7c1478f1fdc8d8051ddadb11435f2e2877e874d0 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit ec9c5a1c480ad702bdb1b17bed766594463d0da5 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit ce6f82e988e530bb073e789e98de571f8981f4cc Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 87e0462cd4dbb4ab4b05107f7e87523419003309 Merge: 4c5d1a6f a2b07c09 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a2b07c09ff9550ccbdbdf941385003696f9379cc Merge: 44e79980 59bb9af3 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 59bb9af306b554bae841f0934502b470b163390d Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 44e79980f80da75b70391276a65de4aec5a1b708 Merge: 8109ff79 62a5de53 Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit 62a5de53deeafe76680a595b154bf25c0e22a52c Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit e6941ed29b917702157850c88163186fc1765e41 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 8109ff7949c7686994ed3299ee1bdb11421ec89c Merge: c5258c96 f07fd371 Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit f07fd3717c7592f3387b520456053046545466a5 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit ddd7ee1bda57a2c8f83c3a34f3c068c7bb2fbbf8 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 375b8ccf341aae7048e1ab97ee8a7d30f9dd3137 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit 4254530f472b2aad5b5a26d6c6622c9856ea8ade Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit 2d61dd7599dad8fa0c6e5006841954e53f97581c Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit c5258c96b99c9c6db0e40b0d88430224bb497404 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit b41d080daa4a262fd5afaca9b19593805a6ee473 Merge: fb59c6cb 92d3d4ce Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit fb59c6cbe42377c72161f9674ed5884064763911 Merge: 92ceee0c 7010fa23 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 92ceee0c44270d7d03d33853a83855e76fd7d728 Merge: 51c1403e c7696149 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit c76961496f509f4bb5c54868fc8b5c7097c1089a Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit 92d3d4ced0898265ef334535184522376cad1a36 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 1054d8adbd0825e34cfb9006c765add2db45b9ef Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 51c1403ed8d14fdfb1cb7bd947425e0d5fcafc6e Merge: f7f15f45 ba3c1f4a Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ba3c1f4af009f554b8751bbea05a178eaf1a4d6a Merge: 9f2145de f7f15f45 Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit 7010fa2382cbae19abaedd9f640a0c5d23eea673 Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit f7f15f451bd740e99fd8e7bfbe723992705cf71a Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit 5bb1722e303746d65b14d32bee43be2a88254d9a Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit d44a1d1596d71fcffdc329430c3fbc9c19263686 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit cc748e9d77f46d567143e57c950669bf8c0dbc5a Merge: 7d4d8af3 a7342ac7 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit a7342ac7f79f320fd0ac897082ee9eff51162d97 Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit 0856d476b6bd9c9ac0dae4ad0b5fc845ae1dbdef Merge: 91a3d9c6 7d4d8af3 Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit 91a3d9c6727e227db0b15881bc3c371888c4b4ae Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 1c051b10ef00faa47451068043f3e387704fe58a Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit d57f2e239ce730ba0e76502bf521f57e5868d270 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 99ec17ae9e3fdf7d2b8b74024935da28bd47c456 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit ae72f2180bd5275769c6aded2d212673daac7291 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 9f2145dec51bcba853eab4ba037cd19d1193c8ca Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b8540ec84d15bb13653a6067bc210921906d94e0 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit d1d4829069b5b8799ccafe7616ddfb08ac2821da Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit da62829f4a16aaf652d08985080f0df509b9dce3 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit 7d4d8af3ad6962f26260ce515c790b9eaf7f5103 Merge: 7f6cfa5d 86139ce8 Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit 7f6cfa5d92942d857ed259db655a24e5ea3132d9 Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit 86139ce84ee8323c0c276fbdd8cf8e5690653b40 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit a2cb9f7138c1f8f004d3b332068cef2d45726dbd Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit f130e6ae168173ca8c649824d1fde9ac93b5a8f2 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 71c37d38675319db3447eb868e33883448159450 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit 116eb19ad4550177d8b4de66142bf60c6ced1711 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 3466d567d16721db27c27f5bec52eb3bef6f7965 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit 93554726616aae10f5a78c9f748f97c5d14cedc0 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit a381d5df198b22151e5ef2a1df6c8184757a1025 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 4772eee0fb09d77e66f4ca52ef301bedbcb9ba9e Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 1751954c9d0da906ab1007d110e42c40d374520d Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 908a1618d29dba0195eb6f280e8610208804534c Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit d49a65e961aa143de72aa1dae3105f2b9c24d43c Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit 989ef83de930f40c666a6c39a60140f4586ca4d5 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit 84bcd6f6f7d7d160fa8d882ca67ebe9056353ca4 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 79402ea60762031bf889b618f0976ad78206ab73 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 03b2f7cbd0e4d810fb75c8340c21213b60be9295 Merge: c51eac2b b4e8ca6d Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit b4e8ca6de57237d4e53771c82f2ea3c379adb5b3 Merge: 5e2459b6 7d3536e9 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit 7d3536e90e05d6f732c349a43ab04f3d59c0bafb Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 2217602cf93c297c35ffcc25117f382a60648b0b Merge: eb88c556 5e2459b6 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 5e2459b64c746020b9aa8e558e8c7aa65c9e3cf7 Merge: d49a032f ae92f69d Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit ae92f69d44c7d849a5c990a4a607930eb4c33770 Merge: 8f6e8460 d49a032f Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 8f6e8460cf65c9a82ae195d40edf44af1f12e6cc Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos commit 20aec53d3d1c2ae4a9d830d6c7f29fbba547bd6f Merge: 7803bce8 edeb34ee Author: teowu <[email protected]> Date: Sun Jun 16 07:25:07 2024 +0000 Merge branch 'main' of https://github.com/teowu/lmms-eval into main commit 7803bce863ff1ecfac517aeb88683e52c8731ff8 Author: teowu <[email protected]> Date: Sun Jun 16 07:23:54 2024 +0000 fix #117, allow auto download with tar format videos commit d49a032fa769eaaf6e0669804f676d626c1a69c8 Merge: ac3a66f9 edeb34ee Author: Li Bo <[email protected]> Date: Sat Jun 15 17:30:59 2024 +0800 Merge pull request #117 from teowu/main LongVideoBench for LMMs-Eval commit edeb34ee28658fe121d5be9449b8f40538c9f98a Merge: e22c66db ac3a66f9 Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sat Jun 15 16:39:20 2024 +0800 Merge pull request #1 from EvolvingLMMs-Lab/main Merge pull request #113 from teowu/main commit e22c66db4a372d8a01fd793f035cdd89e527a693 Author: teowu <[email protected]> Date: Sat Jun 15 08:30:11 2024 +0000 LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) commit ac3a66f97042d728dbe8992ee66f104eb93fa68f Merge: e23e9886 043d8d04 Author: Li Bo <[email protected]> Date: Sat Jun 15 14:10:22 2024 +0800 Merge pull request #113 from teowu/main Q-Bench, Q-Bench2, A-Bench commit eb88c5567b9d23e4a592950ed6dda5ed3273ed9f Author: Jingyang <[email protected]> Date: Fri Jun 14 16:20:42 2024 -0400 add tinyllava commit 043d8d049066b36d7abb1d666640b845bd037476 Author: teowu <[email protected]> Date: Fri Jun 14 15:01:52 2024 +0000 Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image commit e23e98861d8d3b7500921adfa206dd114ffb2837 Merge: 43fe1e14 e68f3935 Author: Li Bo <[email protected]> Date: Fri Jun 14 02:14:43 2024 +0800 Merge pull request #111 from XinrunDu/main add II-Bench commit e68f393585fe5f6360b336819a5bf2a0340ad8c0 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:37:02 2024 +0000 fix dataset_path commit c76d74a72227a26fe097eea2e651be92ce6d6622 Author: XinrunDu <[email protected]> Date: Thu Jun 13 09:32:06 2024 +0000 add II-Bench commit 43fe1e14d3cc7cbf98aaf5098c70ee01f0c735eb Merge: 7968b5e5 a0425ced Author: Li Bo <[email protected]> Date: Thu Jun 13 11:14:47 2024 +0800 Merge pull request #109 from EvolvingLMMs-Lab/pufanyi/update_version [Small Update] Update the version of LMMs-Eval commit a0425cede8c8b2432799b44280ae3df6e2801d45 Author: Fanyi Pu <[email protected]> Date: Thu Jun 13 11:13:00 2024 +0800 update version commit 7968b5e562bf7afadce059cfb7a4fe0014df372c Author: Li Bo <[email protected]> Date: Thu Jun 13 11:04:32 2024 +0800 Update README.md commit dea0ddde16c57233645b4b515dac03b525d7b953 Merge: 8dac15dc 631ab72d Author: Li Bo <[email protected]> Date: Thu Jun 13 04:00:12 2024 +0800 Merge pull request #105 from tianyu-z/main Include VCR commit 631ab72d89090400511c3ad634da243cf64e5dac Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:56:34 2024 -0400 update README.md commit 6a41461ab3a9db814db0ebf092083505902e102a Merge: d099e45b 8dac15dc Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:50:26 2024 -0400 merged readme.md commit d099e45b099cc5aa08c89833ef0d87331076b255 Author: Suyuchen <[email protected]> Date: Wed Jun 12 15:30:52 2024 -0400 update aggregation function for vcr_wiki commit 8dac15dc8d153baa780963a218e4d335e2205024 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:21:42 2024 +0800 Update README.md commit cf4219e03ed3f9d74f5ac74695608ad9189f29e2 Author: Li Bo <[email protected]> Date: Thu Jun 13 03:13:59 2024 +0800 Update README.md commit 34850b6c1139b7cb843bb7806af28d260125d782 Merge: d84b4f9f f1a0241a Author: Li Bo <[email protected]> Date: Thu Jun 13 03:11:49 2024 +0800 Merge pull request #108 from EvolvingLMMs-Lab/internal_main_dev [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval commit f1a0241af3f3340863c4c63a236c2bd15edca990 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:56:04 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit f6841c94bc1511c3f0a31229b6939b632e9ddbf7 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:50:30 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit 93baf580caa031471647eb2bdea9e54b062798a0 Author: Bo Li <[email protected]> Date: Wed Jun 12 15:46:33 2024 +0000 Update image alignment in README.md commit 629c240a5654c9be0f69da5972cc72be7f1f7dab Author: Bo Li <[email protected]> Date: Wed Jun 12 15:43:16 2024 +0000 Update llava conv_template in lmms_eval/models/llava.py commit ccf4fbff4a28a696dc5cda5faf6a8d87c8602dea Author: Bo Li <[email protected]> Date: Wed Jun 12 15:39:03 2024 +0000 chore: Update lmms-eval to support video evaluations for LLaVA models commit 380a8b5bbb36bad23ee6392743bfee7a9d0a446a Author: Bo Li <[email protected]> Date: Wed Jun 12 15:33:48 2024 +0000 Bump version to 0.2.0.dev0 commit e88ce1f81572e7ebeb932d58c2757f8e37cebdaa Merge: a8ce3b5b d84b4f9f Author: Bo Li <[email protected]> Date: Wed Jun 12 15:04:25 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval into internal_main_dev commit a8ce3b5b69f7c5daed2b99b1e5850f36c969f610 Author: Bo Li <[email protected]> Date: Wed Jun 12 14:54:06 2024 +0000 chore: Remove unnecessary files and code related to live_bench and sft_eval tasks commit d84b4f9f18a6f14c2781db1acfbf465c0c73c23a Merge: a11d13f7 45769abf Author: Li Bo <[email protected]> Date: Wed Jun 12 19:45:57 2024 +0800 Merge pull request #107 from AtsuMiyai/new_task/upd_update update gpt-3.5-turbo version commit 45769abfdf68aaf7b9f5ae92561612630777489a Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 17:05:17 2024 +0900 update gpt-3.5-turbo version commit 73c8f1bace9e45bfd06afb4714c7f43392e32f43 Author: AtsuMiyai <[email protected]> Date: Wed Jun 12 16:50:53 2024 +0900 update gpt-3.5-turbo version commit 93e02a0103f8d702015234560218cd48e9746581 Author: tianyu-z <[email protected]> Date: Mon Jun 10 20:07:40 2024 -0400 include std and confidence interval commit 0864156cb0a1094b90ad0c5aae30d869c0e77299 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:49:47 2024 -0400 update vcr_wiki tasks in README.md commit 8deb4d31f7419549f57c4786cffc8d972fb08538 Author: Suyuchen <[email protected]> Date: Mon Jun 10 18:43:15 2024 -0400 update vcr_wiki tasks commit dd4ffe57ef11b7835f59859d24d6869232c1f6e0 Author: tianyu-z <[email protected]> Date: Mon Jun 10 16:13:58 2024 -0400 include the try-except logic for spacy commit 760bfa5e6c3746640c03eaf6dfbd362c30cb9387 Author: Suyuchen <[email protected]> Date: Mon Jun 10 15:51:05 2024 -0400 add crossed_text to vcr_wiki output commit 039307892132f4c28522616af355009da6cacb66 Author: tianyu-z <[email protected]> Date: Mon Jun 10 15:47:00 2024 -0400 switch logic commit bcd3f3ab70f5efa24b42b0ba66f6c89b0d8493b5 Author: tianyu-z <[email protected]> Date: Mon Jun 10 02:38:21 2024 -0400 modify the form of VCR commit 64533fad7f858198f212ea5a62287029386472ac Author: tianyu-z <[email protected]> Date: Mon Jun 10 00:10:30 2024 -0400 init include vcr commit a11d13f712ef6c7cfbfd3f6fea7112869e52c56a Merge: c94ef9d5 83c958ef Author: Kaichen Zhang - NTU <[email protected]> Date: Fri Jun 7 20:25:48 2024 +0800 Merge pull request #101 from Gumpest/main Update conbench in README commit c94ef9d581135986a7e8d6b21077a0157203a53c Author: Li Bo <[email protected]> Date: Thu Jun 6 15:42:15 2024 +0800 Update README.md commit 83c958efd27b851e7d1d088baae311cabdbd0f74 Merge: 2bb4447e bd6eca08 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:22:24 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 2bb4447e425f0bc26d978d5d44770d0d5f1a71d1 Author: Yuan Zhang <[email protected]> Date: Thu Jun 6 11:21:05 2024 +0800 update README commit bd6eca08057bb53f01b5f683a6af26f93053e513 Merge: 517603e4 b0266783 Author: Li Bo <[email protected]> Date: Wed Jun 5 23:12:58 2024 +0800 Merge pull request #100 from Gumpest/main add Conbench commit b02667838d8d7d0e689d4bd81382fe0eacf0e450 Author: Yuan Zhang <[email protected]> Date: Wed Jun 5 21:52:31 2024 +0800 add conbench commit 517603e4735e21ec1a0b7169a7afe2850cecf398 Merge: a7451a2c cb2f2d1c Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:33 2024 +0800 Merge pull request #95 from AtsuMiyai/new_task/upd add MM-UPD commit a7451a2c63bda51e41d614c5f1a73c389c9d0b87 Merge: 057227e0 e5193d26 Author: Li Bo <[email protected]> Date: Tue Jun 4 17:09:04 2024 +0800 Merge pull request #97 from CaraJ7/update Add MathVerse in README.md commit cb2f2d1c457a419509d455a0752ec04285db592d Author: AtsuMiyai <[email protected]> Date: Tue Jun 4 17:36:39 2024 +0900 update utils.py for leaderboard submission commit a0659af89db1d3f531ff9260cd0228a8252ab434 Author: AtsuMiyai <[email protected]> Date: Sun Jun 2 23:28:27 2024 +0900 slightly change query_prompt for the reproduction commit e5193d2679a623c0309de1c9be5b4443b9f91d88 Author: CaraJ7 <[email protected]> Date: Sun Jun 2 17:05:28 2024 +0800 Add MathVerse in README.md commit d392ddbae57979a637d999a0adc0ba8e39113547 Author: AtsuMiyai <[email protected]> Date: Fri May 31 16:09:45 2024 +0900 merge model_specific_prompt_kwargs and dataset_name into each task yaml commit 057227e0479357943c4d9d3be71b23cc7641c7a2 Author: Pu Fanyi <[email protected]> Date: Sat May 4 19:23:39 2024 +0800 Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> commit dc102f6a453fa3e06bde7b5c66b68e266d6b688d Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:15:59 2024 +0900 add upd commit 82fc27bb44300c93dddd292038f25edb44aa3d77 Author: AtsuMiyai <[email protected]> Date: Wed May 29 15:03:30 2024 +0900 add upd commit 0464e31655dab11d19a08ca075ce156e37ba5d25 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:52:21 2024 +0900 add upd commit 23103e63eb754321d00ffacba92a4887d74ad6e2 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:50:32 2024 +0900 add upd commit 0907c3b2c0fb640a0d37f5ef01c0fa998bb04054 Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:46:58 2024 +0900 add upd commit 3e96bb8d7ff2fada86ea96c0f62249798d7f86ca Author: AtsuMiyai <[email protected]> Date: Wed May 29 12:41:21 2024 +0900 add upd commit 093da38e52f4ad2914c2d18e7b1736dc3c8b6145 Author: Bo Li <[email protected]> Date: Mon May 27 10:17:32 2024 +0000 fix compatibility issue of older version llava commit 9a4d4223f02661e33e5a78af1bfd7fadf9f3a67c Author: Bo Li <[email protected]> Date: Mon May 27 09:32:26 2024 +0000 [Fix] import issues of multilingual llava and olympiadbench commit e796d4710b80fe19e2842a6f2f563398c351e3ab Merge: 4e9b71dc d0b6e7c9 Author: Li Bo <[email protected]> Date: Mon May 27 14:19:53 2024 +0800 Merge pull request #87 from vfragoso/vifragos/phi3v Adding microsoft/Phi-3-vision-128k-instruct model. commit d0b6e7c938295540be5449ffeb00896afecdfa91 Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:36:37 2024 +0000 Adding documentation of Phi3v class. commit 654ea7f30624add491b58b2b467f89462fe7d66c Author: Victor Fragoso <[email protected]> Date: Fri May 24 16:25:02 2024 +0000 Adding prompt arguments for Phi3v on MathVista-TestMini commit 67ca9c804540eb5e3bfaf7cff7e7fedee1a8ad19 Author: Victor Fragoso <[email protected]> Date: Fri May 24 13:24:16 2024 +0000 Adding Phi3v model. commit 4e9b71dc73a56d7b823c5dca4ce16d9cbc77ef0a Author: kcz358 <[email protected]> Date: Thu May 23 03:47:36 2024 +0000 Set printing info for llava_hf to debug level commit d4324f52b6f02e5b3f4b978a32b6760ef2400ea2 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:39 2024 +0000 Fix pope random name in pope full commit fc895210654cdcefe6413d986249a31bc4106160 Author: kcz358 <[email protected]> Date: Thu May 23 03:41:14 2024 +0000 Add separated pope tasks by category commit aff6711189229e4ee7defa0e9ee86ead9cf1b4d0 Author: kcz358 <[email protected]> Date: Thu May 9 08:36:02 2024 +0000 Update gitignore commit 80606b5765bfd5d22fdf19327ef6dc805adcd79e Author: kcz358 <[email protected]> Date: Mon May 20 07:45:11 2024 +0000 Comment out Spice in caption task so that don't need to download stanford nlp model commit 1b04c05f74ce1aab9eda5c925026123c17dcd0ff Author: kcz358 <[email protected]> Date: Mon May 20 03:13:13 2024 +0000 Comment out parse result in xcomposer commit 85d132b8ba17815b057a14a21401243919f5ef3e Author: kcz358 <[email protected]> Date: Thu May 16 03:55:39 2024 +0000 Fix instructblip qformer size mismatch and multi-images problem commit ce95d68f6350d816536cbf75f257fd0fba322159 Author: kcz358 <[email protected]> Date: Thu May 16 03:11:41 2024 +0000 Remove redundant code in fuyu commit 0e9d79368ab3625332af4c85f60e3988aea496b0 Author: kcz358 <[email protected]> Date: Thu May 16 01:45:24 2024 +0000 Fix idefics2 llava in the wild bugs commit 0e3dc63524e8ec465df16bb30373d4e895e335d6 Author: kcz358 <[email protected]> Date: Wed May 15 11:07:35 2024 +0000 Better task list_with_num commit 59844a947d79da70eaa686b7e6640d65fdec87d2 Author: Li Bo <[email protected]> Date: Sat May 18 02:35:52 2024 +0800 Update LICENSE commit c256d08e5719e85be1048fc953956674d3adabec Author: Li Bo <[email protected]> Date: Sat May 18 02:29:09 2024 +0800 Update LICENSE commit f14a009ae77dc1dad93bc493c5e95efc8307756b Author: Li Bo <[email protected]> Date: Sat May 18 02:28:03 2024 +0800 Create LICENSE commit 2341ac05dd1b82ea43764fdff362e1fee9c1e765 Merge: 3b3b56a4 67a6ad08 Author: Li Bo <[email protected]> Date: Mon May 13 11:45:26 2024 +0800 Merge pull request #73 from EvolvingLMMs-Lab/kc/qwen_vl_api [Feat] Add qwen vl api commit 3b3b56a489664e29234a6bf522d48647b6f2ac81 Author: kcz358 <[email protected]> Date: Sat May 11 06:11:19 2024 +0000 Fix llava_hf image tokens number issue commit 7fbdaf7965df831f6bc62d3a9eec33aa19bb943d Author: kcz358 <[email protected]> Date: Thu May 9 02:04:10 2024 +0000 Fix endless warning for llava_hf generation commit 5f81e17693fc4c98352001be30c47450318e2001 Author: Bo Li <[email protected]> Date: Thu May 2 06:13:56 2024 +0000 Add model_name parameter to Llava constructor commit 71890d429a63b9a87bcde21b8129a0ba4845a7e8 Author: kcz358 <[email protected]> Date: Tue May 7 03:15:59 2024 +0000 Parse result for llava_hf 1.6 commit bfd518703927de6304fea37ba7fa71239a81fe16 Author: kcz358 <[email protected]> Date: Tue May 7 03:09:56 2024 +0000 Fix llava_hf generation for 1.6 commit b6e3cc389b1000973828b7195e3111d62a8f6703 Author: kcz358 <[email protected]> Date: Mon May 6 08:32:57 2024 +0000 Fix llava conv template for llama3 commit 67a6ad081b759bc8a237ed234832f5181bb276f3 Author: kcz358 <[email protected]> Date: Sun May 5 07:54:52 2024 +0000 Add qwen vl api commit 760a2e0b04670f7ba5474744c02b76a30c2fd336 Merge: 5c91a9d2 e8c9c85d Author: Li Bo <[email protected]> Date: Sun May 5 13:19:48 2024 +0800 Merge pull request #59 from EvolvingLMMs-Lab/add_idefics2 add idefics2 commit 5c91a9d28e6d743a0a65dfdc0f182550ed90ada3 Merge: ef89f655 c9d4e91b Author: Li Bo <[email protected]> Date: Fri May 3 01:18:18 2024 +0800 Merge pull request #36 from cocoshe/main [Fix] repr llava doc commit ef89f655b90562d88ce5e738158022f1c6909c21 Merge: 4fb93efe d57a2f5d Author: Li Bo <[email protected]> Date: Fri May 3 01:17:34 2024 +0800 Merge pull request #56 from gagan3012/main Multilingual LLava bench commit 4fb93efe9fd2b790c027c72b6768ba00348be293 Merge: dac58a81 0a6b210f Author: Li Bo <[email protected]> Date: Fri May 3 01:12:14 2024 +0800 Merge pull request #70 from hunterheiden/hsh/new_task/WebSRC Bugfix: WebSRC should be token-level F1 NOT character-level commit 0a6b210f676d8b2703ce64bc1f278795a39a34d1 Author: Hunter Heidenreich <[email protected]> Date: Thu May 2 09:31:03 2024 -0400 Bugfix: WebSRC should be token-level F1 NOT character-level commit dac58a813e2498b0aa5c86483641c34a2c98f4ef Merge: 2d61ada6 1dadbd3b Author: Li Bo <[email protected]> Date: Thu May 2 14:38:17 2024 +0800 Merge pull request #69 from hunterheiden/hsh/new_task/WebSRC [New Task] WebSRC (multimodal Q&A on web screenshots) commit 1dadbd3bec75ca717aa47aac6d5e494d8ee7eee2 Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 11:07:29 2024 -0400 Add code to enable compilation of submission for WebSRC test split commit bccb19500fe2896b6e8440e56a9f153d0c31956f Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:47:32 2024 -0400 Draft and validate websrc eval on dev split commit 7d8b93613fa1ecf50e8154715f8bf93d3ced9f1c Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:54 2024 -0400 Update main README with new task names commit 829612c494e438abe70ec808bf723fc0d82bffaf Author: Hunter Heidenreich <[email protected]> Date: Wed May 1 10:46:20 2024 -0400 Draft README for WebSRC commit 14cf6f233204f0c329fdd23bfff3f6ca937d3753 Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 30 10:16:21 2024 -0400 Init webSRC commit 2d61ada6012d85cf473c9077f10b4bb56d00a02d Merge: a6fa5a25 d918cff9 Author: Li Bo <[email protected]> Date: Fri Apr 26 14:37:22 2024 +0800 Merge pull request #63 from hunterheiden/hsh/new_task/screenspot New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens commit d918cff9bd8c780921b18794a5ee6ef0b1f0e38a Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:44:34 2024 -0400 slight update commit 5e88530d5787026b6f466e49650ea072241cf2a2 Author: Hunter Heidenreich <[email protected]> Date: Thu Apr 25 11:41:04 2024 -0400 Add README file specific to ScreenSpot commit e58d1415e48a2b8bbf0bfffd04acb278a381f143 Author: Hunter Heidenreich <[email protected]> Date: Wed Apr 24 11:52:33 2024 -0400 Update README to reflect new tasks commit b37d1e54db3f522c763469bbee8f8953e75b939a Author: Hunter Heidenreich <[email protected]> Date: Tue Apr 23 18:33:16 2024 -0400 Create ScreenSpot on clean branch commit a6fa5a2580850cd221a2f13da236665ec59ab69f Merge: b84e04aa 73a5c94b Author: Li Bo <[email protected]> Date: Tue Apr 23 10:34:03 2024 +0800 Merge pull request #61 from tupini07/patch-1 Fix typo in Qwen-VL that was causing "reference before assignment" commit 73a5c94b814b545ced9c7743fa89ad1ef9e98609 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:56:41 2024 -0600 refactor query construction for clarity commit 91b61a73dc6b5a13f5e3ac6d9d3627fe2706907a Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:54:29 2024 -0600 convert contexts to list if necessary and remove unnecessary construction of `questions` commit f7437176afb382d5154bb8f78eb572cda835d6b9 Author: Andrea Tupini <[email protected]> Date: Mon Apr 22 14:47:33 2024 -0600 Fix typo in qwen_vl that was causing "reference before assignment" commit b84e04aa9bcd7ec8678f1cf013f2a9e3363cb686 Merge: ab5c58a3 cee76719 Author: Li Bo <[email protected]> Date: Sat Apr 20 22:03:16 2024 +0800 Merge pull request #60 from CaraJ7/main Add MathVerse commit cee767192d1bd878b4d4c05703e77b60c9c7b23b Merge: e04b23ac ab5c58a3 Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:49:02 2024 +0800 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit e04b23aca089aa033fbed9e19a3182825a2e610e Author: CaraJ7 <[email protected]> Date: Sat Apr 20 21:45:34 2024 +0800 Add MathVerse commit d57a2f5de2c62694fbc7a9fe3f1ba05d987ddc2c Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:21:39 2024 -0700 Add files via upload commit 13373de5b34e2409836bd4c1d536a774a849852d Author: Gagan Bhatia <[email protected]> Date: Fri Apr 12 17:19:49 2024 -0700 Create README.md commit c51eac2bef9991ba7abd639fe96c1e11bffc1b2f Author: Bo Li <[email protected]> Date: Thu Apr 4 17:12:43 2024 +0000 [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) commit c9d4e91b6973ddae73fe204e35364f6960ab8de4 Author: cocoshe <[email protected]> Date: Thu Mar 28 13:38:36 2024 +0800 fix doc * feat: Add LlavaOneVision model to available models chore: Update sqlitedict dependency to version 2.1.0 * Revert "Squashed commit of the following:" This reverts commit 11b00999df3c43cb225482e030b791b2d454124c. * Refactor available models in lmms_eval Remove duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary in lmms_eval/models/__init__.py. * fix: Handle import errors in lmms_eval models/__init__.py The code changes in this commit fix the handling of import errors in the lmms_eval/models/__init__.py file. Previously, when an import error occurred, the code simply ignored it. This commit updates the code to log an error message using the logger module when an import error occurs. This commit also removes duplicate entries for "llava_hf", "llava_onevision", and "longva" in the AVAILABLE_MODELS dictionary. Recent user commits: - Refactor available models in lmms_eval - Revert "Squashed commit of the following:" - feat: Add LlavaOneVision model to available models - chore: Update sqlitedict dependency to version 2.1.0 * fix: Handle import errors in lmms_eval models/__init__.py * chore: Remove unused imports in lmms_eval/models/__init__.py and lmms_eval/tasks/vcr_wiki/utils.py * Remove unused imports in lmms_eval/tasks/vcr_wiki/utils.py * chore: Update lmms_eval/tasks/vcr_wiki/utils.py This commit updates the `lmms_eval/tasks/vcr_wiki/utils.py` file. It removes unused imports and fixes the condition for loading Spacy models based on the `load_package` value in the config file. Additionally, it adds a debug log message when the Spacy models are not loaded due to `load_package` being set to False. Remove unused imports in `lmms_eval/tasks/vcr_wiki/utils.py` * feat: Add new subtasks to overall score calculation The code changes in this commit add new subtasks to the overall score calculation in the `overall_score` function. The subtasks "ScanQA", "BLINK", "MathVerse", "SciVerse", and "Mantis" are included in the `categories` dictionary. This ensures that the scores for these subtasks are calculated and included in the evaluation results. Remove unused imports and update subtask categories in `utils.py` * feat: Add new subtasks to overall score calculation * chore: Update lmms_eval/tasks/llava_interleave_bench/_default_template_interleave_yaml Update the image aspect ratio in the default template for the llava_interleave_bench task. Change the value of "image_aspect_ratio" from "original" to "pad". This ensures that the generated images have a padded aspect ratio. * if no response directly return 0 * Squashed commit of the following: commit b2a009b6bbf8353172f5a1dd9c29ea1f67610c02 Author: Pu Fanyi <[email protected]> Date: Mon Jul 15 19:12:25 2024 -0700 if no response directly return 0 (#142) commit 5fc5f2f5acf454fc99448b0d62eb52b4bffba0d5 Author: Kaichen Zhang - NTU <[email protected]> Date: Tue Jul 16 10:12:11 2024 +0800 Add Muirbench (#143) * handle gen kwargs in internvl2 * Add muirbench * Add files via upload (cherry picked from commit 59bb9af306b554bae841f0934502b470b163390d) * update --------- Co-authored-by: Fanyi Pu <[email protected]> Co-authored-by: Yan Shu <[email protected]> * Fix llava onevision loglikelihood video bug * LiveBench July (#146) * claude auto detect json mode * extract information * use claude to generate * fix bugs * fix * generate data * chore: Update dataset name and version for live_bench task * gpt-4-turbo => gpt-4o * chore: Update dataset capture settings in create_dataset.py * everything use gpt-4o * websites * livebench_july * Refactor code to simplify data assignment in example.ipynb * chore: Update dataset name for live_bench task * Add xcomposer2d5 from fanyi, revise something for better usage (#145) * internvl2 * fix some bugs * fix * lint * feat: Add XComposer2D5 model to AVAILABLE_MODELS * xcomposer * Fix llava vid error when using public * Fix xcomposer2d5 * Add generation tokens --------- Co-authored-by: Fanyi Pu <[email protected]> * Dev/ov evals (#147) * fix doc * [WIP] adding mmbench dev evaluation (#75) * WIP * Update GPT evaluation model name and sys prompt * 🛠️ Scale accuracy to percentage The accuracy value is now multiplied by 100 in the aggregation function to represent it as a percentage. Regarding the evaluation process, `math` module importation and refactoring reduce progress log verbosity by logging every 100 evaluations instead of 10. It prevents potential logging overflow. Handling of NaN values is added to ensure 'default_value' is set in case of missing data, avoiding errors in split, category, and l2-category assignments. Finally, reporting of categorical and l2-categorical accuracies is streamlined through a new `calculate_hit_rates` function, improving code readability and maintenance. Issue refs: #1427, #1533 * Update GPT evaluation model name and API configuration * Refactor MMBench_Evaluator class to handle missing columns * Add print statements for detailed results in MMBench-CN(CC), MMBench-CN(Dev), and MMBench-EN(Dev) evaluations * Refactor MMBench-CN and MMBench-EN evaluation functions * 🔄 Refactor result processing and logging logic - Simplified the result processing functions across different utility modules (`cc_utils.py`, `cn_utils.py`, `en_utils.py`) to unify the handling of multiple-choice options. Now, all options ("A" to "E") are dynamically added to the result data, and default to "nan" if not provided in the document. - Removed redundant keys directly from the process results dict creation to avoid clutter and align with the new dynamic addition of options. - In `mmbench_evals.py`, removed the unnecessary check for all splits being 'dev' and streamlined the evaluation loop by eliminating the progress bar (tqdm) for a cleaner log output. - Commented-out code and verbose logging during evaluation, which may have interfered with performance, has been removed for a more efficient and less intrusive logging experience. This cleanup reduces redundancy in the codebase and improves evaluation performance. Refs #2045 --------- Co-authored-by: Bo Li <[email protected]> (cherry picked from commit a19278c2ea6ddcbca64d3cc7f4efec7fe5775121) * Create README.md * Add files via upload * Add MathVerse * Fix typo in qwen_vl that was causing "reference before assignment" * convert contexts to list if necessary and remove unnecessary construction of `questions` * refactor query construction for clarity * Create ScreenSpot on clean branch * Update README to reflect new tasks * Add README file specific to ScreenSpot * slight update * Init webSRC * Draft README for WebSRC * Update main README with new task names * Draft and validate websrc eval on dev split * Add code to enable compilation of submission for WebSRC test split * Bugfix: WebSRC should be token-level F1 NOT character-level * Add qwen vl api * Fix llava conv template for llama3 * Fix llava_hf generation for 1.6 * Parse result for llava_hf 1.6 * Add model_name parameter to Llava constructor * Fix endless warning for llava_hf generation * Fix llava_hf image tokens number issue * Create LICENSE * Update LICENSE * Update LICENSE * Better task list_with_num * Fix idefics2 llava in the wild bugs * Remove redundant code in fuyu * Fix instructblip qformer size mismatch and multi-images problem * Comment out parse result in xcomposer * Comment out Spice in caption task so that don't need to download stanford nlp model * Update gitignore * Add separated pope tasks by category * Fix pope random name in pope full * Set printing info for llava_hf to debug level * Adding Phi3v model. * Adding prompt arguments for Phi3v on MathVista-TestMini * Adding documentation of Phi3v class. * [Fix] import issues of multilingual llava and olympiadbench * fix compatibility issue of older version llava * add upd * add upd * add upd * add upd * add upd * add upd * Group MMMU images into one image (#83) * update * update font * Add matplotlib.font_manager import in utils.py * Refactor font handling in add_order_label function in utils.py * group mmmu --------- Co-authored-by: Li Bo <[email protected]> * merge model_specific_prompt_kwargs and dataset_name into each task yaml * Add MathVerse in README.md * slightly change query_prompt for the reproduction * update utils.py for leaderboard submission * add conbench * update README * Update README.md * init include vcr * modify the form of VCR * switch logic * add crossed_text to vcr_wiki output * include the try-except logic for spacy * update vcr_wiki tasks * update vcr_wiki tasks in README.md * include std and confidence interval * update gpt-3.5-turbo version * update gpt-3.5-turbo version * chore: Remove unnecessary files and code related to live_bench and sft_eval tasks * Bump version to 0.2.0.dev0 * chore: Update lmms-eval to support video evaluations for LLaVA models * Update llava conv_template in lmms_eval/models/llava.py * Update image alignment in README.md * chore: Update lmms-eval to support video evaluations for LLaVA models * chore: Update lmms-eval to support video evaluations for LLaVA models * Update README.md * Update README.md * update aggregation function for vcr_wiki * update README.md * Update README.md * update version * add II-Bench * fix dataset_path * Add qbench, qbench2, abench; fix phi3v as its current implementation does not support multi-image * add tinyllava * LongVideoBench support: image LMMs (idefics2, phi3) and video LMMs (LLaVA-Next-Video-34B) * fix #117, allow auto download with tar format videos * fix #117, allow auto download with tar format videos * fix typo * feat: Add support for auto downloading tar format videos * Release llava-wilder * chore: Update dependencies to fix potential risks and improve compatibility * tutorial * docs * update preparation * small fix * small fix * lint * to sh script * update readme * Remove handling non-visual loop in llava * Add llava_hf back to registry * Update README.md * Update README.md * update ablation for videomme datasets * chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. * chore: Remove unused models from lmms_eval package * feat: Allow loading model configurations from other packages * feat: Allow including external tasks from plugins * chore: Add loguru for logging in lmms_eval package * Remove unnecessary lines since use batched visuals now in llava * Add longva * Revise model registry for llava_hf and longva * Delete unnecessary lines * Remove unnecessary lines for video llava * Update pyproject.toml * Update activitynetqa_generation.yaml * Fix vid mme post prompt issue * new task gqa-ru * add mmbench_ru_dev * change prompt to ru * create new task vitatecs * Update README.md * Add wild vision 0617 * Hardcode to keep image for wild vision * Fixing scoring logic * Fixing dataset name * Fixing handling None filtered score * Add detailcaps * Add install capture_metric in env * Add files via upload * feat: Add tie_weights parameter to Llava model initialization * Upgrade lmms-eval to support more models and evaluation tasks * Upgrade lmms-eval to version 0.2.1 * Rename xcomposer 4KHD * chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py * Update utils.py * Update _default_template_vcr_yaml * add process sync via temp file in lmms_eval/evaluator.py * Update utils.py * Update _default_template_vcr_yaml * Add muirbench * Squashed commit of the following: commit dfdba507b5fbe985b0030ffec575f9f2638bc1ed Author: Li Bo <[email protected]> Date: Tue Jul 16 11:13:52 2024 +0800 merge ov evals (#144) * chore: Update gpt_eval_model_name to "gpt-3.5-turbo" in mathvista.yaml * Squashed commit of the following: commit 994c9f97a2f8db3e9b7d7933d1e1680acde5b70b Author: Yan Shu <[email protected]> Date: Mon Jul 8 17:21:23 2024 +0800 Add files via upload * Squashed commit of the following: commit ec9904af65ac296756db05e958a31c36eb3d9058 Author: Bo Li <[email protected]> Date: Wed Jul 10 12:08:08 2024 +1000 chore: Update lmms_eval/models/vila.py and lmms_eval/tasks/__init__.py commit 0ab768c8bad11da505c9acc884a54136cf5980ce Author: kcz358 <[email protected]> Date: Tue Jul 9 02:08:52 2024 +0000 Rename xcomposer 4KHD commit 7c1478f1fdc8d8051ddadb11435f2e2877e874d0 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:55:56 2024 +1000 Upgrade lmms-eval to version 0.2.1 commit ec9c5a1c480ad702bdb1b17bed766594463d0da5 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:52:23 2024 +1000 Upgrade lmms-eval to support more models and evaluation tasks commit ce6f82e988e530bb073e789e98de571f8981f4cc Author: Bo Li <[email protected]> Date: Tue Jul 9 11:43:41 2024 +1000 feat: Add tie_weights parameter to Llava model initialization commit 87e0462cd4dbb4ab4b05107f7e87523419003309 Merge: 4c5d1a6f a2b07c09 Author: Bo Li <[email protected]> Date: Tue Jul 9 11:37:12 2024 +1000 Fix gen kwargs image aspect ratio in internvl2 commit a2b07c09ff9550ccbdbdf941385003696f9379cc Merge: 44e79980 59bb9af3 Author: Li Bo <[email protected]> Date: Tue Jul 9 09:15:56 2024 +0800 Merge pull request #137 from shuyansy/main add MLVU task commit 59bb9af306b554bae841f0934502b470b163390d Author: Yan Shu <[email protected]> Date: Mon Jul 8 16:56:50 2024 +0800 Add files via upload commit 44e79980f80da75b70391276a65de4aec5a1b708 Merge: 8109ff79 62a5de53 Author: Li Bo <[email protected]> Date: Mon Jul 8 11:53:06 2024 +0800 Merge pull request #136 from Dousia/main Add detailcaps commit 62a5de53deeafe76680a595b154bf25c0e22a52c Author: ByteDance <[email protected]> Date: Sun Jul 7 23:24:19 2024 +0800 Add install capture_metric in env commit e6941ed29b917702157850c88163186fc1765e41 Author: ByteDance <[email protected]> Date: Sun Jul 7 23:04:13 2024 +0800 Add detailcaps commit 8109ff7949c7686994ed3299ee1bdb11421ec89c Merge: c5258c96 f07fd371 Author: Li Bo <[email protected]> Date: Tue Jul 2 23:05:12 2024 +0800 Merge pull request #133 from EvolvingLMMs-Lab/dev/wild_vision Add wild vision bench commit f07fd3717c7592f3387b520456053046545466a5 Author: kcz358 <[email protected]> Date: Mon Jul 1 15:10:02 2024 +0000 Fixing handling None filtered score commit ddd7ee1bda57a2c8f83c3a34f3c068c7bb2fbbf8 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:25:42 2024 +0000 Fixing dataset name commit 375b8ccf341aae7048e1ab97ee8a7d30f9dd3137 Author: kcz358 <[email protected]> Date: Mon Jul 1 08:24:51 2024 +0000 Fixing scoring logic commit 4254530f472b2aad5b5a26d6c6622c9856ea8ade Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:57 2024 +0000 Hardcode to keep image for wild vision commit 2d61dd7599dad8fa0c6e5006841954e53f97581c Author: kcz358 <[email protected]> Date: Mon Jul 1 06:06:38 2024 +0000 Add wild vision 0617 commit c5258c96b99c9c6db0e40b0d88430224bb497404 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:53:31 2024 +0800 Update README.md commit b41d080daa4a262fd5afaca9b19593805a6ee473 Merge: fb59c6cb 92d3d4ce Author: Li Bo <[email protected]> Date: Mon Jul 1 11:47:09 2024 +0800 Merge pull request #129 from Dannoopsy/mmbench_ru add task MMBench-ru commit fb59c6cbe42377c72161f9674ed5884064763911 Merge: 92ceee0c 7010fa23 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:58 2024 +0800 Merge pull request #128 from Dannoopsy/gqa-ru add task gqa-ru commit 92ceee0c44270d7d03d33853a83855e76fd7d728 Merge: 51c1403e c7696149 Author: Li Bo <[email protected]> Date: Mon Jul 1 11:46:16 2024 +0800 Merge pull request #130 from lscpku/vitatecs Add task VITATECS commit c76961496f509f4bb5c54868fc8b5c7097c1089a Author: lscpku <[email protected]> Date: Fri Jun 28 20:37:06 2024 +0800 create new task vitatecs commit 92d3d4ced0898265ef334535184522376cad1a36 Author: Dannoopsy <[email protected]> Date: Fri Jun 28 12:21:05 2024 +0300 change prompt to ru commit 1054d8adbd0825e34cfb9006c765add2db45b9ef Author: Dannoopsy <[email protected]> Date: Thu Jun 27 17:17:29 2024 +0000 add mmbench_ru_dev commit 51c1403ed8d14fdfb1cb7bd947425e0d5fcafc6e Merge: f7f15f45 ba3c1f4a Author: Li Bo <[email protected]> Date: Fri Jun 28 00:14:10 2024 +0800 Merge pull request #126 from lorenzomammana/feature/external-package-integration External package integration using plugins commit ba3c1f4af009f554b8751bbea05a178eaf1a4d6a Merge: 9f2145de f7f15f45 Author: Lorenzo Mammana <[email protected]> Date: Thu Jun 27 15:38:10 2024 +0000 Merge branch 'main' into feature/external-package-integration commit 7010fa2382cbae19abaedd9f640a0c5d23eea673 Author: Dannoopsy <[email protected]> Date: Tue Jun 25 11:11:37 2024 +0000 new task gqa-ru commit f7f15f451bd740e99fd8e7bfbe723992705cf71a Author: kcz358 <[email protected]> Date: Tue Jun 25 06:41:13 2024 +0000 Fix vid mme post prompt issue commit 5bb1722e303746d65b14d32bee43be2a88254d9a Author: Li Bo <[email protected]> Date: Sun Jun 23 22:31:16 2024 +0800 Update activitynetqa_generation.yaml commit d44a1d1596d71fcffdc329430c3fbc9c19263686 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:25 2024 +0800 Update pyproject.toml commit cc748e9d77f46d567143e57c950669bf8c0dbc5a Merge: 7d4d8af3 a7342ac7 Author: Li Bo <[email protected]> Date: Sun Jun 23 14:02:02 2024 +0800 Merge pull request #125 from EvolvingLMMs-Lab/dev/interleave [Model] aligned llava-interleave model results on video tasks commit a7342ac7f79f320fd0ac897082ee9eff51162d97 Author: kcz358 <[email protected]> Date: Sat Jun 22 12:07:13 2024 +0000 Remove unnecessary lines for video llava commit 0856d476b6bd9c9ac0dae4ad0b5fc845ae1dbdef Merge: 91a3d9c6 7d4d8af3 Author: Li Bo <[email protected]> Date: Sat Jun 22 13:57:31 2024 +0800 Merge branch 'main' into dev/interleave commit 91a3d9c6727e227db0b15881bc3c371888c4b4ae Author: kcz358 <[email protected]> Date: Sat Jun 22 02:57:08 2024 +0000 Delete unnecessary lines commit 1c051b10ef00faa47451068043f3e387704fe58a Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:41 2024 +0000 Revise model registry for llava_hf and longva commit d57f2e239ce730ba0e76502bf521f57e5868d270 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:38:24 2024 +0000 Add longva commit 99ec17ae9e3fdf7d2b8b74024935da28bd47c456 Author: kcz358 <[email protected]> Date: Fri Jun 21 08:35:39 2024 +0000 Remove unnecessary lines since use batched visuals now in llava commit ae72f2180bd5275769c6aded2d212673daac7291 Author: Bo Li <[email protected]> Date: Thu Jun 20 18:15:32 2024 +0000 chore: Add loguru for logging in lmms_eval package commit 9f2145dec51bcba853eab4ba037cd19d1193c8ca Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:40:41 2024 +0000 feat: Allow including external tasks from plugins commit b8540ec84d15bb13653a6067bc210921906d94e0 Author: Lorenzo Mammana <[email protected]> Date: Wed Jun 5 13:04:55 2024 +0000 feat: Allow loading model configurations from other packages commit d1d4829069b5b8799ccafe7616ddfb08ac2821da Author: Bo Li <[email protected]> Date: Thu Jun 20 12:11:03 2024 +0000 chore: Remove unused models from lmms_eval package commit da62829f4a16aaf652d08985080f0df509b9dce3 Author: Bo Li <[email protected]> Date: Thu Jun 20 12:07:09 2024 +0000 chore: Handle ImportError when importing models Handle the ImportError exception when importing models in the lmms_eval package. This change adds a try-except block to catch the ImportError and print an error message indicating the failed import. This will help with troubleshooting and identifying any issues with the model imports. commit 7d4d8af3ad6962f26260ce515c790b9eaf7f5103 Merge: 7f6cfa5d 86139ce8 Author: Li Bo <[email protected]> Date: Thu Jun 20 20:02:12 2024 +0800 Merge pull request #120 from EvolvingLMMs-Lab/pufanyi/hf_dataset_docs Add docs for datasets upload to HF commit 7f6cfa5d92942d857ed259db655a24e5ea3132d9 Author: choiszt <[email protected]> Date: Thu Jun 20 15:14:21 2024 +0800 update ablation for videomme datasets commit 86139ce84ee8323c0c276fbdd8cf8e5690653b40 Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:59 2024 +0800 Update README.md commit a2cb9f7138c1f8f004d3b332068cef2d45726dbd Author: Li Bo <[email protected]> Date: Thu Jun 20 13:30:29 2024 +0800 Update README.md commit f130e6ae168173ca8c649824d1fde9ac93b5a8f2 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:55:30 2024 +0000 Add llava_hf back to registry commit 71c37d38675319db3447eb868e33883448159450 Author: kcz358 <[email protected]> Date: Thu Jun 20 03:54:33 2024 +0000 Remove handling non-visual loop in llava commit 116eb19ad4550177d8b4de66142bf60c6ced1711 Author: Fanyi Pu <[email protected]> Date: Thu Jun 20 02:11:18 2024 +0800 update readme commit 3466d567d16721db27c27f5bec52eb3bef6f7965 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:52 2024 +0800 to sh script commit 93554726616aae10f5a78c9f748f97c5d14cedc0 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:37:04 2024 +0800 lint commit a381d5df198b22151e5ef2a1df6c8184757a1025 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:48 2024 +0800 small fix commit 4772eee0fb09d77e66f4ca52ef301bedbcb9ba9e Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:36:43 2024 +0800 small fix commit 1751954c9d0da906ab1007d110e42c40d374520d Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:35:05 2024 +0800 update preparation commit 908a1618d29dba0195eb6f280e8610208804534c Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:23:24 2024 +0800 docs commit d49a65e961aa143de72aa1dae3105f2b9c24d43c Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 15:04:16 2024 +0800 tutorial commit 989ef83de930f40c666a6c39a60140f4586ca4d5 Author: Bo Li <[email protected]> Date: Wed Jun 19 06:51:19 2024 +0000 chore: Update dependencies to fix potential risks and improve compatibility commit 84bcd6f6f7d7d160fa8d882ca67ebe9056353ca4 Author: kcz358 <[email protected]> Date: Wed Jun 19 10:25:58 2024 +0800 Release llava-wilder commit 79402ea60762031bf889b618f0976ad78206ab73 Author: Fanyi Pu <[email protected]> Date: Wed Jun 19 07:44:26 2024 +0800 feat: Add support for auto downloading tar format videos commit 03b2f7cbd0e4d810fb75c8340c21213b60be9295 Merge: c51eac2b b4e8ca6d Author: Bo Li <[email protected]> Date: Tue Jun 18 17:01:03 2024 +0000 Merge branch 'main' of https://github.com/EvolvingLMMs-Lab/lmms-eval commit b4e8ca6de57237d4e53771c82f2ea3c379adb5b3 Merge: 5e2459b6 7d3536e9 Author: Li Bo <[email protected]> Date: Tue Jun 18 13:13:38 2024 +0800 Merge pull request #114 from zjysteven/add-tinyllava add tinyllava commit 7d3536e90e05d6f732c349a43ab04f3d59c0bafb Author: Jingyang Zhang <[email protected]> Date: Mon Jun 17 17:57:02 2024 -0400 fix typo commit 2217602cf93c297c35ffcc25117f382a60648b0b Merge: eb88c556 5e2459b6 Author: Jingyang Zhang <[email protected]> Date: Sun Jun 16 10:56:05 2024 -0400 Merge branch 'EvolvingLMMs-Lab:main' into add-tinyllava commit 5e2459b64c746020b9aa8e558e8c7aa65c9e3cf7 Merge: d49a032f ae92f69d Author: Li Bo <[email protected]> Date: Sun Jun 16 17:59:19 2024 +0800 Merge pull request #118 from teowu/main Fix the potential risk by PR #117 commit ae92f69d44c7d849a5c990a4a607930eb4c33770 Merge: 8f6e8460 d49a032f Author: Teo (Timothy) Wu Haoning <[email protected]> Date: Sun Jun 16 15:32:13 2024 +0800 Merge branch 'EvolvingLMMs-Lab:main' into main commit 8f6e8460cf65c9a82ae195d40edf44af1f12e6cc Author: teowu <[email protected]> Date: Sun Jun 16 07:27:28 2024 +0000 fix #117, allow auto download with tar format videos comm…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While evaluating model responses with metrics that require GPUs to perform computation, the current codebase uses
cuda:0
to compute the metric score and lets other GPU processes waiting in barrier. The barriers take up a large amount of computation, slowing down metric computation on these GPUs. For metrics like CAPTURE used for detail image caption evaluation, this leads to intolerably long evaluation time.In this PR, a process synchronization operation is added to metric evaluation code. This operation synchronizes processes with a temp file stored in
{cli_args.output_path}/rank0_metric_eval_done.txt
. This synchronization does not take up the computation of GPUs with rank>0.