[Benchmark] Add MEGA-Bench

[TianhaoLiang2000]

Hi, VLMEvalKit team,

This PR incorporates our recent work, MEGA-Bench, a multimodal evaluation suite with over 500 real-world tasks and 45 metrics.

The evaluation process involves two steps: 1) run VLMEvalKit to produce the response/submission file; 2) run our evaluator with 45 diverse metrics to get the scores.

Example usage:

python3 run.py
--data MEGABench_core_single_image_16frame
--model Qwen2-VL-7B-Instruct
--verbose
--work-dir ~/LMUData

[mary-0830]

Hi, I would like to ask when this benchmark can be added?

[FangXinyu-0913]

Hi @mary-0830, @TianhaoLiang2000. I'm sorry for not getting back to you sooner. Due to the complexity of this dataset, we will conduct integration this week. Thank you for being so patient.

[mary-0830]

Hi @TianhaoLiang2000 @FangXinyu-0913. I want to ask why the mega-benchmark (https://huggingface.co/datasets/TIGER-Lab/MEGA-Bench), which is a benchmark for image and text, why was it added to the video-vqa this time?

[FangXinyu-0913]

MEGA-Bench is a benchmark containing both the image (single-image and multiple-image) and video, so in order to be compatible with multiple modalities, it has been added to video-vqa. In the actual code processing, the input for video and single/multiple image inputs follow different logic to construct prompts, which does not affect the evaluation results.

[mary-0830]

MEGA-Bench is a benchmark containing both the image (single-image and multiple-image) and video, so in order to be compatible with multiple modalities, it has been added to video-vqa. In the actual code processing, the input for video and single/multiple image inputs follow different logic to construct prompts, which does not affect the evaluation results.

嗨，谢谢你的回复。有一个疑问，如果我想在抽取一部分的子集进行测试，我应该如何修改呢？比如，我只选取了Information_Extraction中的App_Function_Understanding和Compound_Search_and_Calculate。

[mary-0830]

MEGA-Bench is a benchmark containing both the image (single-image and multiple-image) and video, so in order to be compatible with multiple modalities, it has been added to video-vqa. In the actual code processing, the input for video and single/multiple image inputs follow different logic to construct prompts, which does not affect the evaluation results.

嗨，谢谢你的回复。有一个疑问，如果我想在抽取一部分的子集进行测试，我应该如何修改呢？比如，我只选取了Information_Extraction中的App_Function_Understanding和Compound_Search_and_Calculate。

嗨，这个问题我已经解决啦。现在又有一个疑问，为什么在每次测试时都需要对图片进行resized和rgba的转换呢，我发现在上一次执行的时候已经转换好了并保存。是不是可以加一个若在图片路径下存在了，就不需要执行这个转换了呢？

[FangXinyu-0913]

MEGA-Bench is a benchmark containing both the image (single-image and multiple-image) and video, so in order to be compatible with multiple modalities, it has been added to video-vqa. In the actual code processing, the input for video and single/multiple image inputs follow different logic to construct prompts, which does not affect the evaluation results.

嗨，谢谢你的回复。有一个疑问，如果我想在抽取一部分的子集进行测试，我应该如何修改呢？比如，我只选取了Information_Extraction中的App_Function_Understanding和Compound_Search_and_Calculate。

嗨，这个问题我已经解决啦。现在又有一个疑问，为什么在每次测试时都需要对图片进行resized和rgba的转换呢，我发现在上一次执行的时候已经转换好了并保存。是不是可以加一个若在图片路径下存在了，就不需要执行这个转换了呢？

Thank you for your proposal and support. You can modify it and submit a Pull request, and we will review it and merge it into the main branch.