Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, lmms-eval team,
This PR incorporates our recent work, MEGA-Bench, a multimodal evaluation suite with over 500 real-world tasks and 45 metrics.
The evaluation process involves two steps: 1) run lmms-eval to produce the response/submission file; 2) run our evaluator with 45 diverse metrics to get the scores and multi-dimensional breakdown results.
Example response/submission file generation:
# Core set (440 tasks) python3 -m accelerate.commands.launch \ --num_processes=8 \ -m lmms_eval \ --model llava_onevision \ --tasks megabench_core \ --batch_size 1 \ --log_samples \ --log_samples_suffix llava_ov_megabench_core \ --output_path ./logs/ \ --model_args=pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwen
Detailed steps are recorded in this README file.
Please advise on the format of the README or other details, thanks!