推理qwen2-vl-7b, vllm模式失败，到底应该用什么版本？

[gouqi666]

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)

scripts:

76GiB

CUDA_VISIBLE_DEVICES=0,1,2,3
MAX_PIXELS=401408
swift infer
--model /home/data/gouqi/ms-swift/swift-qwen2vl-7b/v4-20250108-174040/checkpoint-2382
--max_batch_size 64
--infer_backend vllm
--val_dataset '/home/data/gouqi/ms-swift/swift/llm/dataset/data/4k_bad_test_new.json'
--template qwen2_vl
--max_length 2048
--stream true
--temperature 0
--top_k 1
--max_new_tokens 300

error:

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)

Additional context
Add any other context about the problem here(在这里补充其他信息)

试了vllm几个版本（0.6.0，0.6.6）都不行，没有看到应该用哪个版本，还有vllm 推理和 default推理模式 都不能batch 推理吗？

[gouqi666]

@DarkLight1337 @jeejeelee @youkaichao

[DarkLight1337]

根據facebookresearch/how-to-autorl#18 ，你可能需要更新你的Python版本

[Jintao-Huang]

使用python3.10或者降低vllm的版本

[gouqi666]

@Jintao-Huang @DarkLight1337 谢谢你们，但是还遇到另外一个问题，当我使用truncation_strategy='right'时，vllm推理会报错. 通过监控gpu memory，发现gpu显存可能会炸，但是我设置的batch_size=1,并且max_length=200依然是这样，想问一下原因
scripts:

76GiB
export CUDA_VISIBLE_DEVICES=0,1,2,3
export CUDA_LAUNCH_BLOCKING=1
export MAX_PIXELS=401408
swift infer
--model /home/data/gouqi/ms-swift/swift-qwen2vl-7b/v4-20250108-174040/checkpoint-2382
--max_batch_size 1
--infer_backend vllm
--truncation_strategy right
--dataset '/home/data/gouqi/ms-swift/swift/llm/dataset/data/4k_bad_test_new.json'
--template qwen2_vl
--system 'You are an expert in B2B foreign trade for cross-border e-commerce, specializing in e-commerce search evaluation.'
--max_length 200
--result_path /home/data/gouqi/ms-swift/swift-qwen2vl-7b/v4-20250108-174040/checkpoint-2382/infer_result_4k_bad
--temperature 0
--top_k 1
--max_new_tokens 200

ouput:

WARNING 01-09 12:39:54 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:54 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:54 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:54 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:55 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:55 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:55 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:55 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
WARNING 01-09 12:39:55 preprocess.py:262] Passing multi_modal_data in TokensPrompt isdeprecated and will be removed in a future update
1%|▉ | 1/174 [00:22<1:03:38, 22.07s/it]INFO 01-09 12:40:03 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250109-124003.pkl...
INFO 01-09 12:40:03 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20250109-124003.pkl.
Exception in callback VllmEngine.patch_remove_log..new_log_task_completion(error_callback=>)(<Task finishe...te(handle)')>) at /home/data/gouqi/ms-swift/swift/llm/infer/infer_engine/vllm_engine.py:391 handle: <Handle VllmEngine.patch_remove_log.<locals>.new_log_task_completion(error_callback=<bound method...7f6db0889460>>)(<Task finishe...te(handle)')>) at /home/data/gouqi/ms-swift/swift/llm/infer/infer_engine/vllm_engine.py:391>
Traceback (most recent call last):
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/worker/model_runner.py", line 1737, in execute_model
logits = self.model.compute_logits(hidden_or_intermediate_states,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/model_executor/models/qwen2_vl.py", line 1186, in compute_logits
return self.language_model.compute_logits(hidden_states,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 487, in compute_logits
logits = self.logits_processor(self.lm_head, hidden_states,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/model_executor/layers/logits_processor.py", line 65, in forward
logits = self._get_logits(hidden_states, lm_head, embedding_bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/model_executor/layers/logits_processor.py", line 88, in _get_logits
logits = lm_head.linear_method.apply(lm_head,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/conda/envs/gouqi2/lib/python3.12/site-packages/vllm/model_executor/layers/vocab_parallel_embedding.py", line 41, in apply
return F.linear(x, layer.weight, bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)

[DarkLight1337]

Please follow https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html