[Bug] LogitsWarper deprecated in transformers? (trying to run Qwen/Qwen2.5-VL-72B-Instruct)

[josephrocca]

Describe the bug

There is some hype around Qwen/Qwen2.5-VL-72B-Instruct due to benchmarks, so I wanted to test, but it needs latest transformers. They recommend this command in their readme:

pip install git+https://github.com/huggingface/transformers accelerate

So I tried it using this Docker image:

openmmlab/lmdeploy:v0.7.0.post2-cu12

with this command:

bash -c "pip install git+https://github.com/huggingface/transformers accelerate && lmdeploy serve api_server Qwen/Qwen2.5-VL-72B-Instruct --model-name Qwen/Qwen2.5-VL-72B-Instruct --server-port 3000"

But got this error:

2025-01-29 18:36:27,564 - lmdeploy - �[33mWARNING�[0m - archs.py:55 - Try to run with pytorch engine because `/root/.cache/huggingface/hub/models--Qwen--Qwen2.5-VL-72B-Instruct/snapshots/699299cdf34b788496a8aac38771054ab8557d7d` is not explicitly supported by lmdeploy.
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 33, in <module>
    sys.exit(load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')())
  File "/opt/lmdeploy/lmdeploy/cli/entrypoint.py", line 39, in run
    args.run(args)
  File "/opt/lmdeploy/lmdeploy/cli/serve.py", line 300, in api_server
    run_api_server(args.model_path,
  File "/opt/lmdeploy/lmdeploy/serve/openai/api_server.py", line 1008, in serve
    VariableInterface.async_engine = pipeline_class(model_path=model_path,
  File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 276, in __init__
    self._build_pytorch(model_path=model_path, backend_config=backend_config, **kwargs)
  File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 327, in _build_pytorch
    from lmdeploy.pytorch.engine import Engine
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/__init__.py", line 2, in <module>
    from .engine import Engine
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 22, in <module>
    from .logits_process import FusedLogitsProcessor, SamplingInputs
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/logits_process.py", line 8, in <module>
    from transformers.generation.logits_process import LogitsWarper
ImportError: cannot import name 'LogitsWarper' from 'transformers.generation.logits_process' (/opt/py3/lib/python3.10/site-packages/transformers/generation/logits_process.py)

I believe it is related to this:

• Generate: unify LogitsWarper and LogitsProcessor huggingface/transformers#32626

[grimoire]

It should be ok to remove the class inheritance in

lmdeploy/lmdeploy/pytorch/engine/logits_process.py

Line 276 in 637435f

class FusedLogitsProcessor(LogitsWarper):

[josephrocca]

Thank you, that does seem to have solved the issue. Though I do get another issue:

2025-01-30 14:19:01,849 - lmdeploy - �[33mWARNING�[0m - archs.py:55 - Try to run with pytorch engine because `/root/.cache/huggingface/hub/models--Qwen--Qwen2.5-VL-72B-Instruct/snapshots/699299cdf34b788496a8aac38771054ab8557d7d` is not explicitly supported by lmdeploy.
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 33, in <module>
    sys.exit(load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')())
  File "/opt/lmdeploy/lmdeploy/cli/entrypoint.py", line 39, in run
    args.run(args)
  File "/opt/lmdeploy/lmdeploy/cli/serve.py", line 300, in api_server
    run_api_server(args.model_path,
  File "/opt/lmdeploy/lmdeploy/serve/openai/api_server.py", line 1008, in serve
    VariableInterface.async_engine = pipeline_class(model_path=model_path,
  File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 276, in __init__
    self._build_pytorch(model_path=model_path, backend_config=backend_config, **kwargs)
  File "/opt/lmdeploy/lmdeploy/serve/async_engine.py", line 328, in _build_pytorch
    self.engine = Engine(model_path=model_path, tokenizer=self.tokenizer, engine_config=backend_config)
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 148, in __init__
    self.model_agent = build_model_agent(model_path,
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 703, in build_model_agent
    model_agent = BaseModelAgent(model_path,
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 208, in __init__
    self.patched_model = self._build_model(model_path, adapters, device=device)
  File "/opt/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 229, in _build_model
    patched_model = build_patched_model(self.model_config, device=device)
  File "/opt/py3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/opt/lmdeploy/lmdeploy/pytorch/models/patch.py", line 195, in build_patched_model
    return build_model_from_hf_config(model_config, dtype=dtype, device=device)
  File "/opt/lmdeploy/lmdeploy/pytorch/models/patch.py", line 185, in build_model_from_hf_config
    model_cls = _get_model_class(model_config, module_map)
  File "/opt/lmdeploy/lmdeploy/pytorch/models/patch.py", line 175, in _get_model_class
    raise RuntimeError(f'Can not found rewrite for architectures: {architectures}')
RuntimeError: Can not found rewrite for architectures: ['Qwen2_5_VLForConditionalGeneration']

[grimoire]

Sorry, after double check the document, I found that qwen2.5vl has not been supported yet. We will add the support ASAP.

[zzf2grx]

Sorry, after double check the document, I found that qwen2.5vl has not been supported yet. We will add the support ASAP.

Will you support qwen2.5-vl in turbomind? Thank you!

[carvychen]

Hi, any progress on this feature?

[n-kostadinov]

Qwen2.5_vl_72b on lmdeploy would be life changing

[lvhan028]

@CUHKSZzxy is working on it.

Qwen2.5_vl_72b on lmdeploy would be life changing

[github-actions]

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

[lvhan028]

#3194 implements qwen2.5-vl in pytorch engine

[kingwe-stack]

It should be ok to remove the class inheritance in

lmdeploy/lmdeploy/pytorch/engine/logits_process.py

Line 276 in 637435f

class FusedLogitsProcessor(LogitsWarper):

after remove class inheritance, the model seems wired.

[lvhan028]

@CUHKSZzxy could you help investigate this isssue?

It should be ok to remove the class inheritance in
lmdeploy/lmdeploy/pytorch/engine/logits_process.py
Line 276 in 637435f
class FusedLogitsProcessor(LogitsWarper):

after remove class inheritance, the model seems wired.

[CUHKSZzxy]

@kingwe-stack Hi, could you please provide more details about the command/code you're using and specify which model you're working with? This information will help me reproduce the issue and assist in resolving it.