Chat completion with Qwen2.5-instruct model

[GaryTang32]

Hello.

Problem:

I developed a SelectorGroupChat cluster to do some task with tools call.

When I uses GPT4o-mini, it works and able to accomplish the task with tools call.
but when i switch the model into Qwen2.5-instruct model, the cluster failed to use any tools call.

Solution:

After investigation, the problem is Qwen model need a different argument structure when calling chat completion with oai.chat.completions.create. Qwen model's tool_use argument structure is different from GPT4o-mini.

GPT4o-mini:

response = client.chat.completions.create(
    model='',
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "task1"
        }
    ],
    tools={
        'tools': [
            {
                "type": "function",
                "function": {......}
            }
        ],
    }
)

Qwen2.5-instruct:

response = client.chat.completions.create(
    model='',
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "task1"
        }
    ],
    extra_body={
        "extra_body": {
            'tools': [
                {
                    "type": "function",
                    "function": {......}
                }
            ],
        }
    }
)

Qwen model tools_use argument is passed with "extra_body", while GPT is "tools".

And under the "_openai_client.py" in the AutoGen 0.4Dev8. It hardcoded the oai.chat.completion.create using "tools" argument for tools use passing.

therefore when using Qwen model, it wont raise error or exception, but the model unable to decode the tools under "tools" argument, leading unable to use any tools in the cluster.

To resolve the case, changing the oat.chat.completions.create() argument it works.

Update autogen_ext/models/_openai/_openai_client.py

line 460 from:

future = asyncio.ensure_future(
    self._client.chat.completions.create(
        messages=oai_messages,
        stream=False,
        tools=converted_tools,
        **create_args,
    )
)

to:

future = asyncio.ensure_future(
    self._client.chat.completions.create(
        messages=oai_messages,
        stream=False,
        extra_body={
            'extra_body':{
                'tools': converted_tools
            }
        },
        **create_args,
    )
)

this resolve the tools_use argument passing.

Tested this approach with both GPT4o-mini and Qwen2.5-instruct.
Both models able to use tools call after changes.

Is it able to include this change in the next updates? Or is there any method to adopt Qwen model tools call?

Thanks so much.

Best regards,
Gary Tang

[coldsaber]

Hi Gary Tang,
Thanks a lot for your solution!
I am wondering if you met the same problem as I mentioned in #6036. In which an error as below will appear if I call tools in the code format of Autogen.
"openai.BadRequestError: Error code: 400 - {'error': {'message': '<400> InternalError.Algo.InvalidParameter: messages with role "tool" must be a response to a preceeding message with "tool_calls". (request id:xxxx)', 'type': 'upstream_error', 'param': '400', 'code': 'bad_response_status_code'}}".
I have tried your method. However, after I modified _openai_client.py, the assistant agent pretended to call tools and fabricated outputs, also, there was no tools calling result and token usage was really low.
I really appreciate it If you have other solutions.
Many thanks!

[ekzhu]

@GaryTang32 thanks for bring this up. This might have cleared both issues in #5869 and #6036 about tool calling with Qwen models.

This seems a very minor difference. I don't have access to Qwen cloud hosted models. The local Qwen 2.5 models when deployed using vllm or ollama works just fine using OpenAI client.

I think at this point it looks like something that may change again soon, I am not sure if that's something we want to include in our library yet -- the client will become a bunch of hacks put together. Instead of making the change directly in the OpenAIChatCompletionClient in the AutoGen library, can you patch the openai.AsyncOpenAI.create method in the openai library to override how the tools are being used. You can use monkey patch or some standard ways to directly override any imported library's behavior in runtime.

[vicarmar]

Hello! Not sure if something has changed since this discussion triggered, but I find a different cause for the same symptom of not being able to use tools by some models.

In my experimental setup, I have run Qwen2.5-VL-7B-Instruct (with vLLM docker container), and for this version of the model, I am able to run tools with the lower level client client.chat.completions.create() by passing tools argument as usual (no need to pass them in extra_body argument, although equally possible and working), but we need to pass tool_choice='required' to force the model to use them, otherwise it will just ignore tools usage.

Regardless of the reason for tool_choice argument being needed as 'required' (I have seen this also in other models like Pixtral), when trying now to use Autogen, the issue is that this argument is not respected after creating a new instance of OpenAIChatCompletionClient.

This tool_choice argument is not passed properly, to the inner _openai_client.py BaseOpenAIChatCompletionClient.create() method, where is expected ,from the AssistantAgent._call_llm() method. (And same applies for extra_create_args)

Here is the issue I see: AssistantAgent._call_llm() makes calls to model_client.create() (or create_stream()), passing the tools argument in every call (defined for the AssistantAgent in its creation), but not passing tool_choice argument (defined in the client). On the client.create() side, then the default args are used (tool_choice='auto') therefore not respecting the client config (when BaseOpenAIChatCompletionClient._process_create_args() is called they are updated)

IMO this should be treated as an issue, since it should be either passed as the tools arguments from the AssistantAgent side (where tools/workbench are configured), or respecting the config from the already configured model_client, where we can only set currently tool_choice (this seems more reasonable, as currently tools are assigned to the AssistantAgent, but how to use them is part of the client config and not part of the agent itself, although maybe AssistantAgent should have a tool_choice argument also to override the client ¿?).

In any case, it does not make too much sense to me that the client is configured to set tool_choice='required' and then the agent using the client will make it 'auto' and not respect that initial config at all.

All described also applies to extra_create_args.

We can create a custom OpenAIChatCompletionClient, with custom create methods, that will check that if no tool_choice is passed from the caller to override the client config, it should respect the current client config if any (as kept in the self._create_args).

Although I would say this should be the fix in the library and not a custom implementation.

Hope I am not missing any other detail, and this helps for a future fix.

Thanks a lot for the efforts in any case!!