Meta Models Error with Tooling Chat History

[Luke-Dornburgh]

• Package Name: azure-ai-inference
• Package Version: 1.0.0b7
• Operating System: Windows
• Python Version: 3.11.2

Describe the bug
Using this SDK in a scenario where the client processing the requests can switch from an OpenAI model to a Meta model.

Working Scenario:

1. Create Client using OpenAI model
2. Ask questions without any tools involved
3. Get response
4. Save all messages to history list
5. Create new Client using a Meta model
6. Ask the model to summarize the chat history

Failed Scenario:

1. Create a client using OpenAI model (gpt-4o)
2. Ask a question that invokes a tool suggestion
3. Execute the tool
4. Update the chat history so that it is formatted like this:

[
  {'role': 'user', 'content': 'Next flight from Seattle to Miami'}
  {'role': 'assistant', 'tool_calls': [{'function': {'arguments': '{"origin_city":"Seattle","destination_city":"Miami"}', 'name': 'get_flight_info'}, 'id': 'call_C7lglfMNVEAjFQU98SYtUF0y', 'type': 'function'}]}
  {'role': 'tool', 'content': '{"airline": "Delta", "flight_number": "DL123", "flight_date": "May 7th, 2024", "flight_time": "10:00AM"}', 'tool_call_id': 'call_C7lglfMNVEAjFQU98SYtUF0y'}
]

5. Re-invoke the client with the history from step 4
6. Get the response and save it to the chat history
7. Create a new client using a Meta model
8. Ask that client another question along with the previous history. This question does not invoke a tool or do anything special. Below is the EXACT chat history that is being passed to client.complete(messages=messages):

[
  {'role': 'user', 'content': 'Next flight from Seattle to Miami'}
  {'role': 'assistant', 'tool_calls': [{'function': {'arguments': '{"origin_city":"Seattle","destination_city":"Miami"}', 'name': 'get_flight_info'}, 'id': 'call_C7lglfMNVEAjFQU98SYtUF0y', 'type': 'function'}]}
  {'role': 'tool', 'content': '{"airline": "Delta", "flight_number": "DL123", "flight_date": "May 7th, 2024", "flight_time": "10:00AM"}', 'tool_call_id': 'call_C7lglfMNVEAjFQU98SYtUF0y'}
  {'role': 'assistant', 'content': 'The next flight from Seattle to Miami is operated by Delta, flight number DL123. It is scheduled for May 7th, 2024, at 10:00 AM.'}
  {'role': 'user', 'content': 'Explain inflation in a sentence'}
]

Error output:

ERROR - Error during standard chat call: (Bad Request) {"object":"error","message":"1 validation error for ValidatorIterator\n0.function.arguments\n  Input should be a valid string [type=string_type, input_value={'origin_city': 'Seattle'...tination_city': 'Miami'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/string_type","type":"BadRequestError","param":null,"code":400}
Code: Bad Request
Message: {"object":"error","message":"1 validation error for ValidatorIterator\n0.function.arguments\n  Input should be a valid string [type=string_type, input_value={'origin_city': 'Seattle'...tination_city': 'Miami'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/string_type","type":"BadRequestError","param":null,"code":400}
2025-01-31 18:07:32 - response_generation.call_llm - ERROR - Traceback (most recent call last):
  File "/app/******/call_llm.py", line 163, in call_llm
    response = await client.complete(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/azure/ai/inference/aio/_patch.py", line 670, in complete
    raise HttpResponseError(response=response)
azure.core.exceptions.HttpResponseError: (Bad Request) {"object":"error","message":"1 validation error for ValidatorIterator\n0.function.arguments\n  Input should be a valid string [type=string_type, input_value={'origin_city': 'Seattle'...tination_city': 'Miami'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/string_type","type":"BadRequestError","param":null,"code":400}
Code: Bad Request
Message: {"object":"error","message":"1 validation error for ValidatorIterator\n0.function.arguments\n  Input should be a valid string [type=string_type, input_value={'origin_city': 'Seattle'...tination_city': 'Miami'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/string_type","type":"BadRequestError","param":null,"code":400}

I have checked the typing of the arguments value and it returns as type str. I have also tried to cast it to no avail.

To Reproduce
Steps to reproduce the behavior:

1. Follow the failed scenario step above

Expected behavior
If I create the new client with an OpenAI model this output is a standard AssistantMessage that answers the question asked. No errors

Additional context
In the call to the Meta-based client, I am not binding any tools to it as that is not supported.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dargilco @jhakulin @trangevi.

[dargilco]

Hi @Luke-Dornburgh, thank you for opening this separate GitHub issue and providing all the details. I appreciate it!

I modified our chat completions with tools sample: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-inference/samples/sample_chat_completions_with_tools.py such that the first complete call was done to a gpt-4o model, and the second complete call was done to a Llama-3.3-70B-Instruct model, without changing any code lines related to how the messages variables is constructed leading to the 2nd complete call. That sample worked fine. So, I was not able to reproduce your issue. Let me try some other Meta models.

[dargilco]

When using model Meta-Llama-3.1-70B-Instruct for the second complete call, looks like it did not error out, but gave an answer that indicates it was not able to ingest the tool response.

[Luke-Dornburgh]

@dargilco thanks for looking into this. If you have an output for the message history from the first scenario you tested I would love to view it. Or even perhaps the sample code.

One note is that I am using .as_dict() on each of our messages BEFORE adding them to our chat_history list. Then the list is fed back to the new client as input. Not sure if this is at all a cause. It does cause any issues with re-using GPT based clients as outlined above.

[dargilco]

@Luke-Dornburgh seems like different Lllama models behaver differently, and I believe this is all going to be resolved once tool support is enabled on all of them. Use of .as_dict() should not make a difference. You would be able to specify tool call in chat history, and mix calls to different models when the models support calling. I'll report back once I head from the relevant service team. For the time being, can you use models other than Meta models, that have tools support?