Standardizing Tool Calling with Chat Models

[eyurtsev]

We're happy to introduce a more standardized interface for using tools:

• ChatModel.bind_tools(): a method for attaching tool definitions to model calls.
• AIMessage.tool_calls: an attribute on the AIMessage returned from the model for easily accessing the tool calls the model decided to make.
• create_tool_calling_agent(): an agent constructor that works with ANY model that implements bind_tools and returns tool_calls.

We'll share some sample code snippets below. You can read more about this
interface on our blog or in the python docs:

• List of chat models that shows status of tool calling capability
• Tool calling explains the new tool calling interface
• Tool calling agent shows how to create an agent that uses the standardized tool calling interface

We'd love to hear feedback from you or any issues that you encounter in this discussion!

Bind Tools:

from langchain_anthropic import ChatAnthropic
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.tools import tool

# ✅ Pydantic class
class multiply(BaseModel):
    """Return product of 'x' and 'y'."""
    x: float = Field(..., description="First factor")
    y: float = Field(..., description="Second factor")
    
# ✅ LangChain tool
@tool
def exponentiate(x: float, y: float) -> float:
    """Raise 'x' to the 'y'."""
    return x**y
    
# ✅ Function

def subtract(x: float, y: float) -> float:
    """Subtract 'x' from 'y'."""
    return y-x
    
# ✅ OpenAI-format dict
# Could also pass in a JSON schema with "title" and "description" 
add = {
  "name": "add",
  "description": "Add 'x' and 'y'.",
  "parameters": {
    "type": "object",
    "properties": {
      "x": {"type": "number", "description": "First number to add"},
      "y": {"type": "number", "description": "Second number to add"}
    },
    "required": ["x", "y"]
  }
}

llm = ChatAnthropic(model="claude-3-sonnet-20240229", temperature=0)

# Whenever we invoke `llm_with_tool`, all three of these tool definitions
# are passed to the model.
llm_with_tools = llm.bind_tools([multiply, exponentiate, add, subtract])

AIMessage.tool_calls:

llm_with_tools.invoke([
	("system", "You're a helpful assistant"), 
	("human", "what's 5 raised to the 2.743"),
])

# 👀 Notice the tool_calls attribute 👀

# -> AIMessage(
# 	  content=..., 
# 	  additional_kwargs={...},
# 	  tool_calls=[{'name': 'exponentiate', 'args': {'y': 2.743, 'x': 5.0}, 'id': '54c166b2-f81a-481a-9289-eea68fc84e4f'}]
# 	  response_metadata={...}, 
# 	  id='...'
#   )

Please let us know if you have any feedback!

[Mcsavvy]

This is great ✨🥵

[axiangcoding]

Great, finally a standardised input and output!

[liugddx]

It's so useful!

[MIRACLE-cowf]

Cool! ty!

[maximeperrindev]

Thanks ! Nice feature to scale apps easier

[roninio]

Can you I ask,
why do I need to set the agent tools in two different functions. in both functions tools is a required.

create_tool_calling_agent
AgentExecutor

[eyurtsev]

create_tool_calling_agent -- creates an agent that can make a single decision (calling a specific tool) or deciding that it's done.

AgentExecutor -- is the agent run time -- it's responsible for calling the agent, invoking tools on its behalf and doing it iteratively until the agent says that it's done. The run time can also control other aspects of agent behavior (e.g., stopping execution if the agent has taken more than some maximum number of steps etc.). We're working on new agent run times using langgraph (https://www.youtube.com/watch?v=9dXp5q3OFdQ)

[javicuriel]

Got the following issue running self.runnable.ainvoke on ChatGroq, it seems it is using the old tools API? (this works just fine with ChatOpenAI model).

llm = ChatGroq(temperature=0, model_name="mixtral-8x7b-32768",
                groq_api_key=env_settings.groq_api_key)

agent = create_tool_calling_agent(llm, tools, prompt)

agent_executor = AgentExecutor(
            agent=agent, tools=tools, max_iterations=5, handle_parsing_errors=True, verbose=True)

  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4523, in ainvoke
    return await self.bound.ainvoke(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4523, in ainvoke
    return await self.bound.ainvoke(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2536, in ainvoke
    input = await step.ainvoke(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4523, in ainvoke
    return await self.bound.ainvoke(
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 212, in ainvoke
    raise e
  File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 203, in ainvoke
    await self._acall(inputs, run_manager=run_manager)
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 1481, in _acall
    next_step_output = await self._atake_next_step(
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 1275, in _atake_next_step
    [
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 1275, in <listcomp>
    [
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 1303, in _aiter_next_step
    output = await self.agent.aplan(
  File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 553, in aplan
    async for chunk in self.runnable.astream(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2900, in astream
    async for chunk in self.atransform(input_aiter(), config, **kwargs):
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2883, in atransform
    async for chunk in self._atransform_stream_with_config(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1984, in _atransform_stream_with_config
    chunk = cast(Output, await py_anext(iterator))
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 2853, in _atransform
    async for output in final_pipeline:
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1316, in atransform
    async for chunk in input:
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 4734, in atransform
    async for item in self.bound.atransform(
  File "/usr/local/lib/python3.10/site-packages/langchain_core/runnables/base.py", line 1333, in atransform
    async for output in self.astream(final, config, **kwargs):
  File "/usr/local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 315, in astream
    raise e
  File "/usr/local/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 293, in astream
    async for chunk in self._astream(
  File "/usr/local/lib/python3.10/site-packages/langchain_groq/chat_models.py", line 361, in _astream
    for rtc in message.additional_kwargs["tool_calls"]
KeyError: 'tool_calls'

[eyurtsev]

Review this table https://python.langchain.com/docs/integrations/chat/ for compatibility

[Rizaldy]

Hi @eyurtsev just curious is it not yet implemented or Groq still not supported tool call?
because on their doc https://console.groq.com/docs/tool-use#models is already can use the tool call

[liugddx]

You need to check whether your dependencies are updated to the latest version.

This is the link of the related PR

#18947

[hujanais]

Thanks for the updated function. I am able to use my custom tools with OpenAI and MistralAI without any issues. However, I also want to implement chat memory. I was able to get it working with OpenAI model but not on Mistral. Do you have any suggestions?
Here is the working code for OpenAI with memory.

class ToolsRagAgent:
    def __init__(self):
        # apiKey = os.environ["GOOGLE_API_KEY"]
        # llm = ChatVertexAI(model="gemini-pro", project=geminiProj, apiKey=apiKey)
        apiKey = os.environ["OPENAI_KEY"]
        llm = ChatOpenAI(openai_api_key=apiKey, model='gpt-3.5-turbo', temperature=0)
        # apiKey = os.environ["MISTRAL_API_KEY"]
        # llm = ChatMistralAI(model="mistral-large-latest", api_key=apiKey)

        llm.bind_tools(tools)

        self.history = []

        prompt = ChatPromptTemplate.from_messages(
            [
                ("system", "you're a helpful assistant"),
                ("placeholder", "{chat_history}"),
                ("human", "{input}"),
                ("placeholder", "{agent_scratchpad}"),
            ]
        )

        agent = create_tool_calling_agent(llm, tools, prompt)
        self.agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
        

    def run(self, query: str):

        resp = self.agent_executor.invoke(
            {"input": query, "chat_history": self.history}
        )

        input = resp['input']
        output = resp['output']

        self.history.append(HumanMessage(content=input))
        self.history.append(AIMessage(content=output))

        print(self.history)

        return resp

when I run with MistralAI, I get the following error.

>> what is the capital of malaysia? 

> Entering new AgentExecutor chain...
The capital of Malaysia is Kuala Lumpur.

> Finished chain.
[HumanMessage(content='what is the capital of malaysia? '), AIMessage(content='The capital of Malaysia is Kuala Lumpur.')]
{'input': 'what is the capital of malaysia? ', 'chat_history': [HumanMessage(content='what is the capital of malaysia? '), AIMessage(content='The capital of Malaysia is Kuala Lumpur.')], 'output': 'The capital of Malaysia is Kuala Lumpur.'}
>> does it snow there? 


> Entering new AgentExecutor chain...
Traceback (most recent call last):
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/app.py", line 39, in <module>
    print(agent.run(user_input))
          ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/agents/tools_rag_agent.py", line 72, in run
    resp = self.agent_executor.invoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 163, in invoke
    raise e
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/chains/base.py", line 153, in invoke
    self._call(inputs, run_manager=run_manager)
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 1432, in _call
    next_step_output = self._take_next_step(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 1138, in _take_next_step
    [
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 1138, in <listcomp>
    [
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 1166, in _iter_next_step
    output = self.agent.plan(
             ^^^^^^^^^^^^^^^^
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain/agents/agent.py", line 514, in plan
    for chunk in self.runnable.stream(inputs, config={"callbacks": callbacks}):
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2875, in stream
    yield from self.transform(iter([input]), config, **kwargs)
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2862, in transform
    yield from self._transform_stream_with_config(
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1880, in _transform_stream_with_config
    chunk: Output = context.run(next, iterator)  # type: ignore
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2826, in _transform
    for output in final_pipeline:
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1283, in transform
    for chunk in input:
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4722, in transform
    yield from self.bound.transform(
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1300, in transform
    yield from self.stream(final, config, **kwargs)
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 249, in stream
    raise e
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 229, in stream
    for chunk in self._stream(messages, stop=stop, **kwargs):
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_mistralai/chat_models.py", line 428, in _stream
    for chunk in self.completion_with_retry(
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/langchain_mistralai/chat_models.py", line 301, in iter_sse
    for event in event_source.iter_sse():
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/httpx_sse/_api.py", line 28, in iter_sse
    self._check_content_type()
  File "/Users/tlee/projects/hackathon/langchain-gemini-hackathon/venv/lib/python3.11/site-packages/httpx_sse/_api.py", line 18, in _check_content_type
    raise SSEError(
httpx_sse._exceptions.SSEError: Expected response header Content-Type to contain 'text/event-stream', got 'application/json'

[eyurtsev]

Could you update the langchain_mistral client? The old client was not propagating the error from the API correctly. There's likely an issue in the code you've pasted, but it's being masked by the wrong exception.

[hujanais]

Thanks for the reply.
I think I am already running the latest version.

Name: langchain-mistralai
Version: 0.1.2
Summary: An integration package connecting Mistral and LangChain
Home-page: https://github.com/langchain-ai/langchain

I tried both import variations and got the same error.

from langchain_mistralai import ChatMistralAI [version 1]
from langchain_mistralai.chat_models import ChatMistralAI [version 2]
 
apiKey = os.environ["MISTRAL_API_KEY"]
llm = ChatMistralAI(model="mistral-large-latest", api_key=apiKey)

[hujanais]

Hello? Is there any further suggestions on using Conversation History with tools? Do you guys have examples for non OpenAI models? Thanks.

[ChristianSch]

Are there any requirements towards the prompt? And does the create_tool_calling_agent/tool_calls have any effects on the prompt of an agent?

[baskaryan]

No strict requirements for the prompt, but of course prompt engineering can improve tool calling performance. The create_too_calling_agent constructor doesn't manipulate the prompt you pass in.

[arose26]

I want to make my agent return structured outputs,
How do I use create_tool_calling_agent together with .with_structured_output()?
If I set llm to model.with_structured_output() and pass llm to create_tool_calling_agent, it gives the error:
ValueError: This function requires a .bind_tools method be implemented on the LLM.

[baskaryan]

What're you hoping to do? You shouldn't need to use with_structured_output when you're creating a tool-calling agent, the agent handles tool calling and parsing already.

[rmalde]

Hi! I'm having the same exact issue! I would like an agent that can do these three things:

1. Read in conversation history
2. Call tools
3. Return a final output as a structured json
   It's a negotiation agent and I want it to output as a json with the final price and quantity that it is negotiating. But I'm running into the same exact error where I can't use with_structured_output along with create_tool_calling_agent

[latteleah]

Love the standardised interface! Just wondering if there is a timeline for other chat models to implement bind_tools? Is this planned for a specific release or dependent on the providers? I'm restricted to a non-implemented provider so just wondering how long I will need to put my Agent dreams on hold :D

[baskaryan]

Which provider are you looking for? There's a lot of providers that don't have a tool/function-calling API yet so we can't implement anything on our end until they do

[latteleah]

Databricks in particular, do they have an API already or are you waiting on that as well just so I can get a sense :)

[baskaryan]

When you say databricks do you mean querying a model via an MLflow deployments server or using one of their foundation model apis or something else

[latteleah]

Using one of their foundation model APIs with ChatDatabricks to allow for tool usage in a lang graph structure! The goal would be to deploy the model using mlflow eventually but the primary concern is the lack of tool calling implementation with ChatDatabricks for their foundation model APIs

[baskaryan]

As far as i can tell none of these support tool calling yet :/ https://docs.databricks.com/en/machine-learning/foundation-models/api-reference.html