Use Groq mixtral-8x7b-32768 in functions/tools OpenAI like agent

[A-Giordano]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.utilities import BingSearchAPIWrapper
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_community.tools import BingSearchRun

api_wrapper = BingSearchAPIWrapper(k=1)
tool = BingSearchRun(api_wrapper=api_wrapper)
tools = [tool]

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are very powerful assistant, but don't know current events",
        ),
        ("user", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

llm = ChatGroq(temperature=0, model_name="mixtral-8x7b-32768", streaming=False, verbose=False)
#llm = ChatOpenAI(model="mixtral-8x7b-32768", temperature=0, openai_api_base="https://api.groq.com/openai/v1")

agent = create_openai_tools_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, streaming=False, verbose=False)

agent_executor.invoke({"input": "what is LangChain?"})


---------------------------------------------------------------------------
BadRequestError                           Traceback (most recent call last)
Cell In[13], line 1
----> 1 agent_executor.invoke({"input": "what is LangChain?"})

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/chains/base.py:163, in Chain.invoke(self, input, config, **kwargs)
    161 except BaseException as e:
    162     run_manager.on_chain_error(e)
--> 163     raise e
    164 run_manager.on_chain_end(outputs)
    166 if include_run_info:

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/chains/base.py:153, in Chain.invoke(self, input, config, **kwargs)
    150 try:
    151     self._validate_inputs(inputs)
    152     outputs = (
--> 153         self._call(inputs, run_manager=run_manager)
    154         if new_arg_supported
    155         else self._call(inputs)
    156     )
    158     final_outputs: Dict[str, Any] = self.prep_outputs(
    159         inputs, outputs, return_only_outputs
    160     )
    161 except BaseException as e:

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/agents/agent.py:1432, in AgentExecutor._call(self, inputs, run_manager)
   1430 # We now enter the agent loop (until it returns something).
   1431 while self._should_continue(iterations, time_elapsed):
-> 1432     next_step_output = self._take_next_step(
   1433         name_to_tool_map,
   1434         color_mapping,
   1435         inputs,
   1436         intermediate_steps,
   1437         run_manager=run_manager,
   1438     )
   1439     if isinstance(next_step_output, AgentFinish):
   1440         return self._return(
   1441             next_step_output, intermediate_steps, run_manager=run_manager
   1442         )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/agents/agent.py:1138, in AgentExecutor._take_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps, run_manager)
   1129 def _take_next_step(
   1130     self,
   1131     name_to_tool_map: Dict[str, BaseTool],
   (...)
   1135     run_manager: Optional[CallbackManagerForChainRun] = None,
   1136 ) -> Union[AgentFinish, List[Tuple[AgentAction, str]]]:
   1137     return self._consume_next_step(
-> 1138         [
   1139             a
   1140             for a in self._iter_next_step(
   1141                 name_to_tool_map,
   1142                 color_mapping,
   1143                 inputs,
   1144                 intermediate_steps,
   1145                 run_manager,
   1146             )
   1147         ]
   1148     )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/agents/agent.py:1138, in <listcomp>(.0)
   1129 def _take_next_step(
   1130     self,
   1131     name_to_tool_map: Dict[str, BaseTool],
   (...)
   1135     run_manager: Optional[CallbackManagerForChainRun] = None,
   1136 ) -> Union[AgentFinish, List[Tuple[AgentAction, str]]]:
   1137     return self._consume_next_step(
-> 1138         [
   1139             a
   1140             for a in self._iter_next_step(
   1141                 name_to_tool_map,
   1142                 color_mapping,
   1143                 inputs,
   1144                 intermediate_steps,
   1145                 run_manager,
   1146             )
   1147         ]
   1148     )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/agents/agent.py:1166, in AgentExecutor._iter_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps, run_manager)
   1163     intermediate_steps = self._prepare_intermediate_steps(intermediate_steps)
   1165     # Call the LLM to see what to do.
-> 1166     output = self.agent.plan(
   1167         intermediate_steps,
   1168         callbacks=run_manager.get_child() if run_manager else None,
   1169         **inputs,
   1170     )
   1171 except OutputParserException as e:
   1172     if isinstance(self.handle_parsing_errors, bool):

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain/agents/agent.py:514, in RunnableMultiActionAgent.plan(self, intermediate_steps, callbacks, **kwargs)
    506 final_output: Any = None
    507 if self.stream_runnable:
    508     # Use streaming to make sure that the underlying LLM is invoked in a
    509     # streaming
   (...)
    512     # Because the response from the plan is not a generator, we need to
    513     # accumulate the output into final output and return that.
--> 514     for chunk in self.runnable.stream(inputs, config={"callbacks": callbacks}):
    515         if final_output is None:
    516             final_output = chunk

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:2875, in RunnableSequence.stream(self, input, config, **kwargs)
   2869 def stream(
   2870     self,
   2871     input: Input,
   2872     config: Optional[RunnableConfig] = None,
   2873     **kwargs: Optional[Any],
   2874 ) -> Iterator[Output]:
-> 2875     yield from self.transform(iter([input]), config, **kwargs)

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:2862, in RunnableSequence.transform(self, input, config, **kwargs)
   2856 def transform(
   2857     self,
   2858     input: Iterator[Input],
   2859     config: Optional[RunnableConfig] = None,
   2860     **kwargs: Optional[Any],
   2861 ) -> Iterator[Output]:
-> 2862     yield from self._transform_stream_with_config(
   2863         input,
   2864         self._transform,
   2865         patch_config(config, run_name=(config or {}).get("run_name") or self.name),
   2866         **kwargs,
   2867     )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:1880, in Runnable._transform_stream_with_config(self, input, transformer, config, run_type, **kwargs)
   1878 try:
   1879     while True:
-> 1880         chunk: Output = context.run(next, iterator)  # type: ignore
   1881         yield chunk
   1882         if final_output_supported:

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:2826, in RunnableSequence._transform(self, input, run_manager, config)
   2817 for step in steps:
   2818     final_pipeline = step.transform(
   2819         final_pipeline,
   2820         patch_config(
   (...)
   2823         ),
   2824     )
-> 2826 for output in final_pipeline:
   2827     yield output

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:1283, in Runnable.transform(self, input, config, **kwargs)
   1280 final: Input
   1281 got_first_val = False
-> 1283 for chunk in input:
   1284     if not got_first_val:
   1285         final = adapt_first_streaming_chunk(chunk)  # type: ignore

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:4722, in RunnableBindingBase.transform(self, input, config, **kwargs)
   4716 def transform(
   4717     self,
   4718     input: Iterator[Input],
   4719     config: Optional[RunnableConfig] = None,
   4720     **kwargs: Any,
   4721 ) -> Iterator[Output]:
-> 4722     yield from self.bound.transform(
   4723         input,
   4724         self._merge_configs(config),
   4725         **{**self.kwargs, **kwargs},
   4726     )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/runnables/base.py:1300, in Runnable.transform(self, input, config, **kwargs)
   1293             raise TypeError(
   1294                 f"Failed while trying to add together "
   1295                 f"type {type(final)} and {type(chunk)}."
   1296                 f"These types should be addable for transform to work."
   1297             )
   1299 if got_first_val:
-> 1300     yield from self.stream(final, config, **kwargs)

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py:243, in BaseChatModel.stream(self, input, config, stop, **kwargs)
    236 except BaseException as e:
    237     run_manager.on_llm_error(
    238         e,
    239         response=LLMResult(
    240             generations=[[generation]] if generation else []
    241         ),
    242     )
--> 243     raise e
    244 else:
    245     run_manager.on_llm_end(LLMResult(generations=[[generation]]))

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py:223, in BaseChatModel.stream(self, input, config, stop, **kwargs)
    221 generation: Optional[ChatGenerationChunk] = None
    222 try:
--> 223     for chunk in self._stream(messages, stop=stop, **kwargs):
    224         run_manager.on_llm_new_token(
    225             cast(str, chunk.message.content), chunk=chunk
    226         )
    227         if chunk.message.id is None:

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/langchain_groq/chat_models.py:258, in ChatGroq._stream(self, messages, stop, run_manager, **kwargs)
    255 params = {**params, **kwargs, "stream": True}
    257 default_chunk_class = AIMessageChunk
--> 258 for chunk in self.client.create(messages=message_dicts, **params):
    259     if not isinstance(chunk, dict):
    260         chunk = chunk.dict()

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/groq/resources/chat/completions.py:175, in Completions.create(self, messages, model, frequency_penalty, logit_bias, logprobs, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
    132 def create(
    133     self,
    134     *,
   (...)
    158     timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
    159 ) -> ChatCompletion | Stream[ChatCompletionChunk]:
    160     """
    161     Creates a completion for a chat prompt
    162 
   (...)
    173       timeout: Override the client-level default timeout for this request, in seconds
    174     """
--> 175     return self._post(
    176         "/openai/v1/chat/completions",
    177         body=maybe_transform(
    178             {
    179                 "messages": messages,
    180                 "model": model,
    181                 "frequency_penalty": frequency_penalty,
    182                 "logit_bias": logit_bias,
    183                 "logprobs": logprobs,
    184                 "max_tokens": max_tokens,
    185                 "n": n,
    186                 "presence_penalty": presence_penalty,
    187                 "response_format": response_format,
    188                 "seed": seed,
    189                 "stop": stop,
    190                 "stream": stream,
    191                 "temperature": temperature,
    192                 "tool_choice": tool_choice,
    193                 "tools": tools,
    194                 "top_logprobs": top_logprobs,
    195                 "top_p": top_p,
    196                 "user": user,
    197             },
    198             completion_create_params.CompletionCreateParams,
    199         ),
    200         options=make_request_options(
    201             extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
    202         ),
    203         cast_to=ChatCompletion,
    204         stream=stream or False,
    205         stream_cls=Stream[ChatCompletionChunk],
    206     )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/groq/_base_client.py:1189, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
   1175 def post(
   1176     self,
   1177     path: str,
   (...)
   1184     stream_cls: type[_StreamT] | None = None,
   1185 ) -> ResponseT | _StreamT:
   1186     opts = FinalRequestOptions.construct(
   1187         method="post", url=path, json_data=body, files=to_httpx_files(files), **options
   1188     )
-> 1189     return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/groq/_base_client.py:891, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
    882 def request(
    883     self,
    884     cast_to: Type[ResponseT],
   (...)
    889     stream_cls: type[_StreamT] | None = None,
    890 ) -> ResponseT | _StreamT:
--> 891     return self._request(
    892         cast_to=cast_to,
    893         options=options,
    894         stream=stream,
    895         stream_cls=stream_cls,
    896         remaining_retries=remaining_retries,
    897     )

File ~/anaconda3/envs/pdf/lib/python3.11/site-packages/groq/_base_client.py:982, in SyncAPIClient._request(self, cast_to, options, remaining_retries, stream, stream_cls)
    979         err.response.read()
    981     log.debug("Re-raising status error")
--> 982     raise self._make_status_error_from_response(err.response) from None
    984 return self._process_response(
    985     cast_to=cast_to,
    986     options=options,
   (...)
    989     stream_cls=stream_cls,
    990 )

BadRequestError: Error code: 400 - {'error': {'message': 'Tool and function calls do not currently support streaming', 'type': 'invalid_request_error'}}

Description

Hello everybody,
I'm new here so forgive me if I made any mistake in the post.
I would like to try the mixtral-8x7b-32768 model provided by Groq in a function/tool agent, since it seems to support functions calling.
I tried 2 different way of implementing it:

• using the provided ChatGroq class

• using ChatOpenAI class setting openai_api_base="https://api.groq.com/openai/v1"
  but both gave me the same error:

• BadRequestError: Error code: 400 - {'error': {'message': 'Tool and function calls do not currently support streaming', 'type': 'invalid_request_error'}}

Now, is there a way to use the function/tool agent in a complete non streaming mode?
I tried to set streaming=False everywhere i could but it doesn't seem to fix it.
If anyone has any suggestion it would be much appreciated.
Thanks

System Info

langchain==0.1.14
langchain-community==0.0.31
langchain-core==0.1.38
langchain-groq==0.0.1
langchain-openai==0.1.1
langchain-text-splitters==0.0.1
openai==1.14.3
groq==0.4.2

[dosubot[bot]]

Hey @A-Giordano, great to see you tackling new challenges! Hope you've been well since our last interaction.

Thank you for providing the detailed context and the code snippet. It seems like you've done a thorough job of setting streaming=False in all the relevant places. However, the error message you're seeing suggests that there might be a misunderstanding about how the streaming parameter is used in the LangChain framework, particularly with the Groq model.

In the LangChain framework, the streaming parameter, when used with the Groq model, enables the functionality to receive and process tokens as they are generated in real-time. This is evident from the tests test_invoke_streaming and test_agenerate_streaming, where setting streaming=True in the ChatGroq constructor results in the invocation of the on_llm_new_token callback for each new token generated by the model. This callback mechanism allows for real-time processing of each token, which is useful for applications that require immediate response or incremental output display.

However, the error message you're seeing suggests that the Groq model you're using, mixtral-8x7b-32768, does not support streaming. This could be due to a limitation of the model itself or the way it's being used in the LangChain framework.

Here are a few suggestions to troubleshoot this issue:

1. Check the Model's Streaming Support: Ensure that the mixtral-8x7b-32768 model actually supports streaming. If it doesn't, you'll need to use it in a non-streaming mode.

2. Check the Streaming Flag: The code checks for a stream keyword argument in several places to determine whether to use streaming. You mentioned setting streaming=False everywhere possible, but the relevant checks in the provided code are for a stream keyword, not streaming. Ensure that you are setting stream=False as a keyword argument in your model invocation to explicitly disable streaming.

3. Check the Model Invocation Parameters: The _get_invocation_params method constructs the parameters for model invocation, including the stop parameter and any additional keyword arguments. Ensure that the parameters passed to the model are compatible with the mixtral-8x7b-32768 model and do not inadvertently enable streaming or conflict with the model's requirements.

If after reviewing and adjusting these areas based on the specific capabilities and requirements of the mixtral-8x7b-32768 model and the LangChain framework you still encounter the BadRequestError, it may be necessary to consult the documentation or support resources for the Groq model and the LangChain framework for model-specific configuration or limitations.

I hope this helps! If you have any further questions or if there's anything else I can assist you with, please let me know.

Sources

• libs/core/langchain_core/language_models/chat_models.py
• libs/partners/groq/langchain_groq/chat_models.py
• libs/partners/groq/tests/integration_tests/test_chat_models.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[leslieetubo]

Hi @A-Giordano, while groq api supports tool calling using invoke, it does not support tool messages for agents. Hang in there. Here is the documentation of models that support tool messages for agents https://python.langchain.com/docs/integrations/chat/.

If you must use groq for its speed, you can read the response from invoke and get function name and params before calling the functions and then passing the response back to the llm for interpretation. I know its pain, but that's how I did it.

[A-Giordano]

Hi and thanks for the reply, I get they might not have fully implemented the ChatGroq class yet, what i don't understand is how tools not works in agent even using the model as:
llm = ChatOpenAI(model="mixtral-8x7b-32768", temperature=0, openai_api_base="https://api.groq.com/openai/v1", openai_api_key='groq_api_key', streaming=False)
This work smoothly used in chains, but not as a tools agent.
To my understanding, since Groq APIs are compatible with OpenAi ones, there shouldn't be any troubles on Langchain side.
Am I missing something? or maybe the 2 APIs are not fully compatible?

[arose26]

According to the tool use docs https://console.groq.com/docs/tool-use#models

These following models powered by Groq all support tool use:

llama3-70b
llama3-8b
llama2-70b
mixtral-8x7b
gemma-7b-it

I am attempting tool use with llama3-70b (streaming=False), and still get your above error of Tool and function calls do not currently support streaming. In my case I am using it in an AgentExecutor chain and am not sure how to get it to stop streaming.

[arose26]

Also on the chat docs ChatGroq is set to yellow, meaning partial support for tools but not tools + agents.

[sepiatone]

Please see https://console.groq.com/docs/openai. The Groq API doesn't appear to support tool calling.

Edit: Groq released tool calling support on 2024-04-09

[mingxuan-he]

They just released tool calling support today: https://console.groq.com/docs/tool-use
Although based on my own tests today, mixtral without specific fine-tunes isn't very good at tool calling. Not sure about Llama70b/Gemma.

[NeevrajKB]

Well how do we make a tools agent with groq in langchain now that the tools support is introduced? Because the groq tools usage is simpler, but some use cases require langchain...