Agent-specific tool call visibility in group chats.

[VeliKivilahti]

Currently the tool call return values are all inserted in the agent group chat and kept during the chat. This can lead to unoptimized token consumption and irrelevant prompt context to the agents.

For example, we can have a agent group chat with multiple RAG agents with different sources / purposes. Currently each agent will add their tool call returns to the chat. The RAG context from the tool call will then be shown as context to the other agents later in the chat, while they only would need the RAG agent's response in the chat. The context from the tool call is in this case irrelevant to other agents and can effect their tasks negatively. For longer context windows and group chats, this will also increase the cost of the group chat as the context is kept in each agent invocation.

If possible, there could be an optional parameter in the plugin / group chat that the output of the plugin is only shown to the agent.
Another option is to allow to delete messages from the group chat mid-chat.

[moonbox3]

Hi @VeliKivilahti, thanks for your question. Are you using C# or Python? Right now, in both languages, for an agent group chat, we aren't broadcasting tool call related messages to other channels. Are you seeing evidence that is indeed happening?

[VeliKivilahti]

C#.
I tested in a case where we have a Nested Aggregator Agent that has a group chat with two ChatCompletionAgents. Agent A has an plugin that has one input parameter and outputs a string. Agent B then only has instructions to repeat all of the tool calls and their outputs in the chat, but has no plugins assigned to itself.
After agent A has called their plugin and made a response to the groupchat, Agent B will then recite the Agent A's tool call result (and input parameter) as is.

This can be seen also from the AgentGroupChat.GetChatMessagesAsync method where the tool call message is in the chat with AuthorName "AgentA" and Role {Tool}, which then is used later in the chat by the Agent B.

[moonbox3]

@crickman tagging you here for visibility/help.

[crickman]

Thank you bring this up @VeliKivilahti.

A couple notes:

• While the AgentGroupChat history represents the primary conversation history, it isn't necessarily the exact history consumed by a given agent. Individual agents manage a specific version of their history according to their type and configuration. For example, an OpenAIAssistantAgent manages its conversation history via a thread_id while ChatCompletionAgent maintains ChatHistory in memory. If you invoke AgentGroupChat.GetChatMessagesAsync with an agent instance, you'll be able to access the agent-specific converstation state. I don't believe this fact fundamentally alters your issue, but I did want to call that out as understanding this may aid in evaluating options for moving forward.
• We specifically preserve function call information across all agents to minimize repeating function calls for the the exact same information. Recently we've recieved customer feedback that this isn't always helpful (very similiar to your case) so we are looking on how to support the ability for developers to opt-in to not retaining this information.
• For ChatCompletionAgent we have introduced a IChatHistoryReducer that you may assign to ChatCompletionAgent.HistoryReducer (currently a truncation and summarization reducer are provided). A ChatCompletionAgent with a reducer will maintain its history differently than an agent without a reducer (see first point). Note: History reduction occurs just prior to invoking the agent.

I'm wondering if providing an implementation of IChatHistoryReducer that removes function-call content from the history might be a way to control / configure the behavior you've described. Based your feedback and that of another customer, I'm inclined to explore this within the Agent Framework; although, you are able to also explore this by implementing your own IChatHistoryReducer (you may be able to more quickly target your case...while I'll need to consider broader usage patterns). If you do work on your own implementation, I'd very much appreciate learning from your work and also recieve feedback on the general solution I'll be pursuing.

#9552

[VeliKivilahti]

Thank you for the information about how the agent-specific history currently works!

The optimal case would be that we could opt-out plugins from the shared history and that way keep some plugins more "contained" to single agents, so I'm glad to hear it is under consideration.

We will look into the IChatHistoryReducer and how it can help us in our scenarios, (We found the recent devblog to be a nice introduction to it! ), but for specific tools to be excluded (We might not want all of the plugins to be excluded!)it probably would be better if was in the agent framework rather than building complex ChatHistoryReducers.