.Net: New Feature: Add tool call status as a message in GetStreamingChatMessageContentsAsync() StreamingChatMessageContent responses

[kcrandall]

When a tool is called there is no way for the user to know that tool usage is in progress. All the tool usage hidden under the hood and there is no way to know that it is happening. Given the below code:

     var stream = _chatCompletionService.GetStreamingChatMessageContentsAsync(
                        prompt.ChatHistory,
                        kernel: _serviceProvider.GetRequiredService<Kernel>(),
                        executionSettings: executionSettiongs
                    );

await foreach (StreamingChatMessageContent res in stream)
{
       res.ToolCalldata <- return something either here or in metadata
}

There should be a way in the StreamingChatMessageContent messages to get the status of a tool there are log messages that show it. See below there are INFORMATION logs that say a tool is called but it isnt returned as a result as when i am printing out all the responses seriazlied there is nothing you can decode to see that the tool is invoked, finished, failed, etc.

response:{"Content":null,"Role":null,"ChoiceIndex":0,"ModelId":"gpt-4o-mini","Metadata":{"CompletionId":"chatcmpl-B0vwinhyvk4FyxnQlwGpuWxh0kECK","CreatedAt":"2025-02-14T19:47:24+00:00","SystemFingerprint":"fp_00428b782a","RefusalUpdate":null,"Usage":{"OutputTokenCount":39,"InputTokenCount":1041,"TotalTokenCount":1080,"OutputTokenDetails":{"ReasoningTokenCount":0,"AudioTokenCount":0},"InputTokenDetails":{"AudioTokenCount":0,"CachedTokenCount":0}},"FinishReason":null}}
INFORMATION | Function "GoogleSearchPlugin"-"search" invoking. {"PluginName":"GoogleSearchPlugin","FunctionName":"search","EventId":"Name: LogFunctionInvoking","SourceContext":"Microsoft.SemanticKernel.KernelFunction","ActionId":"040c98ce-1969-4a73-b56f-42f140909536","ActionName":"APIApp.Controllers.TextAssistant.GetTextAssistantJson (APIApp)","RequestId":"0HNAD61IK9L0V:00000001","RequestPath":"/TextAssistant/json","ConnectionId":"0HNAD61IK9L0V"}
INFORMATION | Function "GoogleSearchPlugin"-"search" succeeded. {"PluginName":"GoogleSearchPlugin","FunctionName":"search","EventId":"Name: LogFunctionInvokedSuccess","SourceContext":"Microsoft.SemanticKernel.KernelFunction","ActionId":"040c98ce-1969-4a73-b56f-42f140909536","ActionName":"APIApp.Controllers.TextAssistant.GetTextAssistantJson (APIApp)","RequestId":"0HNAD61IK9L0V:00000001","RequestPath":"/TextAssistant/json","ConnectionId":"0HNAD61IK9L0V"}
INFORMATION | Function "GoogleSearchPlugin"-"search" completed. Duration: 0.6508721s {"PluginName":"GoogleSearchPlugin","FunctionName":"search","Duration":"0.6508721","EventId":"Name: LogFunctionComplete","SourceContext":"Microsoft.SemanticKernel.KernelFunction","ActionId":"040c98ce-1969-4a73-b56f-42f140909536","ActionName":"APIApp.Controllers.TextAssistant.GetTextAssistantJson (APIApp)","RequestId":"0HNAD61IK9L0V:00000001","RequestPath":"/TextAssistant/json","ConnectionId":"0HNAD61IK9L0V"}
Metadata:{"CompletionId":"chatcmpl-B0vwkZsqeEzccCinlNC6thAHoIbJg","CreatedAt":"2025-02-14T19:47:26+00:00","SystemFingerprint":"fp_00428b782a","RefusalUpdate":null,"Usage":null,"FinishReason":null}
response:{"Content":"","Role":{"Label":"Assistant"},"ChoiceIndex":0,"ModelId":"gpt-4o-mini","Metadata":{"CompletionId":"chatcmpl-B0vwkZsqeEzccCinlNC6thAHoIbJg","CreatedAt":"2025-02-14T19:47:26+00:00","SystemFingerprint":"fp_00428b782a","RefusalUpdate":null,"Usage":null,"FinishReason":null}}
Metadata:{"CompletionId":"chatcmpl-B0vwkZsqeEzccCinlNC6thAHoIbJg","CreatedAt":"2025-02-14T19:47:26+00:00","SystemFingerprint":"fp_00428b782a","RefusalUpdate":null,"Usage":null,"FinishReason":null}

I propose adding a an additional property on the StreamingChatMessageContent or putting it in the Metadata object that a tool has been called. Send a message when it is invoked, as well as when it succeeds or fails. Include the tool name, parameters, etc. As well as a message with the response from the tool. If multiple tools are called there should either be multiple messages returned or a list of all tools. I think multiple messages would be better as a UI could update each tool when they complete. For example there might be a widget that turns red or green and when tool 1 updates it would turn green and the other one would stay red until it completes.

[markwallace-microsoft]

@kcrandall have you considered using auto invocation filters to achieve what you need?

• https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/filters?pivots=programming-language-csharp#auto-function-invocation-filter
• https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Filtering/AutoFunctionInvocationFiltering.cs