Description
Is your feature request related to a problem? Please describe.
While working on this PR deepset-ai/haystack-core-integrations#1711 I realized that we don't have a standard way of passing tool_call
information into our StreamingChunk
dataclass. In our OpenAI integration we pass it as metadata under tool_calls
and we directly pass the native OpenAI object (e.g. ChoiceDeltaToolCall
). Then until the linked PR we didn't store the tool_call
information from Bedrock in the StreamingChunk making it not possible to stream this information.
Then when working on the deepset-ai/haystack-core-integrations#1711 I realized it would have been helpful to have a standard way of filling out information like tool_calls
, finish_reason
, and usage
in our StreamingChunk
. Since we lacked this standardization for Bedrock I followed the same structure as used by our OpenAIChatGenerator for generating streaming chunks.
Describe the solution you'd like
My request would be to extend and standardize the StreamingChunk dataclass to include:
tool_call
: Using a new dataclass calledToolCallDelta
tool_call_result
: Using theToolCallResult
dataclassstart
: boolean to indicate if this is the start of a new content block. A content block being regular text, a tool call, or a tool call resultindex
: The content block index. For example, a response from an LLM could contain two tool calls so the first tool call would haveindex
of 0 and the second anindex
of 1.finish_reason
: Could be a Literal or Enum for different finish reasons to choose from. The idea would be to convert the integration's specific finish reason into a standard set. E.g. OpenAI usestool_calls
whereas Bedrock usestool_use
to indicate a finish reason regarding a tool request.
Additional context
Doing this would make it possible to create a single streaming callback function that would work on all of our Chat Generator integrations that follow the StreamingChunk specification.