Skip to content

Extend StreamingChunk to have tool_call, tool_result, start and index #9358

Open
@sjrl

Description

@sjrl

Is your feature request related to a problem? Please describe.
While working on this PR deepset-ai/haystack-core-integrations#1711 I realized that we don't have a standard way of passing tool_call information into our StreamingChunk dataclass. In our OpenAI integration we pass it as metadata under tool_calls and we directly pass the native OpenAI object (e.g. ChoiceDeltaToolCall). Then until the linked PR we didn't store the tool_call information from Bedrock in the StreamingChunk making it not possible to stream this information.

Then when working on the deepset-ai/haystack-core-integrations#1711 I realized it would have been helpful to have a standard way of filling out information like tool_calls, finish_reason, and usage in our StreamingChunk. Since we lacked this standardization for Bedrock I followed the same structure as used by our OpenAIChatGenerator for generating streaming chunks.

Describe the solution you'd like
My request would be to extend and standardize the StreamingChunk dataclass to include:

  • tool_call: Using a new dataclass called ToolCallDelta
  • tool_call_result: Using the ToolCallResult dataclass
  • start: boolean to indicate if this is the start of a new content block. A content block being regular text, a tool call, or a tool call result
  • index: The content block index. For example, a response from an LLM could contain two tool calls so the first tool call would have index of 0 and the second an index of 1.
  • finish_reason: Could be a Literal or Enum for different finish reasons to choose from. The idea would be to convert the integration's specific finish reason into a standard set. E.g. OpenAI uses tool_calls whereas Bedrock uses tool_use to indicate a finish reason regarding a tool request.

Additional context
Doing this would make it possible to create a single streaming callback function that would work on all of our Chat Generator integrations that follow the StreamingChunk specification.

Metadata

Metadata

Assignees

Labels

P1High priority, add to the next sprint

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions