Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ #172 Add Anthropic integration for chat streaming #182

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from

Conversation

curi-adi
Copy link
Collaborator

Pushing the code for Anthropic integration at the following link

#172

@roma-glushko roma-glushko changed the title Add Anthropic integration for chat streaming#172 ✨ #172 Add Anthropic integration for chat streaming Mar 19, 2024
@roma-glushko roma-glushko linked an issue Mar 19, 2024 that may be closed by this pull request

"glide/pkg/api/schemas"
"glide/pkg/providers/clients"
)

type Client struct {
Copy link
Member

@roma-glushko roma-glushko Mar 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Golang uniquely, compared to other programming languages, allows struct/object methods can be separate across go files in the module. For example, in case of Anthropic client is spread across these three files:

Screenshot 2024-03-19 at 22 25 12

Specifically, the client is defined in https://github.com/EinStack/glide/blob/develop/pkg/providers/anthropic/client.go#L22

and it has already pulling a lot of useful values like API key (e.g. c.config.APIKey) and a bunch of other configurations. So these three files are really worth checking and reuse that existing client.

}

func (c *Client) ChatStream(ctx context.Context, chatReq *schemas.ChatRequest) (clients.ChatStream, error) {
apiURL := "https://api.anthropic.com/v1/complete"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this existing chatURL field work here? https://github.com/EinStack/glide/blob/develop/pkg/providers/anthropic/client.go

Seems like it's one and the same completion endpoint for both sync and streaming API in Anthropic just like in OpenAI


func (c *Client) ChatStream(ctx context.Context, chatReq *schemas.ChatRequest) (clients.ChatStream, error) {
apiURL := "https://api.anthropic.com/v1/complete"
requestBody := map[string]interface{}{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have this interesting method to combines configs provided by user in Glide provider YAML config + incoming request data to map them into a real Anthropic request:
https://github.com/EinStack/glide/blob/develop/pkg/providers/anthropic/chat.go#L82-L88

Would be great to leverage it here.

return true
}

func (c *Client) ChatStream(ctx context.Context, chatReq *schemas.ChatRequest) (clients.ChatStream, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Over the weekend, I was finalizing the ChatStream interface and added this Open() method where the request initialization is supposed to happen (here is an example from OpenAI). Would good to move most of this code to AnthropicChatStream.Open() so we can properly track the initial request latency.

responseBody io.ReadCloser
}

func (s *AnthropicChatStream) Receive() (string, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be called Recv() to fulfil the ChatStream interface:

Suggested change
func (s *AnthropicChatStream) Receive() (string, error) {
func (s *AnthropicChatStream) Recv() (string, error) {

It also should return (*schemas.ChatStreamChunk, error) but I think you will get to that:

https://github.com/EinStack/glide/blob/develop/pkg/providers/clients/stream.go#L9

@roma-glushko roma-glushko self-requested a review March 30, 2024 19:59
}

func (c *Client) ChatStream(_ context.Context, _ *schemas.ChatRequest) (clients.ChatStream, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There has to be ChatStream() method that would create an instance of AnthropicChatStream in this case.
For example, in OpenAI case, it looked this way:
https://github.com/EinStack/glide/blob/develop/pkg/providers/openai/chat_stream.go#L162-L177

Without that there is nothing that would use the AnthropicChatStream struct

}

decoder := json.NewDecoder(s.response.Body)
var chunk schemas.ChatStreamChunk
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

schemas.ChatStreamChunk is Glide's unified schema for stream chunk, but it's not going to be useful directly to parse Anthropic chunks most likely. You need to define a Anthropic-specific chunk schema, use it to parse incoming chunks, and then finally remap useful fields to an instance of schemas.ChatStreamChunk.

This is how it's done in OpenAI case:
https://github.com/EinStack/glide/blob/develop/pkg/providers/openai/chat_stream.go#L115-L147

return nil, fmt.Errorf("stream not opened")
}

decoder := json.NewDecoder(s.response.Body)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using just json.Unmarshal() to unmarshal chunks with the default decoder config:
https://github.com/EinStack/glide/blob/develop/pkg/providers/openai/chat_stream.go#L115
so I feel like the same goes in Anthropic case, too:


// Recv listens for and decodes incoming messages from the chat stream into ChatStreamChunk objects.
func (s *AnthropicChatStream) Recv() (*schemas.ChatStreamChunk, error) {
if s.response == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Anthropic uses server-side events (SSE) for chat streaming? If so, you need to use a special parser to read that stream just like OpenAI:

https://github.com/EinStack/glide/blob/develop/pkg/providers/openai/chat_stream.go#L75-L95

SSE has a special format that has to be parsed before you can even unmarshal the real chunk from JSON into an Anthropic chat stream struct.

@roma-glushko roma-glushko self-requested a review March 30, 2024 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

✨ [Lang] Integrate Anthropic with Chat Streaming
2 participants