Skip to content

Wrap Claude 3.7 thinking tokens in <thinking> tags #11486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 19, 2025
Merged

Wrap Claude 3.7 thinking tokens in <thinking> tags #11486

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c6aedfb
add thinking tags at the beginning and end of thinking blocks
aubin-tchoi Mar 19, 2025
68413d2
fmt
aubin-tchoi Mar 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion core/src/providers/anthropic.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1127,6 +1127,14 @@ impl AnthropicLLM {
StreamContent::AnthropicStreamThinking(
thinking,
) => {
// Send <thinking> tag at the start of a thinking block
let _ = event_sender.send(json!({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't you sending one at each event?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at each new thinking block only it would seem (side note: trying out an AI agent tool here, launched the agent, tested, opened the PR)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if i get multiple thinking blocks I separate them nicely in <thinking> elements I think

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aren't we getting such events per token? or token group?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here it's a group of token with content_block_start as the event_type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The question is can we get multiple in a row? If not which is your argument here we should just wrap here instead of the extra logic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can't, we're supposed to get a block start then deltas then a block stop

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I hadn't seen the name of the event up top. 👍

"type": "tokens",
"content": {
"text": "<thinking>",
},
}));
// Then send the actual thinking content
let _ = event_sender.send(json!({
"type": "tokens",
"content": {
Expand Down Expand Up @@ -1233,7 +1241,7 @@ impl AnthropicLLM {
}
}
"content_block_stop" => {
let _: StreamContentBlockStop =
let stop_event: StreamContentBlockStop =
match serde_json::from_str(event.data.as_str()) {
Ok(event) => event,
Err(error) => {
Expand All @@ -1245,6 +1253,28 @@ impl AnthropicLLM {
break 'stream;
}
};

// Check if the stopping block is a thinking block
match final_response.as_ref() {
Some(response) => {
if let Some(content) =
response.content.get(stop_event.index as usize)
{
if let StreamContent::AnthropicStreamThinking(_) =
content
{
// Send </thinking> tag at the end of a thinking block
let _ = event_sender.send(json!({
"type": "tokens",
"content": {
"text": "</thinking>",
},
}));
}
}
}
None => {}
}
}
"message_delta" => {
let event: StreamMessageDelta =
Expand Down
Loading