Handling of thinking output in multi-turn conversations

[FrauH0lle]

Hey @karthink ,

I wanted to get some feedback from you about the way reasoning is handled by gptel using Anthropic's API.

Currently, reasoning content from a previous turn is send back to the model as a normal assistant message and within a turn as a dedicated message of type thinking (given that gptel-include-reasoning is not set to 'ignore). For example:

"role": "assistant",
      "content": [
        {
          "type": "thinking",
          "thinking": "The user wants me to read the first 10 lines of a specific file and provide a brief analysis of what I see. This is a straightforward read task - I know the exact file path, so I can use the Read tool with start_line and end_line parameters.",
          "signature": {}
        },

and on the next turn:

{
      "role": "assistant",
      "content": "The user wants me to read the first 10 lines of a specific file and provide a brief analysis of what I see. This is a straightforward read task - I know the exact file path, so I can use the Read tool with start_line and end_line parameters."
    },

The API documentations state

During tool use, you must pass thinking blocks back to the API, and you must include the complete unmodified block back to the API. This is critical for maintaining the model's reasoning flow and conversation integrity.
https://platform.claude.com/docs/en/build-with-claude/extended-thinking#preserving-thinking-blocks

As far as I understand, this is the case.

But a bit further down it is stated:

While you can omit thinking blocks from prior assistant role turns, we suggest always passing back all thinking blocks to the API for any multi-turn conversation. The API will:

• Automatically filter the provided thinking blocks
• Use the relevant thinking blocks necessary to preserve the model's reasoning
• Only bill for the input tokens for the blocks shown to Claude

and my understanding from this and the examples given in the cookbook is that thinking messages should be send as dedicated "type": "thinking" messages.

So long story short, I gave it a try and introduced some modifications into gptel. At first together with Claude Code (because he should know about its API, right) but then I continued on my own pretty fast.

• I introduced a text property for reasoning content which also stores the corresponding signature
• Use it to highlight the reasoning block accordingly
• Make sure the signature is collected and added to the corresponding reasoning block
• When parsing the buffer for a new turn, the reasoning is send back as thinking type messages along with the signature
• I tested it with Claude as well as the Anthropic API endpoints of GLM and DeepSeek (and of course GLM send the signature in content_block_start and not in content_block_delta like the other two...) and all three seem to work.

Example (using DeepSeek):

      "role": "assistant",
      "content": [
        {
          "type": "thinking",
          "thinking": "The user wants me to read the first 10 lines of a file called `data_structure.md` located at ...",
          "signature": "5eff91fec6454ce6a7698904"
        },
        {
          "type": "tool_use",
          "id": "call_3e6541f635754d4c89247a0d",
       ...
          }
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "tool_result",
         ...
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "thinking",
          "thinking": "The file appears to be a markdown file documenting the data structure for the dashboard's input data. ...",
          "signature": "2fc240b5adad48e394161b1f"
        },
        {
          "type": "text",
          "text": "The file documents ..."
        }
      ]
    },
    {
      "role": "user",
      "content": "Please read the last 5 lines of ...."
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "thinking",
          "thinking": "The user wants me to read the last 5 lines of the same file. ...",
          "signature": "72955b52443a4695b7ccf891"
        },
        {
          "type": "tool_use",
          ...
          }
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "tool_result",
          ...
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "thinking",
          "thinking": "The file has 31 lines total. ...",
          "signature": "53e3476492d8460f843ff9ed"
        },
        {
          "type": "text",
          "text": "The last 5 lines (27-31) show the ...."
        }
      ]
    },
    {
      "role": "user",
      "content": "Thank you"
    }

Before I add now the patch below, I wanted to ask you:

• Is my reasoning correct or did I get it totally wrong? I am sure you know much more about the APIs than I do.
• Is there any interest from your side in a PR?
• Note, that I did not touch anything regarding the OpenAI API code, I am sure there need to be some adjustments as well.
• Should redacted_thinking be handled as well? I never encountered this I think, but for sake of completeness I think this could be included as well.

And here is the patch. Thank you very much for reading this!

diff --git a/gptel-anthropic.el b/gptel-anthropic.el
index 6ef77ba..6ef43b5 100644
--- a/gptel-anthropic.el
+++ b/gptel-anthropic.el
@@ -84,8 +84,11 @@ information if the stream contains it.  Not my best work, I know."
                       (plist-put info :reasoning
                                  (concat (plist-get info :reasoning) thinking))
                     (if-let* ((signature (plist-get delta :signature)))
-                        (plist-put info :signature
-                                   (concat (plist-get info :signature) signature))))))))
+                        (plist-put info :signature signature)
+                      ;; REVIEW: Why concatenate here?
+                        ;; (plist-put info :signature
+                        ;;            (concat (plist-get info :signature) signature))
+                        ))))))
 
            ((looking-at "content_block_start") ;Is the following block text or tool-use?
             (forward-line 1) (forward-char 5)
@@ -97,6 +100,8 @@ information if the stream contains it.  Not my best work, I know."
                                                    :name (plist-get cblock :name))
                                              (plist-get info :tool-use))))
                 ("thinking" (plist-put info :reasoning (plist-get cblock :thinking))
+                 (when-let* ((signature (plist-get cblock :signature)))
+                   (plist-put info :signature signature))
                  (plist-put info :reasoning-block 'in)))))
 
            ((looking-at "content_block_stop")
@@ -388,6 +393,55 @@ TOOL-USE is a list of plists containing tool names, arguments and call results."
           ;; point against the previous.
           (unless (save-excursion (skip-syntax-forward " ") (>= (point) prev-pt))
             (pcase (get-char-property (point) 'gptel)
+              ;; Reasoning block (Anthropic thinking): prepend to following assistant message
+              (`(reasoning . ,signature)
+               (let* ((raw-content (buffer-substring-no-properties (point) prev-pt))
+                      ;; Check if we're merging with an existing thinking block
+                      (merging-p (and prompts
+                                      (equal (plist-get (car prompts) :role) "assistant")
+                                      (let ((old-content (plist-get (car prompts) :content)))
+                                        (and (vectorp old-content)
+                                             (> (length old-content) 0)
+                                             (equal (plist-get (aref old-content 0) :type) "thinking")
+                                             (equal (plist-get (aref old-content 0) :signature)
+                                                    (or signature nil))))))
+                      ;; Only trim if not merging (to preserve spaces at boundaries)
+                      (content (if merging-p
+                                   raw-content
+                                 (gptel--trim-prefixes raw-content))))
+                 (unless (string-blank-p content)
+                   ;; Check if the most recent message is an assistant message
+                   ;; (we iterate backwards, so this is the one that follows
+                   ;; reasoning in buffer order)
+                   (if (and prompts
+                            (equal (plist-get (car prompts) :role) "assistant"))
+                       ;; Check if we can merge with existing thinking block
+                       (let* ((msg (car prompts))
+                              (old-content (plist-get msg :content)))
+                         (if (and (vectorp old-content)
+                                  (> (length old-content) 0)
+                                  (equal (plist-get (aref old-content 0) :type) "thinking")
+                                  (equal (plist-get (aref old-content 0) :signature)
+                                         (or signature nil)))
+                             ;; Merge with existing thinking block (prepend text, preserving spaces)
+                             (plist-put (aref old-content 0) :thinking
+                                        (concat content
+                                                (plist-get (aref old-content 0) :thinking)))
+                           ;; Create new thinking block and prepend
+                           (let ((thinking-block
+                                  `(:type "thinking" :thinking ,content
+                                    :signature ,(or signature nil))))
+                             (plist-put
+                              msg :content
+                              (if (vectorp old-content)
+                                  (vconcat (vector thinking-block) old-content)
+                                (vector thinking-block
+                                        `(:type "text" :text ,old-content)))))))
+                     ;; No assistant message yet - create one with just thinking
+                     (push (list :role "assistant"
+                                 :content (vector `(:type "thinking" :thinking ,content
+                                                    :signature ,(or signature nil))))
+                           prompts)))))
               ('response
                (when-let* ((content
                             (gptel--trim-prefixes
diff --git a/gptel.el b/gptel.el
index 3d0ae9a..e529d1a 100644
--- a/gptel.el
+++ b/gptel.el
@@ -948,24 +948,26 @@ must be set."
 (defun gptel-highlight--margin-prefix (type)
   "Create margin prefix string for TYPE.
 
-Supported TYPEs are response, ignore and tool calls."
+Supported TYPEs are response, ignore, reasoning and tool calls."
   (propertize ">" 'display
               `( (margin left-margin)
                  ,(propertize "▎" 'face
                               (pcase type
                                 ('response 'gptel-response-fringe-highlight)
                                 ('ignore 'shadow)
+                                (`(reasoning . ,_) 'gptel-response-fringe-highlight)
                                 (`(tool . ,_) 'shadow))))))
 
 (defun gptel-highlight--fringe-prefix (type)
   "Create fringe prefix string for TYPE.
 
-Supported TYPEs are response, ignore and tool calls."
+Supported TYPEs are response, ignore, reasoning and tool calls."
   (propertize ">" 'display
               `( left-fringe gptel-highlight-fringe
                  ,(pcase type
                     ('response 'gptel-response-fringe-highlight)
                     ('ignore 'shadow)
+                    (`(reasoning . ,_) 'gptel-response-fringe-highlight)
                     (`(tool . ,_) 'shadow)))))
 
 (defun gptel-highlight--decorate (ov &optional val)
@@ -977,6 +979,7 @@ Supported TYPEs are response, ignore and tool calls."
                  (pcase val
                    ('response 'gptel-response-highlight)
                    ('ignore 'shadow)
+                   (`(reasoning . ,_) 'gptel-response-highlight)
                    (`(tool . ,_) 'shadow))))
   (when-let* ((prefix
                (cond ((memq 'margin gptel-highlight-methods)
@@ -996,7 +999,7 @@ BEG and END delimit the region to refresh."
                               (point) 'gptel nil beg))
                   (/= (point) prev-pt))
         (pcase (get-char-property (point) 'gptel)
-          ((and (or 'response 'ignore `(tool . ,_)) val)
+          ((and (or 'response 'ignore `(reasoning . ,_) `(tool . ,_)) val)
            (if-let* ((ov (or (cdr-safe (get-char-property-and-overlay
                                         (point) 'gptel-highlight))
                              (cdr-safe (get-char-property-and-overlay
@@ -1474,7 +1477,12 @@ Optional RAW disables text properties and transformation."
                      (gptel--insert-response
                       (concat (car blocks) text (cdr blocks)) info t))
                  (gptel--insert-response (concat separator (car blocks)) info t)
-                 (gptel--insert-response text info)
+                 ;; Apply reasoning property with signature
+                 (let ((sig (plist-get info :signature)))
+                   (add-text-properties
+                    0 (length text)
+                    `(gptel (reasoning . ,sig) front-sticky (gptel)) text))
+                 (gptel--insert-response text info t)
                  (gptel--insert-response (cdr blocks) info t))
                (save-excursion
                  (goto-char (plist-get info :tracking-marker))
@@ -1607,6 +1615,51 @@ for streaming responses only."
         (with-current-buffer (marker-buffer start-marker)
           (if (eq text t)               ;end of stream
               (progn
+                ;; Update reasoning property with final signature, if not
+                ;; already present.
+                (when-let* ((sig (plist-get info :signature)))
+                  (save-excursion
+                    ;; We should be at the end of the reasoning block, before
+                    ;; the insertion of #+end_reasoning
+                    (goto-char tracking-marker)
+                    ;; Find reasoning block using text-property-search-backward
+                    (when-let* ((prop-match (text-property-search-backward 'gptel 'reasoning
+                                                                           (lambda (val prop)
+                                                                             (equal (car-safe prop) val)))))
+                      (let ((end (prop-match-end prop-match))
+                            (start nil)
+                            (ended nil))
+                        ;; prop-match-end is reliable, but prop-match-beginning is not
+                        ;; Search backward from end to find actual start
+                        (save-excursion
+                          (goto-char end)
+                          ;; Go backward while we have reasoning property
+                          (while (and (not ended)
+                                      (> (point) start-marker)
+                                      ;; If the preceeding char does not have a
+                                      ;; '(resoning . ...) property, we reached
+                                      ;; the end.
+                                      (let ((prop (get-text-property (1- (point)) 'gptel)))
+                                        (and (consp prop)
+                                             (eq (car prop) 'reasoning))))
+                            (if-let* ((prev (previous-single-property-change (point) 'gptel nil start-marker)))
+                                (goto-char prev)
+                              ;; If the property is constant until the limit
+                              ;; (start-marker), we enter this branch. I think,
+                              ;; this should not happen in the first place and is an error.
+                              (user-error "Warning: reasoning property extends to start-marker")
+                              (goto-char start-marker)
+                              (setq ended t)))
+                          (setq start (point)))
+                        ;; Only update signature if not already set to a non-empty string
+                        (let* ((current-prop (get-text-property start 'gptel))
+                               (current-sig (and (consp current-prop)
+                                                 (eq (car current-prop) 'reasoning)
+                                                 (cdr current-prop))))
+                          (message "current-sig=%s" current-sig)
+                          (message "sig=%s" sig)
+                          (when (or (not current-sig) (string-empty-p current-sig))
+                            (add-text-properties start end `(gptel (reasoning . ,sig) front-sticky (gptel)))))))))
                 (gptel-curl--stream-insert-response
                  (concat (if (derived-mode-p 'org-mode)
                              "\n#+end_reasoning"
@@ -1644,7 +1697,14 @@ for streaming responses only."
                   (add-text-properties
                    0 (length text) '(gptel ignore front-sticky (gptel)) text)
                   (gptel-curl--stream-insert-response text info t))
-              (gptel-curl--stream-insert-response text info)))
+              ;; Apply reasoning property (signature might be updated at end of
+              ;; block)
+              (let ((sig (plist-get info :signature)))
+                (message "sig=%s" sig)
+                (add-text-properties
+                 0 (length text)
+                 `(gptel (reasoning . ,sig) front-sticky (gptel)) text))
+              (gptel-curl--stream-insert-response text info t)))
           (setq tracking-marker (plist-get info :tracking-marker))
           (if reasoning-marker
               (move-marker reasoning-marker tracking-marker)

[FrauH0lle]

from #1391, @bo-tato

we're supposed to send "include": [ "reasoning.encrypted_content" ] with the request, and then take encrypted_content from the output and send it along in future conversation turns, so the model can make use of past reasoning efforts. Without this it works fine, just presumably the model is slightly less inteligent or spends slightly more tokens reasoning again on stuff it already reasoned over.

The difference is in my experience bigger than expected. I asked gpt 5.5 to implement encrypted_content and the phase parameter and I can clearly observe, that the model reasons faster, stays more "on track" and has less of a tendency repeat itself with slight variations.