Interacting with privateGPT (specifically for RAG)

[Aquan1412]

Hello, I'm looking for a way to use privateGPT as a backend for gptel, in order to use its simple RAG pipeline.
Specifically, I'm looking for a way to provide additional keywords to the backend (such as "use_context" and "include_sources").

I can generally use privateGPT as an "openai"-like backend with the following configuration:

(gptel-make-openai "privateGPT"
  :protocol "http"
  :host "localhost:8001"
  :models '("private-gpt"))

I tried simply adding the keyword to the configuration:

(gptel-make-openai "privateGPT"
  :protocol "http"
  :host "localhost:8001"
  :use_context t
  :models '("private-gpt"))

However, if I do that, I get the following error message:

Debugger entered--Lisp error: (error "Keyword argument :use_context not one of (:curl-ar...")
signal(error ("Keyword argument :use_context not one of (:curl-ar..."))
error("Keyword argument %s not one of (:curl-args :models..." :use_context)
gptel-make-openai("privateGPT-Context" :protocol "http" :host "localhost:8001" :use_context t :models ("private-gpt"))
(progn (gptel-make-openai "privateGPT-Context" :protocol "http" :host "localhost:8001" :use_context t :models '("private-gpt")))
eval((progn (gptel-make-openai "privateGPT-Context" :protocol "http" :host "localhost:8001" :use_context t :models '("private-gpt"))) t)
elisp--eval-last-sexp(nil)
eval-last-sexp(nil)
funcall-interactively(eval-last-sexp nil)
call-interactively(eval-last-sexp nil nil)
command-execute(eval-last-sexp)

So, is there a way to get this to work? Or am I approaching it completely wrong?

[karthink]

You're going to have to provide more context for me to understand what you're looking for. What do you expect use_context and include_sources to do?

[Aquan1412]

I want to use privateGPT to generate responses based on embeddings I previously generated by "ingesting" several PDFs (in privateGPT lingo). According to the API reference, if I set use_context = true, it should use the embeddings to generate responses based on the ingested papers. Additionally, if I set include_sources = true, it should include the source chunks, on which the generated answers are based.

[kenbolton]

I am an elisp novice reading open issues to learn.

gptel/gptel-openai.el

Lines 102 to 107 in 8ccdc31

	(let ((prompts-plist
	`(:model ,gptel-model
	:messages [,@prompts]
	:stream ,(or (and gptel-stream gptel-use-curl
	(gptel-backend-stream gptel-backend))
	:json-false))))

I believe inserting:

  :use_context t
  :include_sources t

after line 103 will do what you want but is likely to break other things. This change will add those key/value pairs to the data sent to the OpenAI API through curl.

[karthink]

I believe inserting: ``` elisp :use_context t :include_sources t ``` after line 103 will do what you want but is likely to break other things. This change will add those key/value pairs to the data sent to the OpenAI API through curl.

This is correct. I'll need to find some way of adding this via the configuration.

[Aquan1412]

I am an elisp novice reading open issues to learn.

gptel/gptel-openai.el

Lines 102 to 107 in 8ccdc31

	(let ((prompts-plist
	`(:model ,gptel-model
	:messages [,@prompts]
	:stream ,(or (and gptel-stream gptel-use-curl
	(gptel-backend-stream gptel-backend))
	:json-false))))

I believe inserting:

  :use_context t
  :include_sources t

after line 103 will do what you want but is likely to break other things. This change will add those key/value pairs to the data sent to the OpenAI API through curl.

I just tried your proposed change, and it worked, at least for respecting the given context! Thanks!
However, now I also noticed that in order to include the used sources, I also need to adjust the parsing of the response.

Maybe I'll try and create a separate gptel-make-privateGPT based on gptel-make-openai. Now that I roughly know where to look, i shouldn't be too hard.

[karthink]

Maybe I'll try and create a separate gptel-make-privateGPT based on gptel-make-openai. Now that I roughly know where to look, i shouldn't be too hard.

You'll need to write a new struct type gptel-privategpt that inherits from gptel-openai, and three cl-defmethods that specialize on the backend-type gptel-privategpt: gptel--request-data, gptel-curl--parse-stream and gptel--parse-response. We can add it to gptel afterwards. I can help if you have questions.

[Aquan1412]

Maybe I'll try and create a separate gptel-make-privateGPT based on gptel-make-openai. Now that I roughly know where to look, i shouldn't be too hard.

You'll need to write a new struct type gptel-privategpt that inherits from gptel-openai, and three cl-defmethods that specialize on the backend-type gptel-privategpt: gptel--request-data, gptel-curl--parse-stream and gptel--parse-response. We can add it to gptel afterwards. I can help if you have questions.

Great, thanks for the information! I'll see how far I get, and if I encounter any big problems, I'll come back to you.

[Aquan1412]

So, after a bit of trial and error, I managed to create a first working version of gptel-privategpt.

Here are the different definitions:

1. The gptel-privategpt struct:

(cl-defstruct (gptel-privategpt (:constructor gptel--make-privategpt)
                               (:copier nil)
                               (:include gptel-openai))
  use_context include_sources
  ) 

2. The three methods for requesting and parsing of the response:

(cl-defmethod gptel-curl--parse-stream ((_backend gptel-privategpt) _info)
  (let* ((content-strs))
    (condition-case nil
        (while (re-search-forward "^data:" nil t)
          (save-match-data
            (unless (looking-at " *\\[DONE\\]")
              (let* ((response (gptel--json-read))
		     (finish-reason (map-nested-elt
				       response '(:choices 0 :finish_reason))))
		(if finish-reason
		    ;; finish_reason "stop": stream has ended, therefore put sources at the bottom of the printed text
		    (progn
		      (setq-local counter 0)
		      (setq-local source-string-list (list))
		      (while-let ((names (map-nested-elt persistent-sources (list :sources counter :document :doc_metadata :file_name)))
				  (pages (map-nested-elt persistent-sources (list :sources counter :document :doc_metadata :page_label)))
				  )
			(cl-pushnew (format "- %s (page %s)" names pages) source-string-list :test #'string=)
			(setq counter (+ 1 counter)))
		      (push (format "\n\nSources:\n%s" (mapconcat (lambda (s) s) (nreverse source-string-list) "\n")) content-strs))
		  ;; finish_reason "nil": stream is still ongoing, therefore extract current content
		  (let* ((delta (map-nested-elt
				 response '(:choices 0 :delta)))
			 (content (plist-get delta :content))
			 (sources (map-nested-elt response '(:choices 0)))
			 )
		    (progn
		      ;; sources are only returned as long as finish_reason is "nil", therefore they have to be buffered so that they can be printed once the stream has ended
		      (setq-local persistent-sources sources)
		      (push content content-strs)))))
	      ))
	  )
    (error
     (goto-char (match-beginning 0))))
  (apply #'concat (nreverse content-strs))
  ))

(cl-defmethod gptel--parse-response ((_backend gptel-privategpt) response _info)
  (let ((response-string (map-nested-elt response '(:choices 0 :message :content)))
	(sources (map-nested-elt response '(:choices 0)))
	(counter 0)
	(source-string-list (list))
	)
    (while-let ((names (map-nested-elt sources (list :sources counter :document :doc_metadata :file_name)))
		(pages (map-nested-elt sources (list :sources counter :document :doc_metadata :page_label)))
		)
      (cl-pushnew (format "- %s (page %s)" names pages) source-string-list :test #'string=)
      (setq counter (+ 1 counter)))
    (format "%s\n\nSources:\n%s" response-string (mapconcat (lambda (s) s) (nreverse source-string-list) "\n")))
)

(cl-defmethod gptel--request-data ((_backend gptel-privategpt) prompts)
  "JSON encode PROMPTS for sending to ChatGPT."
  (let ((prompts-plist
         `(:model ,gptel-model
	   :messages [,@prompts]
	   :use_context t
	   :include_sources t
           :stream ,(or (and gptel-stream gptel-use-curl
                         (gptel-backend-stream gptel-backend))
                     :json-false))))
    (when gptel-temperature
      (plist-put prompts-plist :temperature gptel-temperature))
    (when gptel-max-tokens
      (plist-put prompts-plist :max_tokens gptel-max-tokens))
    prompts-plist))

Generally the code is working, however I still have an open question:
I'd like to make the two new keywords use_context and include_sources configurable when the backend is registered. Currently they are hardcoded to t. How can I access the values I set when I register the backend gptel-make-privategpt?

[karthink]

Thanks! Would you like to raise a PR? I can review the code and we can add it to gptel.

[Aquan1412]

Sure, I just raised the PR. Let me know if there any further changes necessary.