Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

custom api #1118

Closed
wants to merge 1 commit into from
Closed

custom api #1118

wants to merge 1 commit into from

Conversation

SubGlitch1
Copy link
Contributor

is this how u want it? @psyray

@yogeshojha
Copy link
Owner

What is this related to? Please provide more context @SubGlitch1

@yogeshojha yogeshojha added question Further information is requested needs more info and removed question Further information is requested labels Dec 8, 2023
@psyray
Copy link
Contributor

psyray commented Dec 8, 2023

@yogeshojha related to #1114
@SubGlitch1 👍
But, please, document explicitely your changes

@SubGlitch1
Copy link
Contributor Author

@yogeshojha
I had seen a lot of people wanting to be able to use their own custom/local LLms. This can be done quite easily by changing the code to include an env variable which the gpt.py treats as the base url if it exists.

There are a growing number of openai compatible api server on github one of the most known ones being text-generation-ui. By using a project like this a user can host their own LLM api server which has the exact same scheme as the official openai api.

This obviously greatly increases privacy as the information of targets doesn't get sent to openai anymore. It also makes the gpt feature free as you don't have to pay for the api tokens anymore obviously. It also enables users who live in geo blocked areas to use the gpt feature.

@yogeshojha
Copy link
Owner

Thanks @SubGlitch1 , do all these api have same response? Can you provide me the response for text-generation-ui

@SubGlitch1
Copy link
Contributor Author

@yogeshojha text-generation-webui has the exact same scheme of openai. The request/response scheme will be exactly the same. I have tested on my own PC
Test details
OS: amd64 arch Linux
llm: openchat3.5_Q3

@SubGlitch1
Copy link
Contributor Author

@yogeshojha

https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API#sse-streaming

You can see the examples here. A lot of projects which use the openai api e.g. gpt-engineer also have options to change the base api url for the exact same reason.

@psyray
Copy link
Contributor

psyray commented Jan 5, 2024

@SubGlitch1 I didn't know the subject, but copilot is usable with your modifications ?

@SubGlitch1
Copy link
Contributor Author

@psyray the issue you are referencing has been opened because the op wants to get gpt4 access for free. His suggested integration is not possible as copilot isn't open source but by integrating my solution one could host any model that they want for free.

@psyray
Copy link
Contributor

psyray commented Jan 5, 2024

@psyray the issue you are referencing has been opened because the op wants to get gpt4 access for free. His suggested integration is not possible as copilot isn't open source but by integrating my solution one could host any model that they want for free.

Thanks 👍

@AnonymousWP AnonymousWP added the Waiting for Merge Already Worked, waiting to merge label Feb 21, 2024
@yogeshojha yogeshojha closed this Aug 16, 2024
@SubGlitch1 SubGlitch1 deleted the custom branch August 16, 2024 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants