Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

【feature】llm添加openai风格服务端支持(自定义代理地址和模型名称) #294

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

QiuZiXian
Copy link
Contributor

llm添加openai风格服务端支持(自定义代理地址和模型名称)

使用mindsearch的时候,发现无法使用自定义代理地址、模型名称的openai风格服务端。
GPTAPI有进行模型校验等原因,秉承只add的原则。基于GPTAPI扩展GPTStyleAPI:

支持openai风格:

已测试通过:

  • xinference 1.2.0
  • ollama 0.5.5
  • one-api v0.6.10-alpha.6
  • baichuan 直连
  • lmdeploy 0.7.0

使用方法:

设置api_base、 model_name、key(如果服务端不需要,可忽略)

    model_name = "deepseek-r1:1.5b"
    gpttool = GPTStyleAPI(
        model_type=model_name,
        api_base=api_base,
        key="sk-IXgCTwuoEwxL1CiBE4744688D8094521B70f4aDeE6830c5e",
    )
    res = gpttool.chat(inputs=[
        {
            "role": "user",
            "content": "世界第一高峰是"
        }])

具体参见tests/test_llms/test_gptstyleapi.py

@flingjie
Copy link

是不是用litellm来统一调用各个provider更加简洁和容易维护?

@QiuZiXian QiuZiXian mentioned this pull request Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants