Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModelRatio and CompletionRatio should be maintained by adapter #2023

Open
WqyJh opened this issue Jan 9, 2025 · 0 comments
Open

ModelRatio and CompletionRatio should be maintained by adapter #2023

WqyJh opened this issue Jan 9, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@WqyJh
Copy link
Contributor

WqyJh commented Jan 9, 2025

目前 model ratio 和 completion ratio 是集中维护的,这带来几个问题:

  1. 不同供应商的同一个模型(例如 llama3)可能名字相同,但是价格不同(例如groq价格低于aws),如何区分并设置不同的费率?
  2. GetCompletionRatio 函数冗长,为什么不像 model ratio 一样写一个 map?

更为合理的方案是,参考 ModelList 的维护方法,每个 Adapter 维护一个 RatioMap,在 DoResponse 返回 usage 的时候连带 ratio 一起返回,或者为 adapter 添加一个获取 ratio 的接口。
进一步,ModelList 可以被这个 RatioMap 替代。

image

涉及到的改动量可能比较大,目前只是提出一个想法,欢迎大家讨论。

@WqyJh WqyJh added the enhancement New feature or request label Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant