Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Groq Models #613

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

lumpidu
Copy link
Contributor

@lumpidu lumpidu commented Feb 3, 2025

The llama-3.1-70b-versatile model has been removed from the Groq API. With deepseek-r1's rising popularity, testing distilled versions on Groq's infrastructure could be worthwhile.

  • add deepseek-r1-distill-llama-70B
  • replace llama-3.1-70b-versatile with llama-3.3-70b-versatile

Note that Groq doesn't support the max. context lengths of 128K on the free tier. At least I get an API error if I exceed ~6000 tokens:

 Request too large for model deepseek-r1-distill-llama-70b in organization XYZ service tier on_demand
 on tokens per minute (TPM): Limit 6000, Requested 16955, please reduce your message size and try again.
 Visit https://console.groq.com/docs/rate-limits for more information.

lumpidu and others added 3 commits February 3, 2025 18:13
- add deepseek-r1-distill-llama-70B
- replace llama-3.1-70b-versatile with llama-3.3-70b-versatile
@krschacht
Copy link
Contributor

@lumpidu I think the issue is that you still had an assistant that was referencing the llama model you removed. I updated the assistants file.

Can you confirm this now works for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants