Is your feature request related to a specific problem?
Running evals in scrapi consume RunSession LLM tokens instead of Evaluation RunSession LLM tokens which result in RESOURCE_EXHAUSTED\ error generic::RESOURCE_EXHAUSTED: ces.googleapis.com/run_session_llm_token_consumption quota exhausted.
Describe the Solution You'd Like
Ideally, have a mechanism for evals triggered by scrapi to only consume Evaluation RunSession LLM tokens. If that is not possible, add a way to rate limit/token limit scrapi evals so that it does not negatively effect actual users.
Impact on your work
How does this feature impact your work and what are you trying to achieve?
This makes running evals against my prod agent difficult. Evals should not cause my agent to be unavailable because of quota exhausted. Having built in rate limiting would help, but need it to be aware of the available quota so that the evals do not take too long to run.
If this is critical for you, tell us if there is a timeline by when you need
this feature.
ASAP since this will block us from automatically triggering evals against our prod agent.
Willingness to contribute
Are you interested in implementing this feature yourself or submitting a PR?
(Yes/No)
Yes
🟡 Recommended Information
Describe Alternatives You've Considered
A clear and concise description of any alternative solutions or workarounds
you've considered and why they didn't work for you.
Proposed API / Implementation
If you have ideas on how this should look in code, please share a
pseudo-code example.
Additional Context
Add any other context or screenshots about the feature request here.
Is your feature request related to a specific problem?
Running evals in scrapi consume RunSession LLM tokens instead of Evaluation RunSession LLM tokens which result in RESOURCE_EXHAUSTED\ error
generic::RESOURCE_EXHAUSTED: ces.googleapis.com/run_session_llm_token_consumption quota exhausted.Describe the Solution You'd Like
Ideally, have a mechanism for evals triggered by scrapi to only consume Evaluation RunSession LLM tokens. If that is not possible, add a way to rate limit/token limit scrapi evals so that it does not negatively effect actual users.
Impact on your work
How does this feature impact your work and what are you trying to achieve?
This makes running evals against my prod agent difficult. Evals should not cause my agent to be unavailable because of quota exhausted. Having built in rate limiting would help, but need it to be aware of the available quota so that the evals do not take too long to run.
If this is critical for you, tell us if there is a timeline by when you need
this feature.
ASAP since this will block us from automatically triggering evals against our prod agent.
Willingness to contribute
Are you interested in implementing this feature yourself or submitting a PR?
(Yes/No)
Yes
🟡 Recommended Information
Describe Alternatives You've Considered
A clear and concise description of any alternative solutions or workarounds
you've considered and why they didn't work for you.
Proposed API / Implementation
If you have ideas on how this should look in code, please share a
pseudo-code example.
Additional Context
Add any other context or screenshots about the feature request here.