-
-
Notifications
You must be signed in to change notification settings - Fork 625
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[PROPOSAL] Allow the client to pass a callback that receives arguments Instructor sends to LLM providers #911
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Looks good to me! Reviewed everything up to ed5aeb9 in 1 minute and 31 seconds
More details
- Looked at
238
lines of code in3
files - Skipped
0
files when reviewing. - Skipped posting
0
drafted comments based on config settings.
Workflow ID: wflow_Mu7WEfmRYd5TFu8u
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet
mode, and more.
Hmm personally I think this could be solved by using something like logfire or langsmith which would help capture and log the information a lot better. Have you tried those options? |
I use third-party logging tools like those. I don't use those ones in particular, but I'm familiar with them. I don't have issues with such tools. They're important in production use cases. However, there are scenarios where:
I also don't think the default mode to access whatever instructor sends to a provider should be to send data to a third-party provider that happens to have instrumented instructor or the underlying client. |
What are your thoughts on attaching the callbacks to the constructor of the client rather than the create call? from_openai(client, callbacks=[]) |
That works, but something to consider is that clients will frequently want to tie these callbacks to a specific invocation, like so: def get_logging_callback(some_id: str):
def log_llm_args(args: list[Any], kwargs: dict[str, Any]):
logger.info(f"Calling LLM provider for invocation {some_id}")
logger.info(f"LLM args: {args}")
logger.info(f"LLM kwargs: {kwargs}")
return log_llm_args
def invoke_llm_for_task(task_id: str):
client = instructor.from_anthropic(
Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
)
user_info = client.chat.completions.create(
model="claude-3-haiku-20240307",
response_model=UserInfo,
messages=[
{
"role": "system",
"content": "You are an expert at extracting information about users from sentences.",
},
{"role": "user", "content": "John Doe is 30 years old."},
],
max_tokens=4096,
max_retries=3,
provider_args_callback=get_logging_callback(task_id)
) I always instantiate the client before invocations or a span of them in my code base, so attaching the callbacks to the constructor is easy. If Or, if you decide to make clients stateful such that passing a previously-instantiated client around is the recommended way to use the library, that might make things challenging later. I don't see why you'd do that. Still, I think attaching the callbacks to the Your call. Some other considerations unrelated to the above:
|
Hi! Bumping to see if this is something you'd want to do. |
sorry been out. i want to do this but almost in the client, vs the create call. what are your thoughts on this? |
we're going to clean this up a bit with a refactor that will make it easier |
Hey! Sorry just saw this. Yeah I'm fine with that strategy but also, keep me posted on the refactor and I can do it after. Thank you! |
Instructor doesn't expose an easy way to see or use the raw arguments that Instructor finally sends over to a language model: #907, #767, #888 (comment))
This PR is a sketch of an idea that solves this problem in a way that:
instructor
clients to receive these arguments and do whatever they want with them, every timeinstructor
calls out to an language model provider (including every retry).It does this by allowing users to pass in a callback, like so:
The code below is a working implementation for Anthropic's synchronous client, that applies to
create
andcreate_with_completion
.This is more-or-less what it'd look like for other providers if this is something you want to do.
An extension we could add is to provide an
attempt_num
and intermediate completions received before retries, which might be helpful for some use cases.I'm personally keen to have this kind of functionality in
instructor
, some way or another (vs. just logging)If the strategy below works for you, I can submit a PR that works with sync/async and the major providers (OpenAI/Anthropic/Google).
If not, would love to know either:
a) This is not a problem you care to solve.
b) You also want to solve it, but differently.
Thanks!
Summary:
Introduced
provider_args_callback
to log or handle arguments passed to language model providers ininstructor
.Key points:
provider_args_callback
parameter tocreate
,create_with_completion
, and other relevant methods ininstructor/client.py
.patch
function ininstructor/patch.py
to handleprovider_args_callback
.retry_sync
andretry_async
functions ininstructor/retry.py
to callprovider_args_callback
with request arguments and keyword arguments.Generated with ❤️ by ellipsis.dev