Retry using a different model for failures #6105
-
Hi, do we have a guidance or tips on how we can perform retry operations using a different openai model by specifiying error codes and/or status codes, and a fallback model? If not, is it something you can add to your plans for the future? |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 3 replies
-
Hey @dmytrostruk is this something you can take a look at and add to your research of filters? Thanks. |
Beta Was this translation helpful? Give feedback.
-
Hi Dmytro,
Thanks a lot! This looks helpful. I have a few questions:
1. I see we changed the model id in your example but I think we can also change the endpoint, deploymentName, and credentials, am I correct? The way I see it is `modelId` is like a custom key that points to the settings used when using the `AddOpenAIChatCompletion` method in the builder.
Reason I'm asking is we have multiple instances behind afd endpoints.
2. We can potentially have multiple fallback endpoints, will we be able to retry in an order?
For example: Try endpoint/model 1, if exception endpoint 2, then endpoint 3, etc.
3. I see a warning saying "IFunctionInvocationFilter" is subject to change or removal. How reliable is this method? Is there any way to make it safer?
Thanks,
Prajwal
…________________________________
From: Dmytro Struk ***@***.***>
Sent: Wednesday, May 8, 2024 7:26 AM
To: microsoft/semantic-kernel ***@***.***>
Cc: Prajwal Pyakurel ***@***.***>; Mention ***@***.***>
Subject: Re: [microsoft/semantic-kernel] Retry using a different model for failures (Discussion #6105)
@prajwalpyakurelMS<https://github.com/prajwalpyakurelMS> Here is a direct link to retry example:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Filtering/RetryWithFilters.cs
—
Reply to this email directly, view it on GitHub<#6105 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BFI26YVXKNSAIGF4TODVDFLZBIY3DAVCNFSM6AAAAABHEFVXUSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJVGUYDI>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Update: I tried this out, and I saw a couple of issues:
|
Beta Was this translation helpful? Give feedback.
-
Yes, but I will need to prepare another example for that, since it will contain more code. Thanks for providing more details!
Actually,
Yes, but you will need to define this retry logic manually. You can also avoid the example with Filters I shared above and try to use another approach by specifying retry policy on
At the moment, Filters are temporarily marked as
I'm trying it in
That could be an issue why you don't see that exception didn't end up getting caught. We have In all cases, I have more information about your case now, and I will try to add new example that covers your scenario soon. |
Beta Was this translation helpful? Give feedback.
@prajwalpyakurelMS Here is a direct link to retry example:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Filtering/RetryWithFilters.cs