Use tools and functions without cache or with a specific cache instance

[sibbl]

In my application I need to have a tool function created from a method, which is a local function inside the method. It should not be cached at all and I only need it for the lifetime of the GPT request itself.

I'm currently migrating from an older version and don't understand why all Function constructors are enforcing a cache - I don't need it.
Is there any way not to use it?

Describe the solution you'd like

I'd be happy if the cache was optional or if I just could have some caching instance, where I can decide myself whether I throw it away or use it for the context of a request and then drop it.

For the convenience of toolCall.InvokeFunctionAsync I could imagine creating some var toolCache = new ToolCache() and pass it to my ChatRequest. I should be able to also pass this toolCache to toolCall.InvokeFunctionAsync or simply use toolCache.InvokeFunctionAsync(toolCall, cancellationToken).

By default, some static DefaultToolCache may be used for backwards compatibility. As long as I can override this default behavior and use my own tool cache, this empty instance doesn't hurt performance wise.

Describe alternatives you've considered

So far I only see the option to add the function with a random ID. However, I'm also missing a way to remove this specific tool/function from the cache - so I can just clear the whole cache. This also means I'm not able to use toolCall.InvokeFunctionAsync safely because the cache may have been cleared somewhere else in my web app meanwhile.

Please let me know what you think. Thanks a lot in advance!

[StephenHodgson]

You can already do this by specifying the json structure and just doing the manual implementation of your function inline.

That was the default behavior and the Tool Cache is meant to be more advanced feature.

[sibbl]

Thanks for the quick reply! by specifying the json structure - do you have a quick reference how to do this?

In the Function class there is absolutely no constructor which does not use the function cache and does not throw an exception if the function is already in the function cache.

I used to use this constructor public Function(string name, string description = null, JsonNode parameters = null) in my old code but as I said, it's now broken as it stores the function internally in the cache.

[StephenHodgson]

I used to use this constructor public Function(string name, string description = null, JsonNode parameters = null) in my old code but as I said, it's now broken as it stores the function internally in the cache.

Gotcha, I'll be sure to fix it so we do not cache json defined functions with #299

[StephenHodgson]

• JSON defined functions/tools should not be cached. #308

[sibbl]

But coming back to my solution proposal: why not allow the user to control whether to use global caching or use caching instances? In my case I'd prefer to use a method which I can invoke over building the JSON schema manually. Do you have an opinion about this? I'd be happy to contribute.

[StephenHodgson]

Mostly because I'm concerned about dangling pointers to functions and memory leaks. I think the library should not expose the internal workings of the cache, as a means to protect the api consumer from having to manage this on their own. Besides the library is the one creating and managing those instances internally anyway, so exposing it doesn't make much sense.

That being said, if you passed a predefined json object it shouldn't cache that function anyway bc there is no instance where we are actually invoking the function directly. The consumer will still be on the hook to match and call the appropriate function/tool call based on the chat/assistant response.