com.openai.unity 7.7.1

[StephenHodgson]

What's Changed

• More Function utilities and invoking methods
  • Added FunctionPropertyAttribute to help better inform the feature how to format the Function json
  • Added FromFunc<,> overloads for convenance
  • Fixed invoke args sometimes being casting to wrong type
  • Added additional protections for static and instanced function calls
  • Added additional tool utilities:
    • Tool.ClearRegisteredTools
    • Tool.IsToolRegistered(Tool) - Tool.TryRegisterTool(Tool)
• Updated Sample Chat Behaviour with updated tool implementations
• com.utilities.rest -> 2.5.3

Full Changelog: 7.7.0...7.7.1

---

This discussion was created from the release com.openai.unity 7.7.1.

[pinkwerks]

Could someone elaborate on the FunctionAttribute a bit more? I read the README and it refers to finding methods with Function attributes, but I was hoping for an example of a method that used it. Previously I had a class where I had the schema with argument descriptions as well as the function itself. I was thrilled to see these attributes thinking it would be awesome to avoid all the JObject schema stuff, so I'm hoping to port code....

public class GetPositionTool
{
    public class Args
    {
        public string ObjectName { get; set; }
    }

    public static OpenAI.Function FunctionSchema = new OpenAI.Function(
    nameof(GetPositionCmd),
    "Gets the position of an object",
    new JObject
    {
        ["type"] = "object",
        ["properties"] = new JObject
        {
            [nameof(Args.ObjectName)] = new JObject
            {
                ["type"] = "string",
                ["description"] = "Name of the object."
            },
        },
        ["required"] = new JArray { nameof(Args.ObjectName) }
    });

    internal static string GetPositionCmd(Args functionArgs)
    {
        var go = GameObject.Find(functionArgs.ObjectName);
        if (go != null)
        {
            return $"World space position: {go.transform.position}";
        }
        else
        {
            return $"Could't find object named {functionArgs.ObjectName}!";
        }
    }
}

--- (elsewhere in ChatBehavior.cs) ----
private readonly List<Tool> assistantTools = new List<Tool>();
assistantTools.Add(GetPositionTool.FunctionSchema);
var request = new ChatRequest(conversation.Messages, tools: assistantTools);

Now with 7.7 this is appearing easier, and at first glance I thought the Function attribute would make this much tidier... but I'm missing out on how to pass argument descriptions and if they are required. At first glance I was thinking I could refactor my code and write something like the below.

Maybe another way of stating it: can Tool.GetOrCreateTool or Tool.FromFunc declare descriptions or optional parameters?

[Function("Gets the position of an object")]
internal static string GetPositionCmd(
    [FunctionProperty("Name of the object.")]
    [Required]
    string ObjectName 
{
        ...
}

--- (elsewhere in ChatBehavior.cs) ---

var tools = Tool.GetAllAvailableTools();
var request = new ChatRequest(conversation.Messages, tools: tools);

[StephenHodgson]

Hi @pinkwerks! I added 3 new attributes in the 7.7.x releases so far:

Previously, you could define a function call as a json object, which was a manual process. Now you can specify tools by registering static methods, instances of objects and their method names, and by func<,> callbacks.

• FuntionAttribute: Marks static methods available for tool calls, and provides a way to optionally define a description.
• FunctionParameterAttribute: Adds a way to define a description to parameters of any method used in tool calls.
• FunctionPropertyAttribute: Adds a way to better define and refine the property json that is automatically generated by reflection. This includes a description, required flag, defaultValue, and possibleValues. This last attribute is really only useful if you have complex objects you're passing in as a parameter in your method.

[StephenHodgson]

On a side note, [Required] is not a valid parameter attribute for this system. Parameters are required by default unless you set a default value in the method like so:

[Function("My Custom Function")]
public static void MyMethod(
    [FunctionParameter("A required value")] string requiredValue, 
    [FunctionParameter("An optional value")] string optionalValue = null) { }

[StephenHodgson]

Btw you can also register tools directly using Tool.GetOrCreateTool(Type type, string methodName), Tool.GetOrCreateTool(object instance, string methodName) or Tool.FromFunc(string name, Func<,> func)

[StephenHodgson]

com.openai.unity/OpenAI/Packages/com.openai.unity/Tests/Weather/WeatherService.cs

Lines 16 to 19 in 4402fa0

	[Function("Get the current weather in a given location")]
	public static async Task<string> GetCurrentWeatherAsync(
	[FunctionParameter("The location the user is currently in.")] string location,
	[FunctionParameter("The units the use has requested temperature in. Typically this is based on the users location.")] WeatherUnit unit)

[pinkwerks]

Wow this is so much nicer - Thanks for the extra explanations!

[pinkwerks]

Any hints on how to use arrays or lists as function parameters? I had Arrays working the old way, but it wasn't obvious how to migrate to the new stuff, I've tried float[] and List<float> with no luck.

I suppose I could just change the signature to have 3 floats in this particular case.... 🤔 You mentioned FunctionPropertyAttribute, but it didn't look like it supported Array types, or at least in a way obvious to me.

    public static string SetPosition(
        [FunctionParameter("X, Y, and Z position in world space")] float[] position)

Error: ArgumentException: Object of type 'Newtonsoft.Json.Linq.JArray' cannot be converted to type 'System.Single[]'.

[StephenHodgson]

Can't read the screen shots. Plz put errors in text/code blocks.

[StephenHodgson]

You're getting invalid operation because the methods are not awaitable. Use regular Invoke, not InvokeAsync.

I'd need to see exactly where you're calling invoke on function call to know for sure.

[pinkwerks]

That's the thing, I'm not calling Invoke or InvokeAsync myself. The only code I've written is this (which worked ok before) 🤔

public class GetPositionTool
{
    [Function("Get the world space position of the object")]
    public static string GetPositionCmdNew(
        [FunctionParameter("Name of object to query")] string objectName)
    {
        var go = GameObject.Find(objectName);
        if (go != null)
        {
            var msg = $"{objectName} position: {go.transform.position}";
            Debug.Log(msg);
            return msg;
        }
        else
        {
            return $"Could't find object named {objectName}!";
        }
    }
}

--- (elsewhere in ChatBehaviour.cs) ---

 assistantTools.Add(
     Tool.GetOrCreateTool(
         typeof(GetPositionTool),
         nameof(GetPositionTool.GetPositionCmdNew)));

--- (stack trace) ---

InvalidOperationException: The function GetPositionTool_GetPositionCmdNew did not return a valid Task.
OpenAI.Function.InvokeAsync[T] (System.Threading.CancellationToken cancellationToken) (at ./Library/PackageCache/[email protected]/Runtime/Common/Function.cs:349)
UnityEngine.Debug:LogException(Exception)
OpenAI.<InvokeAsync>d__47`1:MoveNext() (at ./Library/PackageCache/[email protected]/Runtime/Common/Function.cs:360)
System.Runtime.CompilerServices.MoveNextRunner:Run()
Utilities.Async.AwaiterExtensions:RunOnUnityScheduler(Action) (at ./Library/PackageCache/[email protected]/Runtime/Async/AwaiterExtensions.cs:248)
Utilities.Async.SimpleCoroutineAwaiter:Complete(Exception) (at ./Library/PackageCache/[email protected]/Runtime/Async/SimpleCoroutineAwaiter.cs:47)
Utilities.Async.<ReturnVoid>d__27:MoveNext() (at ./Library/PackageCache/[email protected]/Runtime/Async/AwaiterExtensions.cs:334)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

Function.cs is doing a function.MethodInfo.Invoke() afaik. Should that "is not" be "is"?

com.openai.unity/OpenAI/Packages/com.openai.unity/Runtime/Common/Function.cs

Line 347 in 052512c

if (function.MethodInfo.Invoke(function.Instance, invokeArgs) is not Task task)

[StephenHodgson]

You got to be invoking it somewhere.
Or else you wouldn't get to this point.

Where are you stopping to handle the tool outputs?

[pinkwerks]

Ah my bad, I was relying heavily on the Sample's ChatBehavior.cs which calls InvokeFunctionAsync as part of Submit chat. When I updated the package and sample, I forgot I had patched up that function to deal with more tools than the non-async image generation stuff. Thanks again!