Fix specifying media uris for google vertex

[mattmatters]

Changes data => fileUri.

[brainlid]

Thanks @mattmatters! I haven't looked into this but I'd love some more information. My guess is that the file data could be Base64 encoded (so the file is uploaded directly), or it could support the fileUri for where it could be downloaded from.

If so, I'm hesitant to break a potentially working solution for some people.

[mattmatters]

Thanks @mattmatters! I haven't looked into this but I'd love some more information. My guess is that the file data could be Base64 encoded (so the file is uploaded directly), or it could support the fileUri for where it could be downloaded from.

If so, I'm hesitant to break a potentially working solution for some people.

Oh sorry I should have supplied more details, I can check on inline data but even just having the data key set at all results in the vertex api returning a 400 error saying that an unknown key data is set. So I don't believe that this has worked for anyone yet.

I did leave the inline data field alone though above it as I haven't been working with inline data and didn't want to mess with that for folks.

[mattmatters]

Just to follow up here I tested with setting inline data to the image_url type and it also does not work. I don't believe this codepath has ever worked for vertex ai.

[brainlid]

Thanks @mattmatters for checking on that. I'd be happy to merge this in. Can you add any relevant documentation and tests for the fix?

[mattmatters]

Ok sorry for the late tests @brainlid but fixed. I did notice some oddities when comparing ChatVertexAI with ChatGoogleAI content part serialization. You can specify an atom for an image type here that gets expanded out (there's also a bug when supplying your own mime type where it always just sets "image/type"), however we don't do that in ChatVertexAI and just inject the mimetype directly in.

I've abused this with ChatVertexAI for the image_url type to send in videos. I think the :image_url and :image content part types may be the wrong abstraction. Maybe it should something like :file_url and :inline_file or something similar and then encoding these content parts makes no assumption on the mime type. (For a seperate project obviously)

[mattmatters]

Sidenote, Vertex AI uses camelcase for content part serialization while the project's implementation of content part serialization is snake case. We should probably check on if that's deliberate or if Google wants folks to use camelcase for their gemini version and they are just maintaining snake case arguments for compatibility. It would greatly simplify both of these implementations if we could just share code between them.

[brainlid]

Thanks @mattmatters! If snake case is supported, I'd prefer to use that as well. The VertexAI module is primarily managed by contribution. If you want to verify that and change it, I'd love to merge it!

[brainlid]

Thanks! Merged!
❤️💛💙💜

[mattmatters]

Thanks @mattmatters! If snake case is supported, I'd prefer to use that as well. The VertexAI module is primarily managed by contribution. If you want to verify that and change it, I'd love to merge it!

Happy to take that on! It's a really weird situation Google has done with splitting these up.