[Feature]: LiteLLM support VertexAI/ Hugging Face Models

[ishaan-jaff]

The Feature

user is trying to use litellm for calling VertexAI / hugging face model

Motivation, pitch

Twitter / LinkedIn details

No response

[krrishdholakia]

i believe this is already supported via model garden - https://docs.litellm.ai/docs/providers/vertex#model-garden

[nerfZael]

@krrishdholakia it is not supported, the Vertex API throws an error:

{"error":{"message":"VertexAIException - 400 Failed to deserialize the JSON body into the target type: instances[0]: missing field `inputs` at line 1 column 74 

FYI this is my minimal reproducible setup:

import litellm
litellm.vertex_project = "0000" # project id
litellm.vertex_location = "us-central1"

response = litellm.completion(
  model="vertex_ai/0000", # endpoint id 
  messages=[{"role": "user", "content": "write code for saying hi from LiteLLM"}],
)
print(response)

My guess is it's because it's a hugging face model that's deployed to vertex