[Feature]: Preloading OIDC/JWT upstream tokens on proxy start

[Manouchehri]

The Feature

Upon the proxy config being parsed, we should proactively fetch (and cache) the OIDC tokens (and the exchanged JWTs/tokens) for upstream LLMs.

Only two open question I would like help with (before I implement this myself) 🙂:

1. Does this already happen automatically for Azure OpenAI, since I think LiteLLM creates the client at startup?
2. For Bedrock (and maybe Azure OpenAI), where should I be inserting a caching/prefetching function that occurs whenever there’s a oidc/ defined in a azure_ad_token and/or aws_web_identity_token?

Motivation, pitch

This should improve perf on the first request to Amazon Bedrock when using OIDC, even more so for GitHub Actions and Google Cloud Run (which need to do a HTTP request first to fetch a OIDC token).

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/