Add OpenAI Priority Load Balancer for Azure OpenAI

[simonkurtz-MSFT]

This PR is only intended to showcase how the OpenAI Prioritized Load Balancer integrates.

This PR should NOT be merged.

[simonkurtz-MSFT]

Hi @pamelafox & @kristapratico,

This is how the OpenAI Priority Load Balancer integrates. Nevermind the hard-coded backend and the location of the backends list in this PR. I don't intend to ask for a merge, but this was the best way to give you an idea of the setup.

If you have two AOAI instances with the same model, you can plug them both in and should see load-balancing.

[simonkurtz-MSFT]

I brought up two AOAI instances and related assets and configured both instances as backends in app.py. Then I started to have a conversation.

Both backends are responding. It's important to note that this is not a uniform distribution because available backends are randomized (have to do so as part of multi-process workloads).

At no point did the conversation break down or showed any kind of error through the chat bot.