Envoy gateway plugin support with random selection

[kerthcet]

What would you like to be added:

• add a envoy plugin as the first routing plugin, it could be very simple like randomly select the instance
• add makefile command for quick bumping the plugin into the gateway
• add documentation about how to write a custom plugin

Why is this needed:

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[googs1025]

/assign
I'am willing to give this a try 😄

[kerthcet]

Great!

[googs1025]

New feedback:

Inspired by: https://aibrix.readthedocs.io/latest/features/gateway-plugins.html#routing-strategies,

curl -v http://${ENDPOINT}/v1/chat/completions \
    -H "routing-strategy: least-request" \
    -H "Content-Type: application/json" \
    -d '{
        "model": "your-model-name",
        "messages": [{"role": "user", "content": "Say this is a test!"}],
        "temperature": 0.7
    }'

• maybe we can also put the strategy in the header?
• maybe we want a subcomponent like deployment or just use sidecar container?

related docs:

• https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/ext_proc.proto#extensions-filters-http-ext-proc-v3-externalprocessor

and
some useful example: https://github.com/Himenon/envoy-ext-proc-sample/blob/main/sample/01-simple-proxy/main.go

[kerthcet]

Thanks for the research.

maybe we can also put the strategy in the header?

I personally don't think this is a good approach, this should be configured by admin, rather than the user.

maybe we want a subcomponent like deployment or just use sidecar container

Side car is a good choice I think.

Generally, I think we need a KEP to describe how the whole architecture looks like, the smart routing heavily depends on the metrics, but we can not count on prometheus because it's not designed for high concurrent reading. So we may have to design a metric-aggregator. AIBrix has similar designs, but the problem with aibrix is the metric collection is somehow vulnerable, one goroutine loops all the pods at a time, which increases the latency.

[kerthcet]

But for random selector, I think we can start with a simple approach.

[kerthcet]

I have a simple demo in my mind:

• We have a sidecar to expose the metrics, isolated with the biz container
• we have an agent(daemonset) to collect the metrics of all its pods, and push the metrics to the gateway (x times requests reduce compared to Pod itself pushing metrics to gateway)
• the gateway maintains a queue to accept the requests and analyze the data for routing

[kerthcet]

It that's sounds good to you, I will raise a KEP ahead.

[googs1025]

• we have an agent(daemonset) to collect the metrics of all its pods, and push the metrics to the gateway (x times requests

Compared with aibrix, this seems better because agent only focuses on the pod metrics of the node. But this will also make the architecture more complicated. 🤔

[kerthcet]

We can make it an option. Node exporter vs Pod exporter. Big cluster vs small cluster, and I think agent is something we can't avoid in the future.