add openvino as a backend choice.

[alanzhai219]

how about adding openvino as a backend choice.

Current backend is fixed on pytorch or ipex, which means customers have no other choice. If other backend can provide better perf or scaling, why not support such backend?

add an extension api, which allowing external developers contribute or integrate better backend into AI-Playground.

Usually, users select specific model, type inputs, click run button and get the output. It means AI-Playground should provide the best outputs to the users. Users doesn't care about what's going on the background.

But AI-Playground cannot make sure current inference engine is fastest, right? So, why not leave an open api for external developers to integrate other excellent inference engine?

Describe alternatives you've considered

AI-Playground can make a decision on which inference engine is used in each release binary. If pytorch is better, this release use pytorch. If openvino is better, this release use openvino.

Leave the pressure on the backend teams and keep the race going.
If so, each release perf and scaling should be better than the last release.

Additional context

N/A

[bobduffy]

OpenVINO is on the roadmap. Stay tuned

For the near term a user will select the backends they want to install and choose those backends.

Many people choose to use GGUF specifically to run a model that they can't run with IPEX. If someone would like to create a decision tree that weighs factors and will recommend, that would be great. Simplest solution for now is to provide choice as we've done with ComfyUI and Llama.cpp

[alanzhai219]

OpenVINO is on the roadmap. Stay tuned

For the near term a user will select the backends they want to install and choose those backends.

Many people choose to use GGUF specifically to run a model that they can't run with IPEX. If someone would like to create a decision tree that weighs factors and will recommend, that would be great. Simplest solution for now is to provide choice as we've done with ComfyUI and Llama.cpp

Make sense. Customers always choose the best solution. Just leave the pressure on internal teams. Performance will be improved in such race.