triton resnet example

[asaiacai]

Requested by #3347. This shows how run a torch model server with NVIDIA triton. Requires torch models to be exportable by torch.jit

To run the example:

sky launch -c triton tritonserver.yaml
pip install tritonclient[http] numpy
export TRITONSERVER=$(sky status --ip triton)
python triton_client.py

[Michaelvll]

Thanks for adding the example for triton @asaiacai! This is awesome. I am wondering if we can add a readme under that repo for the usage of the example, which could simply be copying the comments at the top of the YAML to a README file.

I will try to test this soon.

Just curious, will it be possible to get the triton server example work with our SkyPilot serve, so it can be easily scaled up?

Another future work would be getting it work for some popular LLMs, and we can consider adding both the image model and LLM on our AI gallery: https://skypilot.readthedocs.io/en/latest/gallery/index.html