Add popular open source models as in-tree support

[kerthcet]

What would you like to be cleaned:

Add a list of open source models as in-tree support for quick start. Models like:

• DeepSeek R1
• Qwen Models
• llama3 models

Why is this needed:

Quick start for users.

[kerthcet]

/milestone v0.2.0

[kerthcet]

Generally the steps look like:

• create a new folder catalog under the docs
• take the latest Qwen model QwQ-32B for example, we'll have a new folder named catalog/qwen/qwq-32b/
• define a model named qwq-32b under the folder, source from HF.
• create a playground for the model with tuned parameters

I think it might be hard to test the model because of the high cost, it's ok, let's have a template for now. I'll test locally and tune the configurations, as well as the modelspec.inferenceConfig.flavors. Quite similar to the https://github.com/InftyAI/llmaz/tree/main/docs/examples.

Let's use vllm by default.

[googs1025]

I will test this part 🤔 , maybe use example presentation first. I will test vllm and ollama part. Also, due to hardware limitations, maybe we should support small parameter examples first. ex 1.5b 7b, because model download is very time-consuming....

https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

https://ollama.com/library/deepseek-r1:1.5b

[googs1025]

I use ollama as backendRuntime in deepseek-r1 and it seems to work fine.(just for example)

apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: deepseek-r1-15b
spec:
  familyName: opt
  source:
    uri: ollama://deepseek-r1:1.5b
---
apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: deepseek-r1-15b
spec:
  replicas: 1
  modelClaim:
    modelName: deepseek-r1-15b
  backendRuntimeConfig:
    backendName: ollama

root@VM-0-11-ubuntu:/home/ubuntu# kubectl get pods
NAME                   READY   STATUS     RESTARTS   AGE
deepeek-playground-0   0/1     Init:0/1   0          9m8s
deepseek-r1-15b-0      1/1     Running    0          3h28m
qwen2-0--5b-0          1/1     Running    0          3h21m
root@VM-0-11-ubuntu:/home/ubuntu# kubectl exec -it qwen2-0--5b-0 -- bash
root@qwen2-0--5b-0:/# ollama list
NAME          ID              SIZE      MODIFIED
qwen2:0.5b    6f48b936a09f    352 MB    3 hours ago
root@qwen2-0--5b-0:/# ollama ps
NAME    ID    SIZE    PROCESSOR    UNTIL
root@qwen2-0--5b-0:/# ollama serve
Error: listen tcp 0.0.0.0:8080: bind: address already in use

root@VM-0-11-ubuntu:/home/ubuntu# kubectl get pods
NAME                   READY   STATUS     RESTARTS   AGE
deepeek-playground-0   0/1     Init:0/1   0          9m8s
deepseek-r1-15b-0      1/1     Running    0          3h28m
qwen2-0--5b-0          1/1     Running    0          3h21m
root@VM-0-11-ubuntu:/home/ubuntu# kubectl exec -it qwen2-0--5b-0 -- bash
root@qwen2-0--5b-0:/# ollama list
NAME          ID              SIZE      MODIFIED
qwen2:0.5b    6f48b936a09f    352 MB    3 hours ago
root@qwen2-0--5b-0:/# ollama ps
NAME    ID    SIZE    PROCESSOR    UNTIL
root@qwen2-0--5b-0:/# ollama serve
Error: listen tcp 0.0.0.0:8080: bind: address already in use

[googs1025]

But when I use vllm as backendruntime, there seems to be an error, but I'm not sure which part is the problem.

apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: deepseek-ai-15b
spec:
  familyName: opt
  source:
    modelHub:
      modelID: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
---
apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: deepeek-playground
spec:
  replicas: 1
  modelClaim:
    modelName: deepseek-ai-15b

root@VM-0-11-ubuntu:/home/ubuntu# kubectl get pods
NAME                   READY   STATUS     RESTARTS   AGE
deepeek-playground-0   0/1     Init:0/1   0          109s
deepseek-r1-15b-0      1/1     Running    0          3h43m
qwen2-0--5b-0          1/1     Running    0          3h36m

event

7m45s       Normal    Killing                     pod/deepeek-playground-0             Stopping container model-runner
7m30s       Warning   FailedPreStopHook           pod/deepeek-playground-0             PreStopHook failed
7m29s       Normal    Scheduled                   pod/deepeek-playground-0             Successfully assigned default/deepeek-playground-0 to cluster1-worker2
7m29s       Normal    Pulled                      pod/deepeek-playground-0             Container image "inftyai/model-loader:v0.0.10" already present on machine
7m29s       Normal    Created                     pod/deepeek-playground-0             Created container: model-loader
7m29s       Normal    Started                     pod/deepeek-playground-0             Started container model-loader
2m10s       Normal    Pulled                      pod/deepeek-playground-0             Container image "vllm/vllm-openai:v0.7.3" already present on machine
2m10s       Normal    Created                     pod/deepeek-playground-0             Created container: model-runner
2m10s       Normal    Started                     pod/deepeek-playground-0             Started container model-runner
2m9s        Warning   Unhealthy                   pod/deepeek-playground-0             Startup probe failed: Get "http://10.6.1.14:8080/health": dial tcp 10.6.1.14:8080: connect: connection refused
2m8s        Normal    Killing                     pod/deepeek-playground-0             Stopping container model-runner
114s        Warning   FailedPreStopHook           pod/deepeek-playground-0             PreStopHook failed
113s        Normal    Scheduled                   pod/deepeek-playground-0             Successfully assigned default/deepeek-playground-0 to cluster1-worker2
112s        Normal    Pulled                      pod/deepeek-playground-0             Container image "inftyai/model-loader:v0.0.10" already present on machine

log

root@VM-0-11-ubuntu:/home/ubuntu# kubectl logs -f -p deepeek-playground-0
Defaulted container "model-runner" out of: model-runner, model-loader (init)
Error from server (BadRequest): previous terminated container "model-runner" in pod "deepeek-playground-0" not found

[googs1025]

The problem seems to be that the machine I'm testing on doesn't have a gpu? vllm seems to require a gpu device to test. 🤔

[pacoxu]

if you don't have GPU, you'd better use ollama or lamma.cpp IIUC.

The example seems correct to me.

[googs1025]

Due to limited bandwidth... , I will confirm those cases and submit example cases in the near feature (maybe weekend).

[kerthcet]

/milestone v0.3.0