Replies: 5 comments 6 replies
-
|
@dmauler1 are you able to provide: Docker run command for your vLLM containervLLM launch commandRunning vLLM with gdbas well as output from: OUTPUT: Building Hello World GPU Example App https://github.com/lamikr/rocm_sdk_builder You should expect to see a following output if the application can communicate with your GPU. Simple CPU vs GPU benchmarks Very simple benchmark that shows how to run the same math operation both in the CPU and GPU is available on as pytorch program which can be run on jupyter notebook. On CPU the expected time is usually around 20-30 seconds. It can be executed with these commands: |
Beta Was this translation helpful? Give feedback.
-
|
Docker command VLLM command I'm unclear what VLLM GDB is but if I can be pointed in its direction I'll run it. Here is the output from rocminfo `root@8c5b2f75969b:/# rocminfo
|
Beta Was this translation helpful? Give feedback.
-
|
you should be able to You can check docs or ask llm how to use, by recollection you run above command, then type run, it'll ask something about like debugd you can say yes or no, it'll try and run, and crash, and when it crashes, type bt, then press enter and it will give you a backtrace (bt) which helps us tell what's going on a bit better. |
Beta Was this translation helpful? Give feedback.
-
|
Ok I figured out how to get gdb working, here is the output assuming I did things correctly. `root@8c5b2f75969b:/# gdb -ex r --args python /opt/rocm_sdk_612/bin/vllm serve Qwen/Qwen2.5-1.5B-Instruct --tensor-parallel-size 2 For help, type "help". INFO 03-15 15:11:02 model_runner.py:1116] Loading model weights took 1.4478 GB Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information You can suppress this exception and fall back to eager by setting: During handling of the above exception, another exception occurred: Traceback (most recent call last): (gdb) |
Beta Was this translation helpful? Give feedback.
-
|
Also worth noting I had to run the lspci command on the host, the container doesn't have the command. |
Beta Was this translation helpful? Give feedback.
-
|
i presume your container is ubuntu, to check the container may not have it, or you may need to run it as sudo below link seems to suggest it's in pciutils on ubuntu https://askubuntu.com/questions/135996/why-my-lscpi-command-not-found that said, makes it pretty clear that at least torch is picking up both your cards so you may not need to bother with pciutils gdb should provide the necessary debug as it seems it's having trouble passing a nested error, probably relating to writing to /tmp/ does your docker setup allow sufficient disk space? i presume you've tried sudo vllm? may need to check vllm write privs. sometimes if you've installed stuff as sudo it wants you to run it as that as well, or add the container's user to relevant privileged group |
Beta Was this translation helpful? Give feedback.
-
|
Installing pciutils did the trick and you are correct the cards are there. root@8c5b2f75969b:/# lspci | grep Radeon Here's the output from uname I also checked the OS file and it shows ubuntu 24 Btw this is using the docker image on the projects main page The container has 304GB of free space and no need to run sudo as the container is setup to run as root. |
Beta Was this translation helpful? Give feedback.
-
|
it is also likely your error is more appropriately channeled to the vllm team as it seems you have build roc successfully unless you are getting errors in other applications, which the hello_world.sh and python script seems to indicate is not the case. |
Beta Was this translation helpful? Give feedback.
-
|
So I managed to get rocm to compile in a vanilla ubuntu docker. I tried installing vllm using ./babs.sh -b binfo/extra/ai_tools.blist and the base vllm does work. However, it looks like --tensor-parallel-size still fails with the same error. I'll continue to dig on my end for a solution. |
Beta Was this translation helpful? Give feedback.
-
you may want to try installing from git to confirm if the cause is lamikr's vllm, rocm installation, or an issue with vllm. if you are able, try running with gdb again but running the 'bt' or 'backtrace' command before 'quit' and share results. although that may be unnecessary, as just passing your error through an LLM (I'm not a contrib here): If this doesn't work, it seems like ollama supports multi-gpu and there is a project for amd in particular. I'm not sure if this fits your use case. But going by that error, if you're not able to get it to cooperate, this seems like an upstream and known issue with Triton rather than this project. |
Beta Was this translation helpful? Give feedback.
-
|
I unfortunately I have not ever tried to run vllm with multiple GPUs. I could in theory try to use that with my framework 16 laptop which has both the gfx1102/7700S discrete gpu and gfx1103 igpu. I am still in process for finalizing the rocm-633 support, and I would say that it's propably better to test again the multi-gpu support with this release once it's finished. I have now builded the rsb 6.3.3 for gfx1030, gfx1150 and gfx1201 gpus on Fedora and Mageia distributions. I however need to test the build process little bit more as I have still needed to fix some build breaks while building myself. (so not tested yet with clean build without needing to make any changes). Once I have time, all the extra apps are also still waiting the update to latest release versions, so far I have only updated the llama-cpp from those. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
First of all thank you so much for putting this project together, being able to run container lamikr/rocm_sdk_builder:612_01_cdna and vllm just work is awesome. So far everything seems to work but if I try to use multi GPU --tensor-parallel-size 2 it eventually errors out with the following exceptions.
`(VllmWorkerProcess pid=2700) INFO 03-13 02:53:42 model_runner.py:1116] Loading model weights took 1.4478 GB
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:02<00:00, 2.46s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:02<00:00, 2.46s/it]
INFO 03-13 02:53:44 model_runner.py:1116] Loading model weights took 1.4478 GB
INFO 03-13 02:53:44 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250313-025344.pkl...
INFO 03-13 02:53:44 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20250313-025344.pkl.
ERROR 03-13 02:53:44 engine.py:387] BackendCompilerFailed.init() missing 1 required positional argument: 'inner_exception'
`
Is there a more correct way to run vllm with multi gpu support or did I trip over a bug?
Thanks for any insight!
Beta Was this translation helpful? Give feedback.
All reactions