How can I train with a 7900xt/ xtx?

[Anothergazz]

I saw that zluda is being integrated, so thought I would give it a go. Unfortunately it does not work when I start training, I get the following. Any ideas ?

activating venv C:\AI\OneTrainer\OneTrainer\venv
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
ZLUDA device failed to pass basic operation test: index=None, device_name=AMD Radeon RX 7900 XTX [ZLUDA]
Clearing cache directory C:/AI/OneTrainer/OneTrainer/training_samples/JL/Cache! You can disable this if you want to continue using the same cache.
TensorFlow installation not found - running with reduced feature set.
Some weights of the model checkpoint were not used when initializing CLIPTextModel:
['text_model.embeddings.position_ids']
Some weights of the model checkpoint were not used when initializing CLIPTextModelWithProjection:
['text_model.embeddings.position_ids']
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.15.1 at http://localhost:6006/ (Press CTRL+C to quit)
Traceback (most recent call last):
File "C:\AI\OneTrainer\OneTrainer\modules\ui\TrainUI.py", line 480, in __training_thread_function
trainer.start()
File "C:\AI\OneTrainer\OneTrainer\modules\trainer\GenericTrainer.py", line 123, in start
self.model_setup.setup_model(self.model, self.config)
File "C:\AI\OneTrainer\OneTrainer\modules\modelSetup\StableDiffusionXLLoRASetup.py", line 127, in setup_model
model.optimizer = create.create_optimizer(
File "C:\AI\OneTrainer\OneTrainer\modules\util\create.py", line 327, in create_optimizer
optimizer = torch.optim.AdamW(
File "C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\torch\optim\adamw.py", line 68, in init
raise RuntimeError("fused=True requires all the params to be floating point Tensors of "
RuntimeError: fused=True requires all the params to be floating point Tensors of supported devices: ['cuda', 'xpu', 'privateuseone'].

[Calamdor]

I am unsure ZLUDA can used AdamW in fused mode. You can try turning fused off in the optimizer settings (... button next to the optimizer name) and then try again.

[Anothergazz]

Thanks for the suggestion (especially telling me were the button is) and it did help, but it now seems to be stuck. After 20 mins it has not moved on with the second 'enumerating sample paths" (the first part was like lightening) and now the machine is just sitting here with only 8% CPU utilisation.

activating venv C:\AI\OneTrainer\OneTrainer\venv
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
ZLUDA device failed to pass basic operation test: index=None, device_name=AMD Radeon RX 7900 XTX [ZLUDA]
Clearing cache directory C:/AI/OneTrainer/Workspace/JL/Cache! You can disable this if you want to continue using the same cache.
No backup found, continuing without backup...
TensorFlow installation not found - running with reduced feature set.
Some weights of the model checkpoint were not used when initializing CLIPTextModel:
['text_model.embeddings.position_ids']
Some weights of the model checkpoint were not used when initializing CLIPTextModelWithProjection:
['text_model.embeddings.position_ids']
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.15.1 at http://localhost:6006/ (Press CTRL+C to quit)
Could not enable memory efficient attention. Make sure xformers is installed correctly and a GPU is available: CUDA error: named symbol not found
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

enumerating sample paths: 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
enumerating sample paths: 0%| | 0/1 [00:00<?, ?it/s]
caching: 0%|

[Calamdor]

I believe also that ZLUDA needs to use SDP and not xformers.

[Anothergazz]

OK I changed the drop down in training>Attention to SDP, but it is exactly the same.
I notice that there is a message early on
"ZLUDA device failed to pass basic operation test: index=None, device_name=AMD Radeon RX 7900 XTX [ZLUDA]"
Maybe that is a clue

[Anothergazz]

No I have nothing else using ZLUDA. I have been using Stability Matrix which only uses WindowsML so there is no support for OneTrainer on AMD. I may well have not installed it correctly. I have not found a clear documented instal process, The ZLUDA page says very little and there are specific examples for programs other than OneTrainer, or this
[https://github.com/vosen/ZLUDA/discussions/82)

Not certain that works with the star-ui.bat file I have been using for oneTrainer up till now

[Heasterian]

AI projects are not using main ZLUDA repo as it lacks some features, it's fork. You need to have HIP SDK installed to make it work. https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html I'll try write about it later in readme.

[Anothergazz]

Thanks for your time.
I thought I had already installed HIP SDK, still I downloaded it again and installed it. I got the latest version of ZLUDA V3.7 https://github.com/lshqqytiger/ZLUDA/releases unzipped to C:\AI\zluda. I had already added to PATH in the ENV variables %HIP_PATH%bin and C:\AI\zluda
and it still seems to stop at the same point as described above
What am I missing?

[Heasterian]

I would give a try with installing OneTrainer in new location from scratch. It can be something like broken venv or changed branch.

[Anothergazz]

I deleted the old version of OneTrainer and installed a fresh copy in a different location. Exactly the same result. I am going to reset/re-instal this copy of windows and start from scratch. Unless you have any other ideas

[Anothergazz]

Thanks for your help. Lets hope some else has got it working

[Heasterian]

If you're willing to test some more solutions, we probably should ping author of fork of zluda used by project.

[Anothergazz]

Thanks for the suggestion but until something official gets posted. I dont want zluda to take over my life :)

[hansolocambo]

That's when you realize that buying Nvidia since day one, especially for anything related to (pro) 3D and now AI is a not a bad idea.
There's no plan B to Nvidia, 0 competitor for hardware innovations focused solely on professional artists using a PC.
Intel+Nvidia and you'll never have issues anymore.

[Heasterian]

There are valid reasons to chose any of vendors. None of them is connected to this issue, so please keep this discussion out of it.

[Anothergazz]

Just to re-enforce Haesterian. I bought my 7900xtx to play games as it was considerable cheaper than the Nvidia alternative, some time before I discovered AI. Im not a professional just an interested engineer.

[tnginako]

Imagine buying Intel chips

[xcrashdayx]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

[kirian555]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!
Hi, I need help to make it work with a 7800XT, how did you do it?

[Anothergazz]

@kirian555 Thanks for the offer. I spent ages battling with the shifting layers of AMD SW, In the end I gave in and bought an RTX4090, (The price recently dropped and my AMD has held its)

[5P33DY1]

I have a similar card. How did you get OneTrainer working on Windows?

[Calamdor]

https://github.com/Nerogar/OneTrainer/wiki/Lessons-Learnt-and-Tutorials#zluda-notes-by-calamdor

or, as the 7900XTX is one of the few AMD cards that works with RocM and WSL, try that.

https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/wsl/install-radeon.html

[HillQi]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

Can you please brief your steps to made it. I am struggling on my 7900xtx. Thanks.

[Anothergazz]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

Can you please brief your steps to made it. I am struggling on my 7900xtx. Thanks.

The short answer is I gave up a bought an Nvidia card.
Long answer
I found that it kept breaking, (This is not just a onetrainer thing) typically an updated broke something, and I then wasted time trying to identify what is happening and then waiting for the fix. Most users seem to be Linux based which meant windows fixes took even longer. Hopefully all the changes have settled down. A few month ago the 7900xtx price went up and a I sold it for the same price I paid for it and at the same time RTX4090 prices dropped (still a silly price), so the gap was doable for me. I see RTX4090 prices have gone up by 40% since I bought mine, ouch.

[HillQi]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

Can you please brief your steps to made it. I am struggling on my 7900xtx. Thanks.

The short answer is I gave up a bought an Nvidia card. Long answer I found that it kept breaking, (This is not just a onetrainer thing) typically an updated broke something, and I then wasted time trying to identify what is happening and then waiting for the fix. Most users seem to be Linux based which meant windows fixes took even longer. Hopefully all the changes have settled down. A few month ago the 7900xtx price went up and a I sold it for the same price I paid for it and at the same time RTX4090 prices dropped (still a silly price), so the gap was doable for me. I see RTX4090 prices have gone up by 40% since I bought mine, ouch.

........ maybe you got the best solution. The VERSION Magic going crazy! Compatible between versions has gone for so many years. In the past new version means bug fix and improvement, but today it means brand new system!

To those who may fighting for your AMD & windows, please check my share: https://pan.quark.cn/s/ab761fea7314. This is comfyUI portable pack and works for me out of box. ROCm 5.7 + Zluda + torch 2.3.1 + cud118.

This pack in Chinese, please fix it yourself.