Getting RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm) when using trained Lora

[Santodan]

have created a Lora with OneTrainer using the following configurations:
OneTrainer_config.json

but when i try to use it, it is giving me this error:

13:03:59-557236 DEBUG    Server: alive=True requests=139 memory=10.44/31.93 status='idle' task='' timestamp=None id='' job=0 jobs=0 total=22 step=0 steps=0 queued=0 uptime=1503
                         elapsed=168.93 eta=None progress=0
13:04:17-097042 DEBUG    Sampler: "default" class=EulerDiscreteScheduler: {'num_train_timesteps': 1000, 'beta_start': 0.00085, 'beta_end': 0.012, 'beta_schedule': 'scaled_linear',
                         'trained_betas': None, 'prediction_type': 'epsilon', 'interpolation_type': 'linear', 'use_karras_sigmas': False, 'use_exponential_sigmas': False,
                         'use_beta_sigmas': False, 'sigma_min': None, 'sigma_max': None, 'timestep_spacing': 'leading', 'timestep_type': 'discrete', 'steps_offset': 1,
                         'rescale_betas_zero_snr': False, 'final_sigmas_type': 'zero', 'clip_sample': False, 'sample_max_value': 1.0, 'set_alpha_to_one': False, 'skip_prk_steps':
                         True}
Load network: D:\AI Generated\models\Lora\niwa_masami.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/395.7 MB -:--:--
13:04:38-392449 DEBUG    Load network: type=LoRA name="niwa_masami" type={'ModuleTypeLora'} keys=794 direct=True

13:04:38-450709 INFO     Base: pipeline=StableDiffusionXLPipeline task=TEXT_2_IMAGE batch=1/1x1 set={'guidance_scale': 6, 'generator': 'cuda:[748301099]', 'num_inference_steps': 20,
                         'eta': 1.0, 'guidance_rescale': 0.7, 'denoising_end': None, 'output_type': 'latent', 'width': 1024, 'height': 1024, 'parser': 'native', 'prompt': 'embeds'}
Load network: D:\AI Generated\models\Lora\niwa_masami.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/395.7 MB -:--:--
13:04:39-259020 DEBUG    Load network: type=LoRA name="niwa_masami" type={'ModuleTypeLora'} keys=794 direct=True
13:04:39-319713 ERROR    Activating network: type=lora args:[<modules.extra_networks.ExtraNetworkParams object at 0x000001DFB164A910>]: RuntimeError
╭──────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────╮
│ D:\AI Generated\sd-next-automatic\modules\lora\networks.py:435 in network_add_weights                                                                                              │
│                                                                                                                                                                                    │
│   434 │   │   try:                                                                                                                                                                 │
│ ❱ 435 │   │   │   new_weight = model_weights.to(devices.device) + lora_weights.to(devices.device)                                                                                  │
│   436 │   │   except Exception:                                                                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: The size of tensor a (2048) must match the size of tensor b (768) at non-singleton dimension 1

During handling of the above exception, another exception occurred:

╭──────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────────────────────────────╮
│ D:\AI Generated\sd-next-automatic\modules\extra_networks.py:102 in activate                                                                                                        │
│                                                                                                                                                                                    │
│   101 │   │   │   if 'include' in signature and 'exclude' in signature:                                                                                                            │
│ ❱ 102 │   │   │   │   extra_network.activate(p, extra_network_args, step=step, include=include, exclude=exclude)                                                                   │
│   103 │   │   │   else:                                                                                                                                                            │
│                                                                                                                                                                                    │
│ D:\AI Generated\sd-next-automatic\modules\lora\extra_networks_lora.py:161 in activate                                                                                              │
│                                                                                                                                                                                    │
│   160 │   │   if has_changed:                                                                                                                                                      │
│ ❱ 161 │   │   │   networks.network_deactivate(include, exclude)                                                                                                                    │
│   162 │   │   │   networks.network_activate(include, exclude)                                                                                                                      │
│                                                                                                                                                                                    │
│ D:\AI Generated\sd-next-automatic\modules\lora\networks.py:543 in network_deactivate                                                                                               │
│                                                                                                                                                                                    │
│   542 │   │   │   │   if shared.opts.lora_fuse_diffusers:                                                                                                                          │
│ ❱ 543 │   │   │   │   │   weights_device, weights_dtype = network_apply_direct(module, batch_updown, batch_ex_bias, deactivate=True)                                               │
│   544 │   │   │   │   else:                                                                                                                                                        │
│                                                                                                                                                                                    │
│ D:\AI Generated\sd-next-automatic\modules\lora\networks.py:461 in network_apply_direct                                                                                             │
│                                                                                                                                                                                    │
│   460 │   │   if updown is not None:                                                                                                                                               │
│ ❱ 461 │   │   │   self.weight = network_add_weights(self, lora_weights=updown, deactivate=deactivate)                                                                              │
│   462                                                                                                                                                                              │
│                                                                                                                                                                                    │
│ D:\AI Generated\sd-next-automatic\modules\lora\networks.py:437 in network_add_weights                                                                                              │
│                                                                                                                                                                                    │
│   436 │   │   except Exception:                                                                                                                                                    │
│ ❱ 437 │   │   │   new_weight = model_weights + lora_weights # try without device cast                                                                                              │
│   438 │   │   self.weight = torch.nn.Parameter(new_weight, requires_grad=False)                                                                                                    │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: The size of tensor a (2048) must match the size of tensor b (768) at non-singleton dimension 1
Progress  1.13s/it █████████████████████████████████ 100% 20/20 00:22 00:00 Base
13:05:03-492804 DEBUG    Decode: vae="vaeFtMse840000EmaPruned_vaeFtMse840k" dtype=torch.float16 device=cuda:0 upcast=False slicing=True tiling=False latents=torch.Size([1, 4, 128,
                         128]):cuda:0:torch.float16 time=0.581
13:05:04-419869 DEBUG    GC: current={'gpu': 1.63, 'ram': 11.17, 'oom': 0} prev={'gpu': 12.94, 'ram': 11.18} load={'gpu': 10, 'ram': 35} gc={'gpu': 11.31, 'py': 36612}
                         fn=process_decode:vae_decode why=threshold time=0.34
13:05:04-455304 INFO     Save: image="D:\AI Generated\Generated\Images\2025-01-12\2025-01-12-02028-obliviousMix_v10_1086272-niwa masami orange hair long hair curly hair.jpg"
                         type=JPEG width=1024 height=1024 size=331154
13:05:04-457531 INFO     Processed: images=1 its=0.42 time=47.36 timers={'move': 22.76, 'pipeline': 22.75, 'decode': 2.08, 'gc': 0.85, 'encode': 0.61, 'offload': 0.58, 'preview':
                         0.23} memory={'ram': {'used': 11.18, 'total': 31.93}, 'gpu': {'used': 1.63, 'total': 15.98}, 'retries': 0, 'oom': 0}
13:05:04-723030 DEBUG    GC: current={'gpu': 1.63, 'ram': 11.18, 'oom': 0} prev={'gpu': 1.63, 'ram': 11.18} load={'gpu': 10, 'ram': 35} gc={'gpu': 0.0, 'py': 381}
                         fn=process_images:process_images_inner why=final time=0.26

but the Lora seem to be loaded since it can generate if i remove the <lora:_name_> from the prompt and the image that is generated is respecting the Lora

I've raised the issue to the SD.Next repo and he gave me the following reply:

lora can include weights for multiple modules - most loras are unet-only, but it can be unet and te1 and te2, etc.
and each module weights can be thousands of layers.
this error means that one of the weights is incorrect and was skipped.
if you set SD_LORA_DEBUG=true env variable, you can get more information, but in general at least one module weight in your lora is not correct. purely guessing based on reported sizes, it sounds like its training te, but using te2 instead of te1 or other way around.

as I'm a newbie in this, how can i check this that he has sent me, and what should i change in the trainer configurations to be able to use my Lora?