dtype mismatch error for IPAdapterPlusXL

[yyl-github-1896]

Error information

Traceback (most recent call last):
File "/data/xxx/source/xxx/flux.py", line 207, in
main(args)
File "/data/xxx/source/xxx/flux.py", line 196, in main
image = image_generation_pipe(prompt, pipe, image_prompt, prior_pipe)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/source/xxx/flux.py", line 126, in image_generation_pipe
image = pipe.generate(
^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/ip_adapter/ip_adapter.py", line 396, in generate
) = self.pipe.encode_prompt(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 406, in encode_prompt
prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/transformers/models/clip/modeling_clip.py", line 1490, in forward
text_embeds = self.text_projection(pooled_output)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/xxx/anaconda3/envs/xxx/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 125, in forward
return F.linear(input, self.weight, self.bias)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c# 10::Half

My code with error

 ```
from diffusers import StableDiffusionXLPipeline
        pipe = StableDiffusionXLPipeline.from_pretrained(
            my_path_to_model,
            torch_dtype=torch.float16,
            add_watermarker=False,
            local_files_only=True
        ).to("cuda")
        if use_image_prior:
            from ip_adapter import IPAdapterPlusXL
            pipe = IPAdapterPlusXL(
                pipe,
                my_path_to_model2, 
                my_path_to_model3, 
                device="cuda",
                num_tokens=16
            )

    image = pipe.generate(
        pil_image=image_prompt, 
        num_samples=1, 
        num_inference_steps=30, 
        seed=42,
        prompt=prompt, 
        scale=0.5)[0][0]
    return image
```

[yyl-github-1896]

Seems that the SDXL models provided in this repo has problems and causes the issue above. I have fixed this issue with the following code:

from diffusers import DiffusionPipeline
        pipe = DiffusionPipeline.from_pretrained(
            "stable-diffusion-xl-base-1.0",
            torch_dtype=torch.float16,
            local_files_only=False,
            variant="fp16", 
            use_safetensors=True
        ).to("cuda")
        if use_image_prior:
            pipe.load_ip_adapter(
                "h94/IP-Adapter/",
                subfolder="sdxl_models",
                weight_name="ip-adapter_sdxl.bin"
            )

[FrankHong77]

Damn，I confront with the same problem. Solve it by truning "RealVisXL_V1.0" into "stable-diffusion-xl-base-1.0"

[FrankHong77]

Thanks for your share !!