How to edit the expression in ip-adapter-plus-face_sdxl？

Hello！I am using ip-adapter-plus-face_stxl-vit-h to generate facial consistency, but I found that the facial expressions are very similar to the original image. Even if I added emoticons in the prompt, I cannot edit them. Is there any way to solve this problem？ @xiaohu2015 