runner/docker: integrate ComfyUI LivePortrait

[emranemran]

No description provided.

[leszko]

@emranemran I've built the ComfyUI image, but I'm not sure how to use it 🙃

I see some samples in https://github.com/kijai/ComfyUI-LivePortraitKJ/tree/main/examples, I think the most closest to what we do is this one: https://github.com/kijai/ComfyUI-LivePortraitKJ/blob/main/examples/liveportrait_realtime_example_01.json

But still this workflow is quite big, could you try to covert it into something that we can use in our runner, like one of these ones? https://www.notion.so/livepeer/Video-Processing-ComfyUI-Nodes-Workflows-1340a34856878005b0fcf9eb02d1819a?pvs=4#1580a348568780f1a718fbf49bf98d4c

[emranemran]

@leszko some notes for you:
I used the realtime one as a reference and made the following changes:

• the source images will be generated via the LoadImageFromUrlOrPath node which I added to the comfyui docker file. This allows us to pass an url in the json provided by the user instead of having to load from disk. This also bypasses the comfystream single node limitation I referenced below.
• replace the webcam capture pieces with just LoadImage node which will be driven from our pipeline (comfystream call).
• TODO: the example uses the ImageResizeKJ and GetImageSizeandCount nodes. I'm not 100% sure if these are needed but I tried adding them via the repo you see in the comfyui docker file - however they don't seem to be detected when I run comfyui locally:
  

So the previous note I shared re: comfystream supporting only a single LoadImage node can (should?) be handled with the above modified workflow if we can figure out the missing dependency. In other words, the condition here and here should still be valid and we won't need to modify comfystream for the time being (discussed with Yondon as well).

Next Steps

• address the TODO above -- i've attached the json files generated by the UI here
  liveportrait_live_example.json
• then integrate the json into the pipeline

[emranemran]

One more note @leszko: i think the PreviewImage node should be replaced with the SaveImage node so that the img is returned to the pipeline. I forgot to update this in the example above.

[emranemran]

One more update @leszko: I was staring at the comfyui logs and noticed this:

ModuleNotFoundError: No module named 'model_management'

Cannot import /Users/emran/custom_nodes/ComfyUI-KJNodes module for custom nodes: No module named 'model_management'

so hopefully installing that module will get us past that missing node error. According to GPT, it's to do with some PATH thing and custom_nodes dir not being able to find that module....

[leszko]

This does not build for me for some reason. But I haven't investigated yet why. I think I found the issue 👇

[leszko]

Suggested change

	cd ComfyUI-load-image-from-url && \
	pip install -r requirements.txt
	cd ComfyUI-load-image-from-url

[leszko]

@emranemran I spent some time today on your PR. Added some inline comments. Apart from that:

1. Error "No module named 'model_management'"
I was not able to solve it. Seems like a missing dependency, but I was not able to install it. Instead I removed the custom node ComfyUI-KJNodes. IMO we may actually not need it.

2. Running E2E
I've created the JSON Workflow which I believe should work. I got this error, but it may be an issue with the input video not having a face 🙃

Error that I get:

  File "/root/.pyenv/versions/3.10.15/lib/python3.10/site-packages/comfy/cmd/execution.py", line 249, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "/root/.pyenv/versions/3.10.15/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/root/.pyenv/versions/3.10.15/lib/python3.10/site-packages/comfy/cmd/execution.py", line 219, in map_node_over_list
    process_inputs(input_dict, i)
  File "/root/.pyenv/versions/3.10.15/lib/python3.10/site-packages/comfy/cmd/execution.py", line 208, in process_inputs
    results.append(getattr(obj, func)(**inputs))
  File "/comfyui/custom_nodes/ComfyUI-LivePortraitKJ/nodes.py", line 349, in process
    out = pipeline.execute(
  File "/comfyui/custom_nodes/ComfyUI-LivePortraitKJ/liveportrait/live_portrait_pipeline.py", line 77, in execute
    raise ValueError("No face detected in FIRST source image")

I've created a draft PR to you branch (the intention is not merge it but rather to present what I'm talking about in points 1 and 2. #359

[emranemran]

@leszko i'm seeing the same No face detected in FIRST source image error when testing with the local trickle method and using an input driving video with human faces on it. Here are the tests I ran today:

• My first step was to try and get a local workflow running on my mac. See attached workflow: (
  liveportrait-works-local-mac-export-for-docker.json ) which did work and running Queue generates frames within ~12s on my mac m1. Feel free to try it on your end.
• Using this as a reference, I modified it to use the Load/SaveTensor nodes and replaced the webcam node with LoadImage -- see my DEBUG commit on this PR. Using this workflow, I still get the same No face detected in FIRST source image error.
• So I'm thinking there's two options here:
  • the Issue list on the LivePortrait github repo has similar complaints from some users. One of the suggestion was to have a very neutral face when starting the pipeline otherwise it's finicky and won't work correctly.
  • the frame being passed to the workflow is corrupted. This isn't true because I saved the frames to disk to verify and I see the frames as expected with the faces (see DEBUG commit as well).

So next steps:

• The server trickle testing method doesn't work on my end - my router/firewal drops packets to mediamtx ingress so rest of pipeline is not reliable. Could you try it out with the changes in this PR and see if it just happens to work?
• This is not a great idea because it might introduce unintended effects but maybe we modify this logic to try multiple frames instead of just 1? https://github.com/kijai/ComfyUI-LivePortraitKJ/blob/main/liveportrait/live_portrait_pipeline.py#L77