Difference between Video Mask and Source Video with pixel set to Grey 127

[deepbeepmeep]

What is the interest of using a Video Mask over setting to grey 127 pixels to be inpainted in the source Video ?

[hanzhn]

If you consider beyond Masked V2V task (in/outpainting), all various video generation and editing tasks in a whole perspective,

1. you need a src_mask to denote where should be changed and where should not be changed. We call them reactive and inactive.
2. And also you need a default value (we choose grey 127) for tasks without actual source video like T2V and R2V, which means nothing here.

Briefly speaking, it depends on the task you are doing.

If in/outpainting or video extension (including I2V and clip expansion), both src_mask and src_video need to be set accordingly.
If V2V task like (pose, depth, colorization) only src_mask need to be set.
If T2V or R2V, the src_mask need to be all white and src_video need to be all grey.
...

To know the deep reason of it, please refer to Chapter 3.2 of our arXiv paper.

[deepbeepmeep]

Thank you for this explanation.

[yuchenli-sony]

thank you so much it is really helpful