Remove People

Proof-of-concept for removing unwanted people and objects from images. You can follow this documentation with a video!

Example 1 of the Remove People notebook results.

Example 2 of the Remove People notebook results.

This feature is available in the main photo apps (Apple and Google), but thought it would be cool to recreate the functionality from scratch using existing over-the-counter AI solutions.

This is how object removal works in this implementation:

1. Detect objects in the image using a Mask R-CNN model with default weights from pytorch. The model was pre-trained using the COCO dataset.
2. Manually select the objects in the image that you want to keep.
3. Remove the objects that you don't want and create a PNG without them.
4. Use the OpenAI API to generate clean background pixels without this objects.

Originally I tried using existing cloud solutions for the object detection, but their results were bounding boxes without the exact contours of the objects. This is why I decided to use a local model and it proved to be a better solution. The results given by cloud providers are available in the Cloud Examples notebook, but they are nothing special.

Environment Setup

Create and activate a Python 3.9 virtual environment:

python3.9 -m venv ~/venvs/removepeople
source ~/venvs/removepeople/bin/activate

Install the required dependencies and create a Jupyter kernel named removepeople:

./cmd/install.sh

To clean up resources, run:

jupyter kernelspec uninstall removepeople
deactivate
rm -rf ~/venvs/removepeople

Set up the OpenAI API key in a .env file:

echo "OPENAI_API=your_api_key" > .env

Brief Code Explanation

This project includes two Jupyter notebooks.

	Description
Remove People	Main notebook for removing unwanted people and objects from images.
Cloud Examples	Examples of object detection with cloud providers. Requires local aws credentials and a service-account.json file for GCP. (Additional dependencies: pip install boto3 google-cloud-vision).

Running Notebooks

1. Finish the environment setup as explained in this file.
2. Open file in Jupyter.
3. Connect to the removepeople kernel.
4. Execute the cells, adding your own images.

To add images, place them in the images/original folder and set the image_name variable in the first code cell of the notebook.

Additional Information

There are a couple of things that could be improved in this implementation:

• It would be cool to automatically detect the protagonists of the image to avoid this being a manual process.
• The results when an image contains shadows are weird because the image editing model adds objects over the shadows. To improve this, the model for object detection could be improved to detect shadows and remove them.
• As of this writing, the OpenAI API for image editing requires a 1:1 ratio. If you change to a different provider you might avoid this limitation. If your image has a different dimension the code will automatically resize.