KERMIT - Knowledge-EmpoweRed systeM In hateful meme deTection

Hello, everyone 👋🏻!

This is a project based on Meta AI's Hateful Meme detection challenge.
The goal is to correctly classify memes that are offensive from those that are not.

To do this, I built a neural network that can take in images, text and knowledge extracted from a meme. The name of this system is KERMIT - Knowledge-EmpoweRed systeM In hateful meme deTection.

If you want to know more, read on.

Images 🖼

The first step is to extract knowledge from the image.

The visual and textual knowledge extraction part can be found in the tutorial file in the repo.
ImageAI system to recognise objects in the image.
DeepFace system to recognise faces in the image.
Image Captioning system to describe what is in the image.

Knowledge 🌍

I created a system for extracting knowledge from the text of a meme using wikidata.

Graph-Based System 🚀

The graph-based system architecture is as shown in the image above. The idea is to take the resulting graph from the meme, apply any graph embedding algorithm to it and return a vector of features. At that point, the feature vector of the knowledge graph is concatenated with the feature vector of the image and text embedding.