An example of building a reverse image search engine i.e. Content Based Image Retrieval
Code from the following blog post:
Uses the Caltech 256 dataset for indexing and testing
The model used to index the dataset features to create an embedding vector from is the DINOV2 model.
Based on experiements from using Convolutional models such as ResNet and ViT models such as CLIP, DINOV2 has the highest accuracy in terms of being able to learn image features specific to each category, based on results of manual search.
DINOV2 is trained using self-supervised learning without training labels so its able to generalize better to unseen datasets?
The above would require evaluation metrics to verify...
Both CLIP and ResNet models are unable to differentiate between object categories and sometimes retrieve images based on backgrounds or textures alone e.g matching an airplane with a helicopter with similar backgrounds; or returning image of rhino for query of image of elephant as they have similar textures...
Run the weaviate DB via compose:
docker compose -f compose.yaml up
Create the schema first:
python create_schema.py
This creates a class of Image with 2 attributes:
- image, which stores a base64 encoding of the image
- filepath, which stores filepath of image
To index the images:
python upload_image_data.py
To start the webapp:
flask --app webapp/app run --debug
-
Using PCA to perform dimensionality reduction to help improve extracted features to improve accuracy ?
-
Do we need to perform individual object detection for each image?