Skip to content

Joint representation of image and text through a Canonical Correlation Analysis

Notifications You must be signed in to change notification settings

theevann/Image-and-Text-Search

Repository files navigation

CCA used for Image to Tag and Tag to Image Search

Implementation of this paper http://slazebni.cs.illinois.edu/publications/yunchao_cca13.pdf
Using CNN features.


You will need to have the MS COCO database and give path to the db folder in some python files.

Make sure to use python 3

1. Install dependencies

Install pycocotools and other python dependencies

git clone https://github.com/pdollar/coco.git && cd coco/PythonAPI && make install && pip3 install -e .
pip3 install --upgrade gensim nltk numpy torch torchvision tqdm Pillow scikit-image
python3 -m nltk.downloader stopwords

2. Extract image and text features from COCO

mkdir features
python3 Helper/extractTextFeatures.py --dataset-path /path/to/coco
python3 Helper/extractImageFeatures.py --dataset-path /path/to/coco

3. Compute CCA

python3 main.py --output CCA_0

4. Test CCA

Try:

python3 I2T.py --name CCA_0.npy
# or
python3 T2I.py --name CCA_0.npy

About

Joint representation of image and text through a Canonical Correlation Analysis

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published