Sohu Chinese Image-Text Matching Competition 2017. We won the third prize.

Requirements

Please clone the Chinese word-segmentation tools https://github.com/thunlp/THULAC-Python into ./NLP

And clone a caffe repo into ./caffe

Download a pre-trained VGG-16 model trained on ImageNet with caffe.

Platform：

tensorflow 1.0

caffe

Model Architecture：

Image -> vgg16 -> 4096-d feature -> fc1 -> fc2 -> distance(image, text)

Text TF-IDF or GMM feature -> fc1 ->  fc2 ->  distance(image, text)

References：

《Learning two-branch neural networks for image-text matching tasks》

《Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation》

Files：

caffe-recurrent/
- extract_feature.py # extract features using VGG-16

Tensorflow/
-BidirectionNet_tfidf.py # train using the tfidf features
-BidirectionNet_lstm.py # train lstm
-test_match_pairList.py # generate the matching result (top-10) using image/text embeddings.

NLP/
-run_SH_split.py # word segmentation and generate vocabulary

-word_type.py # filter nouns and verbs

-run_create_train_txt.py # generate one file .txt based on the word-segmented files and a vocabulary.

-tfidf_from_seg.py # compute tfidf

-kmeans_word2vec.py cluster based on word2vec (fasttext word2vec is recommended. Search 'fasttext' repo)

Key of performance improvment：

model ensemble

OCR （which we did not use）