Deep Feature Factorization for Content-based
Image Retrieval and Localization

Edo Collins    Sabine Süsstrunk

Image and Visual Representation Lab (IVRL), EPFL




Abstract


State of the art content-based image retrieval algorithms owe their excellent performance to the rich semantics encoded in the deep activations of a convolutional neural network. The difference between these algorithms lies mostly in how activations are combined into a compact global image descriptor. In this paper, we propose to use deep feature factorization to achieve this goal. By factorizing CNN activations, we decompose an input image into semantic regions, represented by both spatial saliency heatmaps and basis vectors serving as descriptors for those regions. When combined to form a global image descriptor, our experiments show that DFF surpasses the state of the art in both image retrieval and localization of the region of interest within the set of retrieved images.


Results

Deep Feature Factorization (DFF) decomposes an image or image set into a “bag-of-concepts”, where a concept is represented by a saliency heatmap, spatially highlighting a particular concept in every image (shown below), and a basis vector representing that concept in deep CNN feature space. When the DFF rank is 𝑘, the factorization often aligns the 𝑘 most salient semantic concepts in the image.




For retrieval, our goal is to obtain for each image a vector, the DFF descriptor v, which will serve as a global descriptor. The steps to obtain the descriptor from the DFF basis vectors are shown in the pipeline image at the top of the page. We score the match between query and target descriptors using cosine similarity.

Given the set of top matching images with respect to some query, it is straight forward to localize the relevant image regions by applying DFF jointly to the query and the matches, which generates corresponding saliency heatmaps.





Materials





[ Paper ]      [ Poster ]      [ Code ]          

Citation

Edo Collins, Sabine Süsstrunk
Deep Feature Factorization for Content-Based Image Retrieval and Localization
In the IEEE International Conference on Image Processing (ICIP), 2019.

        
  @InProceedings{collins2019,
	author = {Collins, Edo and S{\"u}sstrunk, Sabine},
        title = {Deep Feature Factorization for Content-Based Image Retrieval and Localization},
        booktitle = {IEEE International Conference on Image Processing (ICIP)},
        pages = {874--878},
	year = {2019}
  }

Acknowledgements


The authors thank the anonymous reviewers for their useful comments and suggestions.