Deep Feature Factorization for Concept Discovery

Edo Collins1    Radhakrishna Achanta2    Sabine Süsstrunk1

1Image and Visual Representation Lab (IVRL), EPFL
2 Swiss Data Science Center, EPFL and ETHZ



Abstract


We propose Deep Feature Factorization (DFF), a method capable of localizing similar semantic concepts within an image or a set of images. We use DFF to gain insight into a deep convolutional neural network's learned features, where we detect hierarchical cluster structures in feature space. This is visualized as heat maps, which highlight semantically matching regions across a set of images, revealing what the network `perceives' as similar. DFF can also be used to perform co-segmentation and co-localization, and we report state-of-the-art results on these tasks.


Results

Unlike NMF in pixel space, DFF decomposes images into parts while the CNN’s learned invariance to rotations, scale, lighting, etc.




Applying DFF with increasing 𝑘 reveals a concept hierarchy, as the cluster qualitatively corresponding to body is split into limbs and midsection, and limbs further into arms and legs.
DFF shows invariance to complex transformations, such as the varied leg positions of the gymnast on the left and back side of the elephants on the right.




Materials





[ Paper ]      [ Poster ]      [ Code ]          

Citation

Edo Collins, Radhakrishna Achanta, Sabine Süsstrunk
Deep Feature Factorization for Concept Discovery
In The Euopean Conference on Computer Vision (ECCV), 2018.

        
  @InProceedings{collins2018,
	author = {Collins, Edo and Achanta, Radhakrishna and Susstrunk, Sabine},
	title = {Deep Feature Factorization For Concept Discovery},
	booktitle = {The European Conference on Computer Vision (ECCV)},
	year = {2018}
  }

Acknowledgements


The authors thank the anonymous reviewers for their useful comments and suggestions.