Self-supervised learning of visual features through embedding images into text topic spaces

Lluis Gomez, Yash Patel, Marçal Rusinol, Dimosthenis Karatzas, C. V. Jawahar

Producció científica: Capítol de llibreCapítolRecercaAvaluat per experts

88 Cites (Scopus)

Resum

End-to-end training from scratch of current deep architectures for new computer vision problems would require Imagenet-scale datasets, and this is not always possible. In this paper we present a method that is able to take advantage of freely available multi-modal content to train computer vision algorithms without human supervision. We put forward the idea of performing self-supervised learning of visual features by mining a large scale corpus of multimodal (text and image) documents. We show that discriminative visual features can be learnt efficiently by training a CNN to predict the semantic context in which a particular image is more probable to appear as an illustration. For this we leverage the hidden semantic structures discovered in the text corpus with a well-known topic modeling technique. Our experiments demonstrate state of the art performance in image classification, object detection, and multimodal retrieval compared to recent self-supervised or natural-supervised approaches.

Idioma originalAnglès
Títol de la publicacióProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
EditorInstitute of Electrical and Electronics Engineers Inc.
Pàgines2017-2026
Nombre de pàgines10
ISBN (electrònic)9781538604571
DOIs
Estat de la publicacióPublicada - 6 de nov. 2017

Sèrie de publicacions

NomProceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
Volum2017-January

Fingerprint

Navegar pels temes de recerca de 'Self-supervised learning of visual features through embedding images into text topic spaces'. Junts formen un fingerprint únic.

Com citar-ho