Multi-Level Sensor Fusion with Deep Learning

Valentin Vielzeuf 1, 2 Alexis Lechervy 2 Stéphane Pateux 1 Frédéric Jurie 2
2 Equipe Image - Laboratoire GREYC - UMR6072
GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen
Abstract : In the context of deep learning, this article presents an original deep network, namely CentralNet, for the fusion of information coming from different sensors. This approach is designed to efficiently and automatically balance the trade-off between early and late fusion (i.e. between the fusion of low-level vs high-level information). More specifically, at each level of abstraction-the different levels of deep networks-uni-modal representations of the data are fed to a central neural network which combines them into a common embedding. In addition, a multi-objective regularization is also introduced, helping to both optimize the central network and the unimodal networks. Experiments on four multimodal datasets not only show state-of-the-art performance, but also demonstrate that CentralNet can actually choose the best possible fusion strategy for a given problem.
Type de document :
Pré-publication, Document de travail
2018
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01910858
Contributeur : Frederic Jurie <>
Soumis le : vendredi 2 novembre 2018 - 06:48:46
Dernière modification le : mardi 6 novembre 2018 - 01:16:09

Fichiers

centralNet-journal.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01910858, version 1
  • ARXIV : 1811.02447

Citation

Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, Frédéric Jurie. Multi-Level Sensor Fusion with Deep Learning. 2018. 〈hal-01910858〉

Partager

Métriques