MFAS: Multimodal Fusion Architecture Search

Juan-Manuel Pérez-Rúa (1, 2) , Valentin Vielzeuf (3, 1) , Stéphane Pateux (1) , Moez Baccouche (1) , Frédéric Jurie (3)

1 Orange Labs R&D [Rennes]
2 Samsung AI Centre [Cambridge]
3 Equipe Image - Laboratoire GREYC - UMR6072

We tackle the problem of finding good architectures for multimodal classification problems. We propose a novel and generic search space that spans a large number of possible fusion architectures. In order to find an optimal architecture for a given dataset in the proposed search space, we leverage an efficient sequential model-based exploration approach that is tailored for the problem. We demonstrate the value of posing multimodal fusion as a neural architecture search problem by extensive experimentation on a toy dataset and two other real multimodal datasets. We discover fusion architectures that exhibit state-of-the-art performance for problems with different domain and dataset size, including the NTU RGB+D dataset, the largest multi-modal action recognition dataset available.

Mots clés: Action Recognition; Multimodal Fusion; Neural Architecture Search

Domaines: Intelligence artificielle [cs.AI]; Vision par ordinateur et reconnaissance de formes [cs.CV]; Réseau de neurones [cs.NE]; Multimédia [cs.MM]

Juan-Manuel Pérez-Rúa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, Frédéric Jurie. MFAS: Multimodal Fusion Architecture Search. CVPR 2019, Jun 2019, Long Beach, United States. ⟨hal-02068293⟩