Event based self-supervised temporal integration for multimodal sensor data.
A method for synergistic integration of multimodal sensor data is proposed in this paper. This method is based on two aspects of the integration process: (1) achieving synergistic integration of two or more sensory modalities, and (2) fusing the various information streams at particular moments during processing. Inspired by psychophysical experiments, we propose a self-supervised learning method for achieving synergy with combined representations. Evidence from temporal registration and binding experiments indicates that different cues are processed individually at specific time intervals. Therefore, an event-based temporal co-occurrence principle is proposed for the integration process. This integration method was applied to a mobile robot exploring unfamiliar environments. Simulations showed that integration enhanced route recognition with many perceptual similarities; moreover, they indicate that a perceptual hierarchy of knowledge about instant movement contributes significantly to short-term navigation, but that visual perceptions have bigger impact over longer intervals.
RIKEN Brain Science Institute, 2-1 Horosawa, Wako-shi, Saitama 351-0198, Japan. emilia@brain.riken.jp
Emilia I
EI
Barakova
Tino
T
Lourens
D000818:Animals; D001330:Electronic Data Processing; D008959:Models, Neurological; D016571:Neural Networks, Computer; D010465:Perception / Q000502:physiology*; D021641:Recognition, Psychology / Q000502:physiology; D012677:Sensation / Q000502:physiology*; D014159:Transducers