CaltechAUTHORS
  A Caltech Library Service

A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments

Parisi, German I. and Barros, Pablo and Fu, Di and Magg, Sven and Wu, Haiyan and Liu, Xun and Wermter, Stefan (2018) A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE , Piscataway, NJ, pp. 2330-2335. ISBN 978-1-5386-8094-0. http://resolver.caltech.edu/CaltechAUTHORS:20190314-132852281

[img] PDF - Accepted Version
See Usage Policy.

1693Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20190314-132852281

Abstract

Crossmodal conflict resolution is crucial for robot sensorimotor coupling through the interaction with the environment, yielding swift and robust behaviour also in noisy conditions. In this paper, we propose a neurorobotic experiment in which an iCub robot exhibits human-like responses in a complex crossmodal environment. To better understand how humans deal with multisensory conflicts, we conducted a behavioural study exposing 33 subjects to congruent and incongruent dynamic audio-visual cues. In contrast to previous studies using simplified stimuli, we designed a scenario with four animated avatars and observed that the magnitude and extension of the visual bias are related to the semantics embedded in the scene, i.e., visual cues that are congruent with environmental statistics (moving lips and vocalization) induce the strongest bias. We implement a deep learning model that processes stereophonic sound, facial features, and body motion to trigger a discrete behavioural response. After training the model, we exposed the iCub to the same experimental conditions as the human subjects, showing that the robot can replicate similar responses in real time. Our interdisciplinary work provides important insights into how crossmodal conflict resolution can be modelled in robots and introduces future research directions for the efficient combination of sensory observations with internally generated knowledge and expectations.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
https://doi.org/10.1109/IROS.2018.8594036DOIArticle
https://arxiv.org/abs/1802.10408arXivDiscussion Paper
Additional Information:© 2018 IEEE. Open-source code: cml.knowledge-technology.info. This research was supported by National Natural Science Foundation of China (NSFC), the China Scholarship Council, and the German Research Foundation (DFG) under project Transregio Crossmodal Learning (TRR 169). The authors would like to thank Jonathan Tong, Athanasia Kanellou, Matthias Kerzel, Guochun Yang, and Zhenghan Li for discussions and technical support.
Funders:
Funding AgencyGrant Number
National Natural Science Foundation of ChinaUNSPECIFIED
China Scholarship CouncilUNSPECIFIED
Deutsche Forschungsgemeinschaft (DFG)TRR 169
Record Number:CaltechAUTHORS:20190314-132852281
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20190314-132852281
Official Citation:G. I. Parisi et al., "A Neurorobotic Experiment for Crossmodal Conflict Resolution in Complex Environments *," 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, 2018, pp. 2330-2335. doi: 10.1109/IROS.2018.8594036
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:93825
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:14 Mar 2019 20:36
Last Modified:14 Mar 2019 20:36

Repository Staff Only: item control page