A Multimodal User Interface for Remote Object Exploration in Teleoperation Systems

This paper describes the design and implementation of a multimodal user interface for exploration and surface reconstruction of remote environments in teleoperation systems. The sensory information collected by heterogeneous sensors at the teleoperated site is integrated by the system to build a meaningful representation of the remote environment. The user interface includes streaming of video information, surface reconstruction and visualization of sampled data from tactile and proximity sensors, and a haptic device (the CyberTouch virtual reality glove) returning distance information through vibrotactile actuators. The glove also enables intuitive guidance of the remote end-effector, as long as an appropriate mapping between the operator's hand and the robotic device is established. The user interface is implemented on top of a distributed architecture for teleoperation, designed according to a software framework exploiting advanced CORBA features. The multimodal user interface, along with the teleoperation framework, has been successfully applied to control simulated robotic devices operating in virtual environments, and is currently being deployed in a real robot workcell comprising a manipulator and several sensory devices.