Prediction of Depth Camera Missing Measurements Using Deep Learning for Next Best View Planning

Depth images usually contain pixels with invalid measurements. This paper presents a deep learning approach that receives as input a partially-known volumetric model of the environment and a camera pose, and it predicts the probability that a pixel would contain a valid depth measurement if a camera was placed at the given pose. The proposed network architecture consists of a 3D Convolutional Neural Network (CNN) module and a 2D CNN module, connected by a deep learning attention-based projection module. The method was integrated into a CNN-based probabilistic Next Best View plan-ner, resulting in a more realistic prediction of the information gain for each possible viewpoint with respect to state of the art approaches. Experiments were carried out in tabletop scenarios using a robot manipulator with an eye-in-hand depth camera.