Point cloud annotation in virtual reality based on selection of control points by gaze or hand pointing

Computer-assisted point cloud annotation is the process where a user assigns labels to points within a 3D point cloud. Research studies indicate that annotation in an immersive virtual reality (VR) environment can provide advantages over conventional 2D annotation tools. Most point cloud annotation methods in VR are based on the selection of groups of points by drawing a bounding volume. However, point selection using bounding volumes may suffer in complex scenarios due to the point cloud containing irregular shapes. This paper proposes a novel computer-assisted approach for point cloud annotation in a VR environment featuring a head-mounted display (HMD). The approach only requires the user to select and label a sparse number of control points. Points in the nearest neighborhood of a selected control point are automatically annotated by applying an algorithm based on a shortest path tree. Two point selection modes, based on ray casting, are also proposed and evaluated: a first mode where control points are selected by pointing with the HMD hand controller, and a second mode where control points are selected through HMD eye-tracking. Due to the high density of points, a data filtering process over multiple frames is also proposed to reduce the effect of ray noise on the selection. The proposed approach has been compared against a state-of-the-art method for point cloud annotation in VR based on bounding box selection. The results show that both selection modes of the proposed approach improve annotation quality and reduce annotation time.