How Much Noise is there in Labels Generated by Humans? A Method to Validate Automatically Generated Bounding Boxes.

In order to train a model or evaluate its safety, high quality labels are necessary. Human labeling is considered gold standard in object detection and object classification problems. This approach is natural - humans do very well in finding cars or pedestrians in an image. However the answers to the same question, provided by different human experts, or even the same expert asked multiple times tend not to be completely identical. In this paper we show better performance of neural networks over humans in 2D object detection tasks by showing neural network labels are closer to human consensus than any particular human labeler. The method we present here may be used to validate labels generated using automated labeling methods, thereby decreasing the need for costly human labeling. For this task we created a dataset of 630 automotive images labeled by 10 different labelers each. Additionally we compare predictions of humans and networks given only single camera images to more accurate labels created using multiple sensors and sequences of images (from Waymo and nuImages datasets). Using the second method we again show better performance of the networks.