Evaluation of manually created ground truth for multi-view people localization

In the field of multi-view people localization, only a few works consider a non-planar ground surface. In this article we introduce a framework for collecting ground truth data in such case, we show characterization of specific errors and introduce a method to automatically merge multiple ground truth data generated by different users to form a more reliable reference ground truth. We use this reference ground truth to evaluate the error rate, the accuracy and the recall of subjects (6 laymen and 3 with domain knowledge). We show that even laymen can work accurately, but even subjects with domain knowledge miss a number of people in a crowded scene. Our findings show that creating ground truth data requires special attention in this field.