A novel knowledge-compatibility benchmarker for semantic segmentation

The quality of a semantic annotation is typically measured with its averaged class-accuracy value, whose computation requires scarce ground-truth annotations. We observe that humans accumulate knowledge through their vision and believe that the quality of a semantic annotation is proportionally related to its compatibility with the vision-based knowledge. We propose a knowledge-compatibility benchmarker, whose backbone is a regression machine. It takes as input a semantic annotation and the vision-based knowledge, then outputs an estimate of the corresponding averaged class-accuracy value. The knowledge encodes three kinds of information, namely: cooccurrence statistics, scene properties and relative positions. We introduce three types of feature vectors for regression. Each specifies the characteristics of a probability vector that captures the compatibility between an annotation and each kind of the knowledge. Experiment results show that the Gradient Boosting regression outperforms the n-Support Vector regression. It achieves best performance at an R²-score of 0.737 and an MSE of 0.034. This indicates not only that the vision-based knowledge resembles humans' common sense but also that the feature vector for regression is justifiable.