An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image Annotation

An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image Annotation

Abstract

Assigning a label to difficult data particularly when non-expert annotators attempt to select the best possible label. However, there have been no detailed studies exploring a label selection style during annotation. This is very important and may affect the efficiency and quality of annotation. In this study, we explored the effects of labeling style on data annotation and machine learning. We conducted an empirical study comparing “quick labeling” and “careful labeling” styles in image-labeling tasks with three levels of difficulty. Additionally, we performed a machine learning experiment using labeled images from the two labeling styles. The results indicated that quick and careful labeling styles have both advantages and disadvantages in terms of annotation efficiency, label quality, and machine learning performance. Specifically, careful labeling improves label accuracy when the task is moderately difficult, whereas it is time-consuming when the task is easy or extremely difficult.

Video [1m55s]

Publication

Chia-Mingc Chang, Xi Yang, and Takeo Igarashi. 2022. An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image Annotation. The 48th International Conference on Graphics Interface and Human-Computer Interaction (Gl 2022), Montreal, QC, Canada, 17-19 May 2022 [PDF]

Related Publications

C. M. Chang, C. H. Lee, and T. Igarashi. 2021. Spatial Labeling: Leveraging Spatial Layout for Improving Label Quality in Non-Expert Image Annotation. In CHI Conference on Human Factors in Computing Systems (CHI ’21), Yokohama, Japan. May 8–13, 2021 [PDF]

C. M. Chang, S. D. Mishra and T. Igarashi, 2019, A Hierarchical Task Assignment for Manual Image Labeling. The IEEE Symposium on Visual Languages & Human-Centric Computing (VL/HCC 2019), Memphis, Tennessee, US, 14-18 October 2019 [PDF]