Large-scale datasets play an important role in the application of deep learning methods to various practical tasks. Many crowdsourcing tools have been proposed for annotation tasks; however, these tasks are relatively easy. Non-obvious annotation tasks require professional knowledge (e.g., medical image annotation) and non-expert annotators need to be trained to perform such tasks. In this paper, we propose Trafne, a framework for the effective training of non-expert annotators by combining feedback from the system (auto validation) and human experts (expert validation). Subsequently, we present a prototype implementation designed for brain tumor image annotation. We perform a user study to evaluate the effectiveness of our framework compared to a traditional training method. The results demonstrate that our proposed approach can help non-expert annotators to complete a non-obvious annotation more accurately than the traditional method. In addition, we discuss the requirements of non-expert training on a non-obvious annotation and potential applications of the framework.
Video [2m19s]
Publication
Shugo Miyata, Chia-Ming Chang and Takeo Igarashi, 2022, Trafne: A Training Framework for Non-Expert Annotators with Auto Validation and Expert Feedback. The 24th International Conference on Human-Computer Interaction (HCI International 2022), Virtual Conference, 26 June-1 July 2022[PDF]
Related Publications
Chia-Mingc Chang, Xi Yang, and Takeo Igarashi. 2022. An Empirical Study on the Effect of Quick and Careful Labeling Styles in Image Annotation. The 48th International Conference on Graphics Interface and Human-Computer Interaction (Gl 2022), Montreal, QC, Canada, 17-19 May 2022 [PDF]
C. M. Chang, C. H. Lee, and T. Igarashi. 2021. Spatial Labeling: Leveraging Spatial Layout for Improving Label Quality in Non-Expert Image Annotation. In CHI Conference on Human Factors in Computing Systems (CHI ’21), Yokohama, Japan. May 8–13, 2021 [PDF]
C. M. Chang, S. D. Mishra and T. Igarashi, 2019, A Hierarchical Task Assignment for Manual Image Labeling. The IEEE Symposium on Visual Languages & Human-Centric Computing (VL/HCC 2019), Memphis, Tennessee, US, 14-18 October 2019 [PDF]
