Crowd Teaching with Imperfect Labels

Yao Zhou, Arun Reddy Nelakurthi, Ross MacIejewski, Wei Fan, Jingrui He

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The need for annotated labels to train machine learning models led to a surge in crowdsourcing - collecting labels from non-experts. Instead of annotating from scratch, given an imperfect labeled set, how can we leverage the label information obtained from amateur crowd workers to improve the data quality? Furthermore, is there a way to teach the amateur crowd workers using this imperfect labeled set in order to improve their labeling performance? In this paper, we aim to answer both questions via a novel interactive teaching framework, which uses visual explanations to simultaneously teach and gauge the confidence level of the crowd workers. Motivated by the huge demand for fine-grained label information in real-world applications, we start from the realistic and yet challenging assumption that neither the teacher nor the crowd workers are perfect. Then, we propose an adaptive scheme that could improve both of them through a sequence of interactions: the teacher teaches the workers using labeled data, and in return, the workers provide labels and the associated confidence level based on their own expertise. In particular, the teacher performs teaching using an empirical risk minimizer learned from an imperfect labeled set; the workers are assumed to have a forgetting behavior during learning and their learning rate depends on the interpretation difficulty of the teaching item. Furthermore, depending on the level of confidence when the workers perform labeling, we also show that the empirical risk minimizer used by the teacher is a reliable and realistic substitute of the unknown target concept by utilizing the unbiased surrogate loss. Finally, the performance of the proposed framework is demonstrated through experiments on multiple real-world image and text data sets.

Original languageEnglish (US)
Title of host publicationThe Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020
PublisherAssociation for Computing Machinery, Inc
Pages110-121
Number of pages12
ISBN (Electronic)9781450370233
DOIs
StatePublished - Apr 20 2020
Event29th International World Wide Web Conference, WWW 2020 - Taipei, Taiwan, Province of China
Duration: Apr 20 2020Apr 24 2020

Publication series

NameThe Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020

Conference

Conference29th International World Wide Web Conference, WWW 2020
CountryTaiwan, Province of China
CityTaipei
Period4/20/204/24/20

Keywords

  • Explanation.
  • Interactive teaching
  • Personalized crowdsourcing

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Fingerprint Dive into the research topics of 'Crowd Teaching with Imperfect Labels'. Together they form a unique fingerprint.

  • Cite this

    Zhou, Y., Nelakurthi, A. R., MacIejewski, R., Fan, W., & He, J. (2020). Crowd Teaching with Imperfect Labels. In The Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020 (pp. 110-121). (The Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020). Association for Computing Machinery, Inc. https://doi.org/10.1145/3366423.3380099