TY - GEN
T1 - Optimizing the wisdom of the crowd
T2 - 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2019
AU - Zhou, Yao
AU - Ma, Fenglong
AU - Gao, Jing
AU - He, Jingrui
N1 - Funding Information:
One category of teaching framework assumes crowdsourcing workers are global learners and their learned concepts are randomly switched in the hypotheses space. The second category of frame-work assumes that the workers are sequential learners who see example one at a time and make gradual progress towards becoming the experts of annotation. ACKNOWLEDGEMENT This work is supported by the National Science Foundation under Grant No. IIS-1552654, Grant No. IIS-1813464, IIS-1553411 and Grant No. CNS-1629888, the U.S. Department of Homeland Security under Grant Award Number 17STQAC00001-02-00, and an IBM Faculty Award. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government. REFERENCES
Publisher Copyright:
© 2019 Copyright held by the owner/author(s).
PY - 2019/7/25
Y1 - 2019/7/25
N2 - The increasing need for labeled data has brought the booming growth of crowdsourcing in a wide range of high-impact real-world applications, such as collaborative knowledge (e.g., data annotations, language translations), collective creativity (e.g., analogy mining, crowdfunding), and reverse Turing test (e.g., CAPTCHA-like systems), etc. In the context of supervised learning, crowdsourcing refers to the annotation procedure where the data items are outsourced and processed by a group of mostly unskilled online workers. Thus, the researchers or the organizations are able to collect large amount of information via the feedback of the crowd in a short time with a low cost. Despite the wide adoption of crowdsourcing, several of its fundamental problems remain unsolved especially at the information and cognitive levels with respect to incentive design, information aggregation, and heterogeneous learning. This tutorial aims to: (1) provide a comprehensive review of recent advances in exploring the power of crowdsourcing from the perspective of optimizing the wisdom of the crowd; and (2) identify the open challenges and provide insights to the future trends in the context of human-in-the-loop learning. We believe this is an emerging and potentially high-impact topic in computational data science, which will attract both researchers and practitioners from academia and industry.
AB - The increasing need for labeled data has brought the booming growth of crowdsourcing in a wide range of high-impact real-world applications, such as collaborative knowledge (e.g., data annotations, language translations), collective creativity (e.g., analogy mining, crowdfunding), and reverse Turing test (e.g., CAPTCHA-like systems), etc. In the context of supervised learning, crowdsourcing refers to the annotation procedure where the data items are outsourced and processed by a group of mostly unskilled online workers. Thus, the researchers or the organizations are able to collect large amount of information via the feedback of the crowd in a short time with a low cost. Despite the wide adoption of crowdsourcing, several of its fundamental problems remain unsolved especially at the information and cognitive levels with respect to incentive design, information aggregation, and heterogeneous learning. This tutorial aims to: (1) provide a comprehensive review of recent advances in exploring the power of crowdsourcing from the perspective of optimizing the wisdom of the crowd; and (2) identify the open challenges and provide insights to the future trends in the context of human-in-the-loop learning. We believe this is an emerging and potentially high-impact topic in computational data science, which will attract both researchers and practitioners from academia and industry.
UR - http://www.scopus.com/inward/record.url?scp=85071187761&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85071187761&partnerID=8YFLogxK
U2 - 10.1145/3292500.3332277
DO - 10.1145/3292500.3332277
M3 - Conference contribution
AN - SCOPUS:85071187761
T3 - Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
SP - 3231
EP - 3232
BT - KDD 2019 - Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
PB - Association for Computing Machinery
Y2 - 4 August 2019 through 8 August 2019
ER -