Fundamental Limits of Budget-Fidelity Trade-off in Label Crowdsourcing
- Creators
- Lahouti, Farshad
- Hassibi, Babak
Abstract
Digital crowdsourcing (CS) is a modern approach to perform certain large projects using small contributions of a large crowd. In CS, a taskmaster typically breaks down the project into small batches of tasks and assigns them to so-called workers with imperfect skill levels. The crowdsourcer then collects and analyzes the results for inference and serving the purpose of the project. In this work, the CS problem, as a human-in-the-loop computation problem, is modeled and analyzed in an information theoretic rate-distortion framework. The purpose is to identify the ultimate fidelity that one can achieve by any form of query from the crowd and any decoding (inference) algorithm with a given budget. The results are established by a joint source channel (de)coding scheme, which represent the query scheme and inference, over parallel noisy channels, which model workers with imperfect skill levels. We also present and analyze a query scheme dubbed k-ary incidence coding and study optimized query pricing in this setting.
Additional Information
© 2005 Neural Information Processing Systems Foundation, Inc.Attached Files
Published - NIPS-2016-fundamental-limits-of-budget-fidelity-trade-off-in-label-crowdsourcing-Paper.pdf
Submitted - 1608.07328.pdf
Supplemental Material - NIPS-2016-fundamental-limits-of-budget-fidelity-trade-off-in-label-crowdsourcing-Supplemental.zip
Files
Additional details
- Eprint ID
- 107319
- Resolver ID
- CaltechAUTHORS:20210105-133359837
- Created
-
2021-01-05Created from EPrint's datestamp field
- Updated
-
2023-06-02Created from EPrint's last_modified field
- Series Name
- Advances in Neural Information Processing Systems
- Series Volume or Issue Number
- 29