We investigate crowdsourcing algorithms for finding the top-quality item within a large collection of objects with unknown intrinsic quality values. This is an important problem with many relevant applications, such as in networked recommendation systems. The core of the algorithms is that objects are distributed to crowd workers, who return a noisy and biased evaluation. All received evaluations are then combined to identify the top-quality object. We first present a simple probabilistic model for the system under investigation. Then we devise and study a class of efficient adaptive algorithms to assign in an effective way objects to workers. We compare the performance of several algorithms, which correspond to different choices of the design parameters/metrics. In the simulations, we show that some of the algorithms achieve near optimal performance for a suitable setting of the system parameters.
Selecting the Top-Quality Item Through Crowd Scoring / Nordio, Alessandro; Tarable, Alberto; Leonardi, Emilio; AJMONE MARSAN, Marco Giuseppe. - In: ACM TRANSACTIONS ON MODELING AND PERFORMANCE EVALUATION OF COMPUTING SYSTEMS. - ISSN 2376-3639. - ELETTRONICO. - 3:1(2018), pp. 1-26. [10.1145/3157736]
Selecting the Top-Quality Item Through Crowd Scoring
Emilio Leonardi;Marco Ajmone Marsan
2018
Abstract
We investigate crowdsourcing algorithms for finding the top-quality item within a large collection of objects with unknown intrinsic quality values. This is an important problem with many relevant applications, such as in networked recommendation systems. The core of the algorithms is that objects are distributed to crowd workers, who return a noisy and biased evaluation. All received evaluations are then combined to identify the top-quality object. We first present a simple probabilistic model for the system under investigation. Then we devise and study a class of efficient adaptive algorithms to assign in an effective way objects to workers. We compare the performance of several algorithms, which correspond to different choices of the design parameters/metrics. In the simulations, we show that some of the algorithms achieve near optimal performance for a suitable setting of the system parameters.Pubblicazioni consigliate
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/11583/2707363
Attenzione
Attenzione! I dati visualizzati non sono stati sottoposti a validazione da parte dell'ateneo