CrowdRMLE: An Efficient Approach to Analyze Crowdsourcing Datasets in Image and Video Quality Assessment

In image and video quality assessment, standardized approaches to analyze subjectively annotated datasets typically consider subject bias and inconsistency. While effective, these methods fail to capture the full complexity of how subjects interpret and use each opinion score on the quality scale. A recently proposed method, Regularized Maximum Likelihood Estimation (RMLE), partially overcomes this limitation by examining how a subject interacts with each opinion score. However, RMLE's high computational cost makes it computationally impractical for large-scale datasets, such as those derived from crowdsourcing experiments. This computational burden stems from solving a nonlinear optimization problem whose complexity grows with the dataset size. In this paper we propose CrowdRMLE, which formulates a different optimization problem and derives an analytical solution, thereby significantly reducing computational cost. Numerical experiments show that CrowdRMLE is not only far more efficient than RMLE but is also more accurate and robust to noise. Using CrowdRMLE we analyze multiple datasets, providing important insights into aspects that need improvement in the design of future methodologies for crowdsourcing tests for image and video quality assessment.