A Scoring Model Considering the Variability of Subjects' Characteristics in Subjective Experiments

Many authors argued that the scoring behavior of a subject in a subjective quality evaluation experiment can be modeled by two main characteristics, i.e., the subject’s bias and the subject’s inconsistency. However, for simplicity’s sake, they disregarded the fact that subjects are usually less inconsistent when evaluating stimuli with very low or very high quality. This work addresses this shortcoming by providing an analytical formulation about how to link subjects’ bias and inconsistency to the ground truth subjective quality of the stimulus under evaluation. By integrating this formulation into a state-of-the-art subject scoring model we obtain a more realistic model to recover the ground truth subjective quality of each stimulus. An iterative algorithm able to estimate the model parameters is also provided. Computational experiments show that our proposed model yields more realistic confidence intervals for the recovered ground truth subjective quality values and exhibits more robustness to synthetically added noise in several testing conditions.