Welfare Quality® proposes a system for aggregation according to which the total welfare score for a group of animals is a non-linear effect of the prevalence of welfare scores across the individuals within the group. Three assumptions serve to justify this: (1) experts do not follow a linear reasoning when they assess a welfare problem; (2) it serves to prevent compensation (severe welfare problems hidden by scoring well on other aspects of welfare); (3) experts agree on the weight of different welfare measures. We use two sources of data to examine these assumptions: animal welfare data from 44 Danish dairy farms with Danish Holstein Friesian cows, and data from a questionnaire study with a convenience sample of 307 experts in animal welfare, of which we received responses from over 50%. Our main results were: (1) the total group-level welfare score as assigned by experts is a non-linear function of the individual animal welfare states within the group; (2) the WQ system does not prevent what experts perceive as unacceptable compensation; (3) the level of agreement among experts appears to vary across measures. Our findings give rise to concerns about the proposed aggregation system offered by WQ.