The Welfare Quality (WQ) protocol for on-farm dairy cattle welfare assessment describes 27 measures and a stepwise method for integrating values for these measures into 11 criteria scores, grouped further into 4 principle scores and finally into an overall welfare categorization with 4 levels. We conducted an online survey to examine whether trained users' opinions of the WQ protocol for dairy cattle correspond with the integrated scores (criteria, principles, and overall categorization) calculated according to the WQ protocol. First, the trained users' scores (n = 8–15) for reliability and validity and their ranking of the importance of all measures for herd welfare were compared with the degree of actual effect of these measures on the WQ integrated scores. Logistic regression was applied to identify the measures that affected the WQ overall welfare categorization into the “not classified” or “enhanced” categories for a database of 491 European herds. The smallest multivariate model maintaining the highest percentage of both sensitivity and specificity for the “enhanced” category contained 6 measures, whereas the model for “not classified” contained 4 measures. Some of the measures that were ranked as least important by trained users (e.g., measures relating to drinkers) had the highest influence on the WQ overall welfare categorization. Conversely, measures rated as most important by the trained users (e.g., lameness and mortality) had a lower effect on the WQ overall category. In addition, trained users were asked to allocate criterion and overall welfare scores to 7 focal herds selected from the database (n = 491 herds). Data on all WQ measures for these focal herds relative to all other herds in the database were provided. The degree to which expert scores corresponded to each other, the systematic difference, and the correspondence between median trained-user opinion and the WQ criterion scores were then tested. The level of correspondence between expert scoring and WQ scoring for 6 of the 12 criteria and for the overall welfare score was low. The WQ scores of the protocol for dairy cattle thus lacked correspondence with trained users on the importance of several welfare measures.