How accurate are subjective judgements of a continuum?
OBJECTIVE
To assess the accuracy of subjective judgements of a uni-dimensional continuum.


RESULTS
Twenty-four observers were asked to estimate the position of a dot placed between two lines using a 100-point scale. Each observer judged 20 different positions. Ninety-five percent of observations were between -7 and +9 of the true value. The median and the mean value of the estimates for each position were within one or two points of the true value in most cases with a maximum deviation of five. There was perfect correlation between the ranks of the median and the mean and the true value (correlation coefficient +1, P < 0.00001). The kappa values for interobserver agreement varied from 0.98 for two categories to 0.55 for 10 categories and 0.39 for 20 categories. Information transmitted increased from 0.98 bits with two categories to 2.83 bits with 20 categories.


CONCLUSIONS
The experiment indicates that subjective judgements can be accurate but it is necessary to use a large number of categories. One hundred-point scales allow confidence intervals to be calculated and the intervals are reduced if the results from several observers are pooled. The mean and the median of the judgements of a large number of observers are close to objective reality and this approach could be used to define a gold standard of rankings of pathological continua such as cervical dyskaryosis or tumour differentiation. There is a reciprocal relationship between kappa values and information transmitted which demonstrates the fallacy of using kappa statistics in studies of continua.
Deolekar
M
M
Department of Pathology, Royal Lancaster Infirmary, Lancaster, UK.
Morris
J A
JA
D016001:Confidence Intervals; D006801:Humans; D007257:Information Theory; D007600:Judgment / Q000145:classification*; D015588:Observer Variation; D010336:Pathology / Q000379:methods* / Q000592:standards / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D018401:Sample Size; D015983:Selection Bias