Statistical methods for the validation of questionnaires--discrepancy between theory and practice.
OBJECTIVE
Questionnaires used in epidemiological studies should be validated. However, unclarity exists about the appropriate statistical methods and interpretation of validation studies. Thus, we investigated the theory and practice of statistical evaluation approaches.


METHODS
Using three platforms, a literature review, own simulations, and a validation study performed by ourselves, we worked out relevant limitations, advantages, and new important aspects of evaluation methods.


RESULTS
Our systematic literature review, based on physical activity questionnaires, revealed that correlation coefficients are still the common approach in validation studies, found in 41 of 46 reviewed publications (89.1%). This practice has been criticized in the theoretically oriented literature for more than 20 years. Appropriate evaluation methods as recommended by Bland and Altman were found in only ten publications (21.7%). We showed that serious bias in questionnaires can be revealed by Bland-Altman plots but may remain undetected by correlation coefficients. With our simulations we refuted the argument that correlation coefficients properly investigate whether a questionnaire ranks the subjects sufficiently well. Further, with Bland-Altman analyses we could evaluate differential errors with respect to case-control status in our validation study. Yet, this was not possible with correlation coefficients, because they generally do not identify systematic bias. In addition, we show a potential pitfall in the interpretation of Bland-Altman plots that might occur in specific rare instances.


CONCLUSIONS
The commonly used correlation approach can yield misleading conclusions in validation studies. A more frequent and proper use of the Bland-Altman methods would be desirable to improve epidemiological data quality.
Schmidt
M E
ME
Unit of Environmental Epidemiology, German Cancer Research Center, Im Neuenheimer Feld 280, 69120 Heidelberg, Germany.
Steindorf
K
K
D003627:Data Interpretation, Statistical; D016021:Epidemiologic Studies; D006801:Humans; D015233:Models, Statistical; D015203:Reproducibility of Results; D013223:Statistics as Topic; D011795:Surveys and Questionnaires / Q000592:standards*