Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.
OBJECTIVE
This study assessed the impact of systematic review and data extraction experience on the accuracy and efficiency of data extraction in systematic reviews.


METHODS
We conducted a prospective cross-sectional study from October to December 2006. Participants were classified as having minimal, moderate, or substantial experience in systematic reviews and data extraction. Three studies on insomnia treatment were extracted. Our primary outcome was the accuracy of data extraction. Data sets of each experience level were analyzed for errors in data extraction and results of meta-analyses. Additionally, the time required for completion of data extraction was compared.


RESULTS
Error rates were similar across the various levels of experience and ranged from 28.3% to 31.2%. Mean rates for errors of omission (11.3-16.4%) were generally lower than those for errors of inaccuracy (13.9-17.9%). There were no significant differences in error rates or accuracy of meta-analysis results between groups. Time required approached significance, with minimally experienced participants requiring the most time.


CONCLUSIONS
Overall, there were high error rates by participants at all experience levels; however, time required for extraction tended to decrease with experience. These results illustrate the need to develop strategies aimed at mastery of data extraction, rather than reliance on previous data extraction experience alone.
Horton
Jennifer
J
Department of Pediatrics, University of Alberta/Capital Health Evidence-Based Practice Centre, Edmonton Alberta, Canada. jhorton@ualberta.ca
Vandermeer
Ben
B
Hartling
Lisa
L
Tjosvold
Lisa
L
Klassen
Terry P
TP
Buscemi
Nina
N
D006801:Humans; D004812:Epidemiologic Methods; D019317:Evidence-Based Medicine / Q000379:methods / Q000592:standards; D011361:Professional Competence; D012015:Reference Standards; D007319:Sleep Initiation and Maintenance Disorders / Q000628:therapy; D013997:Time Factors; D000078202:Systematic Reviews as Topic