Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies.
The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light-especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or 'spin') in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government's commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform's champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended.
London School of Economics and Political Science, London, United Kingdom.
Arnaud
http://orcid.org/0000-0001-5584-9050
A
Vaganay
Acad Emerg Med. 2000 Jun;7(6):691-4
10905651
Acad Med. 2001 Sep;76(9):889-96
11553504
J Eval Clin Pract. 2002 Aug;8(3):353-9
12164983
BMJ. 2003 Jun 28;326(7404):1453-5
12829562
JAMA. 2004 May 26;291(20):2457-65
15161896
Ann Intern Med. 1992 Jan 1;116(1):78-84
1530753
Respir Care. 2004 Oct;49(10):1246-52
15447812
BMJ. 2005 Apr 2;330(7494):753
15681569
Ann Intern Med. 1992 Jun 1;116(11):958
1580460
Lancet. 2005 May 7-13;365(9471):1657-61
15885299
Qual Health Res. 2005 Nov;15(9):1277-88
16204405
Trials. 2007 Mar 06;8:9
17341316
BMJ. 2007 Jul 14;335(7610):96-7
17626964
BMJ. 2007 Oct 20;335(7624):806-8
17947786
Med Law. 2007 Sep;26(3):511-25
17970249
N Engl J Med. 2007 Nov 22;357(21):2189-94
18032770
BMJ. 2008 Jan 5;336(7634):23-5
18174597
Am Psychol. 2008 Apr;63(3):160-8
18377106
PLoS One. 2008 Aug 28;3(8):e3081
18769481
PLoS Med. 2008 Nov 25;5(11):e217; discussion e217
19067477
PLoS Med. 2008 Nov 25;5(11):e230
19067481
J Clin Epidemiol. 2009 Oct;62(10):1006-12
19631508
JAMA. 2009 Sep 2;302(9):977-84
19724045
BJU Int. 2010 Jan;105(2):162-4
19912181
BMJ. 2010 Mar 23;340:c869
20332511
BMJ. 2010 Mar 30;340:c117
20354011
Trials. 2010 Apr 13;11:37
20388211
Trials. 2010 May 12;11:52
20462436
J Nurs Scholarsh. 2010 Mar;42(1):58-65
20487187
JAMA. 2010 May 26;303(20):2058-64
20501928
Ann Intern Med. 2010 Aug 3;153(3):158-66
20679560
J Womens Health (Larchmt). 2010 Oct;19(10):1919-23
20831430
BMJ. 2011 Jan 06;342:c7153
21212122
J Clin Epidemiol. 2011 Jul;64(7):718-42
21216130
J Epidemiol Community Health. 2012 Apr;66(4):286-9
21652522
J Clin Epidemiol. 2011 Dec;64(12):1317-24
21889307
Trials. 2011 Dec 22;12:264
22192231
J Invest Dermatol. 2012 Dec;132(12):2727-34
22832489
PLoS Med. 2012;9(9):e1001308
22984354
Int J Risk Saf Med. 2012;24(4):233-42
23135338
Ann Surg. 2013 May;257(5):818-23
23407296
Intensive Care Med. 2013 Aug;39(8):1386-95
23743522
PLoS One. 2013 Jul 05;8(7):e66844
23861749
PLoS One. 2013 Jul 08;8(7):e69462
23861972
Ann Intensive Care. 2013 Sep 02;3(1):29
24004521
Biostatistics. 2014 Apr;15(2):370-83
24215031
PLoS Med. 2013 Dec;10(12):e1001578; dicsussion e1001578
24391479
Trends Cogn Sci. 2014 May;18(5):235-41
24656991
J Clin Pharm Ther. 2014 Oct;39(5):501-6
24828874
PLoS One. 2014 Jul 03;9(7):e101533
24992571
Cochrane Database Syst Rev. 2014 Oct 01;(10):MR000035
25271098
J Clin Oncol. 2014 Dec 20;32(36):4120-6
25403215
Q J Exp Psychol (Hove). 2015;68(4):829-32
25484109
Trials. 2015 Jan 27;16:28
25626862
PeerJ. 2015 Jan 22;3:e733
25650272
JAMA Psychiatry. 2015 May;72(5):500-10
25806940
Syst Rev. 2015 Jun 13;4:85
26071043
Eur J Clin Invest. 2015 Sep;45(9):895-8
26108701
BMC Med Ethics. 2015 Jul 25;16:51
26206479
BMC Med Res Methodol. 2015 Oct 13;15:85
26462565
J Natl Cancer Inst. 1996 Feb 21;88(3-4):206-7
8632495