Journal Article

Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results

Annie Franco, Neil Malhotra and Gabor Simonovits

in Political Analysis

Published on behalf of Society for Political Methodology

Volume 23, issue 2, pages 306-312
Published in print April 2015 | ISSN: 1047-1987
Published online March 2015 | e-ISSN: 1476-4989 | DOI: https://dx.doi.org/10.1093/pan/mpv006
Underreporting in Political Science Survey Experiments: Comparing Questionnaires to Published Results

Show Summary Details

Preview

The accuracy of published findings is compromised when researchers fail to report and adjust for multiple testing. Preregistration of studies and the requirement of preanalysis plans for publication are two proposed solutions to combat this problem. Some have raised concerns that such changes in research practice may hinder inductive learning. However, without knowing the extent of underreporting, it is difficult to assess the costs and benefits of institutional reforms. This paper examines published survey experiments conducted as part of the Time-sharing Experiments in the Social Sciences program, where the questionnaires are made publicly available, allowing us to compare planned design features against what is reported in published research. We find that: (1) 30% of papers report fewer experimental conditions in the published paper than in the questionnaire; (2) roughly 60% of papers report fewer outcome variables than what are listed in the questionnaire; and (3) about 80% of papers fail to report all experimental conditions and outcomes. These findings suggest that published statistical tests understate the probability of type I errors.

Journal Article.  3264 words.  Illustrated.

Subjects: Politics

Full text: subscription required

How to subscribe Recommend to my Librarian

Users without a subscription are not able to see the full content. Please, subscribe or login to access all content. subscribe or login to access all content.