The Centre for the Study of Democratic Citizenship, presents:
Kyle Peyton (Yale University)
The Generalizability of Online Experiments Conducted During The COVID-19 Pandemic
You can learn more about Kyle Peyton by clicking here.
Where and When: Thursday, October 8, at 1pm on Zoom.
RSVP: please contact Costin Ciobanu at costin.ciobanu@mail.mcgill.ca
Abstract:Significant disruptions caused by the COVID-19 pandemic led many social scientists toward online survey experimentation for empirical research. Generalizing from the experiments conducted during a period of persistent crisis may be challenging due to changes in who participates in online survey research and how the participants respond to treatments. We investigate the generalizability of COVID-era survey experiments with 32 replication studies fielded across 13 surveys on the Lucid survey platform between March and July of 2020. We find strong evidence that these experiments replicate in terms of sign and significance, but at somewhat reduced magnitudes. In some cases, we implemented COVID- specific versions of framing treatments that appeared to generate equivalent psychological responses to the standard versions. These findings mitigate concerns about the temporal validity of online research during this period. The pandemic does not appear to have changed how subjects respond to treatments, provided that they pay close enough attention to the treatment information and the outcome questions. In this light, we offer some suggestions for renewed care in the design, analysis, and interpretation of experiments conducted during the COVID-19 pandemic.
See more Montreal Methods Workshops events here.
