In the first quarter of 2021 we explore design steps, starting with a January focus on research questions. We’ll continue to learn about the design stage in February with a focus on Choosing Methodology and Methods.
This month we are looking at issues associated with selecting the methodology best suited for the study. Throughout the month we’ll highlight various methodologies, link to multidisciplinary open access articles that explain and/or apply them, and point to relevant texts. In this post we look at the use of experiments in quantitative studies. In a future post we will explore the use of experiments in qualitative or mixed methods research.
What is experimental research design?
Here is a brief overview from the SAGE Encyclopedia of Survey Research Methods:
Experimental design is one of several forms of scientific inquiry employed to identify the cause-and-effect relation between two or more variables and to assess the magnitude of the effect(s) produced. The independent variable is the experiment or treatment applied (e.g. a social policy measure, an educational reform, different incentive amounts and types) and the dependent variable is the condition (e.g. attitude, behavior) presumed to be influenced by the treatment. In the course of the experiment it is necessary to demonstrate the existence of covariation between variables, its nonspuriousness, and to show that the cause occurred before the effect. This sort of inquiry can take the form of an artificial experiment, carried out in a laboratory scenario, or a natural experiment implemented in a real-life context, where the level of control is lower. For both cases, the literature presents several taxonomies, from which four main types are considered: (1) true or classical experimental, (2) pre-experimental, single-subject experimental, and (3) quasi-experimental. In addition, there are a number of variations of the classic experimental design as well as of the quasi-experimental design.
In a true or classic experimental design, there are at least two groups of individuals or units of analysis: the experiment group and the control group. Participants are randomly assigned to both groups. These two groups are identical except that one of them is exposed to the experiment or causal agent, and the other, the control group, is not. In many instances, a pretest and a posttest are administered to all individuals in the two groups; but the pretest is not a necessary aspect of the true experiment. If there is a significant difference between members of the two groups, it is inferred that there is a cause-and-effect link between that treatment and the outcome.
The pre-experimental design does not have a control group to be compared with the experiment group. There is a pretest and a posttest applied to the same participants. In a single-subject experimental design, there is only one participant, or a small number, that is analyzed over a period of time. In quasi-experimental designs, participants come from naturally assembled or pre-determined groups (e.g. a family, a school class, a professional category, or inhabitants of a neighborhood) and are not therefore assigned randomly to the control and treatment groups.
An experimental design has to fulfill several conditions. The variables must be measured with accuracy and precision, and the statistical test must be denned before starting the experiment. If necessary, it must be possible to repeat the experiment in order to confirm that the outcome is statistically significant and that no other factors, other than the independent variable) the researcher manipulates, are responsible for the outcome. In practice, however, not every experiment meets all these conditions. Internal and external validity can be affected by several factors. For example, internal validity (cause-effect relation) can be influenced by the length of the time between the pretest and the posttest, by changes in the measurement instruments, the influence of the pretest on subsequent behavior, and so on, but can be improved with matching procedures and by randomization. External validity can be enhanced by the use of a representative sample and by avoiding artificial experiment settings lacking mundane realism.
Lavrakas, P. J. (2008). Encyclopedia of survey research methods (Vols. 1-0). Thousand Oaks, CA: Sage Publications, Inc. doi: 10.4135/9781412963947
Open Access Experimental Studies
Dai, Y., & Luqiu, L. (2020). Camouflaged propaganda: A survey experiment on political native advertising. Research & Politics. https://doi.org/10.1177/2053168020935250
We examine a new form of propaganda, political native advertising, in which political actors, including foreign governments, buy space in independent media outlets to publish advertisements that are camouflaged as standard news stories. Those who engage in this form of propaganda hope to exploit the higher credibility of the hosting media site to enhance the persuasiveness of their message. Despite the obvious political implications and ethical issues at stake, political native advertising has received almost no scholarly attention. Our article begins to redress this imbalance. Using an online survey experiment with real political native advertisements in the Washington Post and The Telegraph bought by the Chinese government, we provide some of the first empirical evidence on basic but important features of political native advertising. We find, among other things, that respondents struggle to distinguish political advertisements from standard news stories regardless of their level of education and media literacy, that political advertisements are more convincing if they appear on and are perceived as news from an independent hosting media site than in a government-controlled news outlet, and that trust in the hosting media site declines if the political advertisement is detected.
Gavriilidou, A., Wierbos, M. J., Daamen, W., Yuan, Y., Knoop, V. L., & Hoogendoorn, S. P. (2019). Large-Scale Bicycle Flow Experiment: Setup and Implementation. Transportation Research Record, 2673(5), 709–719. https://doi.org/10.1177/0361198119839974
Abstract. Cycling research at the operational behavioral level is limited, mainly because of the lack of empirical data. To overcome this data shortage, we performed a controlled, large-scale cycling experiment in the Netherlands. In this paper we describe the methodology for setting up and implementing such an experiment, from the motivation of its design using a conceptual model describing cyclist behavior to adjustments that were required during the experiment. The main contribution of this paper is, therefore, to be used as a guide in future experimental data collections. Moreover, we present the characteristics of the participants and their bicycles, and provide a qualitative description of phenomena observed during the experiment. Finally, we elaborate on the potential that the collected dataset holds for future research into understanding and modeling operational cycling behavior.
Kapelner, A., & Weinberg, D. (2019). Do Readers Judge Books by Author Gender? Results from a Randomized Experiment. Socius. https://doi.org/10.1177/2378023119868893
Abstract. We run a randomized experiment to examine gender discrimination in book purchasing with 2,544 subjects on Amazon’s Mechanical Turk. We manipulate author gender and book genre in a factorial design to study consumer preferences for male versus female versus androgynous authorship. Despite previous findings in the literature showing gender discrimination in book publishing and in evaluations of work, respondents expressed no gender preference across a variety of measures, including quality, interest, and the amount they were willing to pay to purchase the book. This nonfinding, if it holds up to additional research, suggests that book consumers may not express the same discriminatory tendencies observed among indie and traditional publishers.
Madsen, A. K., & Munk, A. K. (2019). Experiments with a data-public: Moving digital methods into critical proximity with political practice. Big Data & Society. https://doi.org/10.1177/2053951718825357
Abstract. Making publics visible through digital traces has recently generated interest by practitioners of public engagement and scholars within the field of digital methods. This paper presents an experiment in moving such methods into critical proximity with political practice and discusses how digital visualizations of topical debates become appropriated by actors and hardwired into existing ecologies of publics and politics. Through an experiment in rendering a specific data-public visible, it shows how the interplay between diverse conceptions of the public as well as the specific platforms and data invoked, resulted in a situated affordance-space that allowed specific renderings take shape, while disadvantaging others. Furthermore, it argues that several accepted tropes in the literatures of digital methods ended up being problematic guidelines in this space. Among these is the prescription to shown heterogeneity by pushing back at established media logics.
Swanson, E., Kopotic, K., Zamarro, G., Mills, J. N., Greene, J. P., & W. Ritter, G. (2021). An Evaluation of the Educational Impact of College Campus Visits: A Randomized Experiment. AERA Open. https://doi.org/10.1177/2332858421989707
Abstract. We study whether visits to a college campus during eighth grade affect students’ interest in and preparation for college. Two cohorts of eighth graders were randomized within schools to a control condition, in which they received a college informational packet, or a treatment condition, in which they received the same information and visited a flagship university three times during an academic year. We estimate the effect of the visits on students’ college knowledge, postsecondary intentions, college preparatory behaviors, academic engagement, and ninth-grade course enrollment. Treated students exhibit higher levels of college knowledge, efficacy, and grit, as well as a higher likelihood of conversing with school personnel about college. Additionally, treated students are more likely to enroll in advanced science/social science courses. We find mixed evidence on whether the visits increased students’ diligence on classroom tasks and a negative impact on students’ desire to attend technical school.
van Gerwen, N., Buskens, V., & van der Lippe, T. (2018). Individual training and employees’ cooperative behavior: Evidence from a contextualized laboratory experiment. Rationality and Society, 30(4), 432–462. https://doi.org/10.1177/1043463118771428
Abstract. Employers are constantly seeking to improve employee performance by means of investing in employee training. The results of training are to a large extent dependent on employees’ willingness to behave productively in a cooperative manner. Yet, systematic evidence investigating the causal relation between training and employees’ cooperative behavior is rare. Here, we present results from a contextualized laboratory experiment in which subjects, who differ in terms of training participation, were asked to contribute resources to a team effort. We conclude that training promotes cooperative behavior, that is, voluntary contributions made to the team effort, in teams of employees working together for short periods of time. Training enhances cooperative behavior the most when provided to the higher skilled subjects. We also find that members in more stable teams act very cooperatively under all conditions, but their contributions do not increase further with training.
Read more about Experimental Research Design
by Ted Palys and Chris Atchison