Quasi-Experimental Designs for Causal Inference in Social Sciences

Quasi-experimental designs are employed to evaluate causal treatment effects when randomized experiments are not feasible (Kim & Steiner, 2016). They are particularly useful in social sciences where controlling all variables is often impossible (Kim & Steiner, 2016). These designs attempt to establish a causal relationship between an intervention and an outcome by using naturally occurring groups or conditions rather than randomly assigning participants (Kim & Steiner, 2016).

Core Principles

Quasi-experimental designs aim to approximate the conditions of a randomized controlled trial (RCT) as closely as possible without the random assignment of participants (Kim & Steiner, 2016). The goal is to infer causality by controlling for as many confounding variables as possible and carefully considering potential threats to validity (Kim & Steiner, 2016).

Key Characteristics

• Lack of Random Assignment: Participants are not randomly assigned to treatment and control groups (Kim & Steiner, 2016).
• Use of Existing Groups: Researchers often work with pre-existing groups or naturally occurring conditions (Kim & Steiner, 2016).
• Control for Confounding Variables: Statistical techniques and design elements are used to control for variables that could influence the outcome (Kim & Steiner, 2016).
• Emphasis on Validity: Careful consideration is given to internal and external validity to ensure the credibility of causal inferences (Kim & Steiner, 2016).

Types of Quasi-Experimental Designs

Several types of quasi-experimental designs are commonly used in social sciences, each with its own strengths and weaknesses (Kim & Steiner, 2016).

Regression Discontinuity Design (RDD)

RDD is used when treatment assignment is based on a cutoff score on a continuous variable (Kim & Steiner, 2016). Participants above the cutoff receive the treatment, while those below do not. The causal effect is estimated by comparing the outcomes of participants just above and below the cutoff (Kim & Steiner, 2016).

• Example: Evaluating the effect of a scholarship program on academic performance, where students with a GPA above a certain threshold receive the scholarship (Kim & Steiner, 2016).
• Assumptions: The relationship between the assignment variable and the outcome is continuous, except at the cutoff point (Kim & Steiner, 2016).

Instrumental Variable (IV) Design

IV design uses a third variable (the instrument) to estimate the causal effect of a treatment on an outcome (Kim & Steiner, 2016). The instrument must be correlated with the treatment but not directly related to the outcome, except through its effect on the treatment (Kim & Steiner, 2016).

• Example: Using proximity to a college as an instrument to estimate the effect of college attendance on earnings (Kim & Steiner, 2016).
• Assumptions: The instrument is relevant (correlated with the treatment), exogenous (not correlated with the outcome except through the treatment), and satisfies the exclusion restriction (only affects the outcome through the treatment) (Kim & Steiner, 2016).

Matching and Propensity Score (PS) Designs

Matching and PS designs are used to create comparable treatment and control groups based on observed covariates (Kim & Steiner, 2016). Propensity score matching involves estimating the probability of treatment assignment (the propensity score) based on observed covariates and then matching treatment and control participants with similar propensity scores (Kim & Steiner, 2016).

• Example: Evaluating the effect of a summer science camp on students' science achievement by matching attendees and non-attendees based on their pre-test scores, liking of science, and socioeconomic status (Kim & Steiner, 2016).
• Assumptions: All relevant confounding variables are observed and included in the matching or propensity score model (unconfoundedness or ignorability) (Kim & Steiner, 2016).

Comparative Interrupted Time Series (CITS) Design

CITS design involves comparing the changes in an outcome variable over time between a treatment group and a control group, with an interruption (intervention) occurring at a specific point in time for the treatment group (Kim & Steiner, 2016).

• Example: Evaluating the effect of a new education policy on student test scores by comparing the trends in test scores before and after the policy implementation in schools that adopted the policy versus schools that did not (Kim & Steiner, 2016).
• Assumptions: The treatment and control groups would have followed similar trends in the absence of the intervention; there are no other simultaneous interventions or events that could explain the observed changes (Kim & Steiner, 2016).

Validity Threats and Mitigation Strategies

Quasi-experimental designs are more susceptible to validity threats than RCTs due to the lack of random assignment (Kim & Steiner, 2016). Researchers must carefully consider and address these threats to strengthen causal inferences (Kim & Steiner, 2016).

Internal Validity Threats

Internal validity refers to the extent to which the observed effect can be attributed to the treatment rather than other factors (Kim & Steiner, 2016).

• Selection Bias: Differences between the treatment and control groups at baseline that could affect the outcome (Kim & Steiner, 2016).
  • Mitigation: Matching, propensity score methods, statistical controls (Kim & Steiner, 2016).
• History: Events occurring during the study that could affect the outcome (Kim & Steiner, 2016).
  • Mitigation: Control groups, careful monitoring of external events (Kim & Steiner, 2016).
• Maturation: Natural changes in participants over time that could affect the outcome (Kim & Steiner, 2016).
  • Mitigation: Control groups, shorter study duration (Kim & Steiner, 2016).
• Testing: The effect of taking a pre-test on subsequent test performance (Kim & Steiner, 2016).
  • Mitigation: Solomon four-group design, alternative assessment methods (Kim & Steiner, 2016).
• Instrumentation: Changes in the measurement instrument or procedures over time (Kim & Steiner, 2016).
  • Mitigation: Standardized protocols, rater training (Kim & Steiner, 2016).
• Mortality: Differential attrition between the treatment and control groups (Kim & Steiner, 2016).
  • Mitigation: Statistical methods to account for missing data, sensitivity analyses (Kim & Steiner, 2016).

External Validity Threats

External validity refers to the extent to which the results can be generalized to other populations, settings, and times (Kim & Steiner, 2016).

• Interaction of Selection and Treatment: The treatment effect may only apply to the specific sample studied (Kim & Steiner, 2016).
  • Mitigation: Replicating the study with different samples (Kim & Steiner, 2016).
• Interaction of Setting and Treatment: The treatment effect may only apply to the specific setting studied (Kim & Steiner, 2016).
  • Mitigation: Conducting the study in different settings (Kim & Steiner, 2016).
• Interaction of History and Treatment: The treatment effect may only apply to the specific time period studied (Kim & Steiner, 2016).
  • Mitigation: Replicating the study at different times (Kim & Steiner, 2016).

Statistical Analysis

Appropriate statistical techniques are crucial for analyzing data from quasi-experimental designs and drawing valid causal inferences (Kim & Steiner, 2016).