An experiment in statistics is the process of assigning either treatment or control to experimental units. The validity of the experiment depends on the structure of the experiment. In randomized experiment design, the assignment is done randomly (by chance), which reduces the interference of factors other than the independent variables. This will help us inferring cause and effect.
But, it’s not always possible to implement randomness and hence, we try to execute a design which will neutralize the effect of the interference. For e.g. Uber wants to measure the impact of payment flow changes made in the app. If Uber decides to conduct the experiment on dividing drivers or users between control or treatment, there is no way to prevent the people in control to not affect the users in treatment, which will violate the SUTVA assumption and hence, lead us to overestimate the effects of the treatment. In order to neutralize the effects of interference, Uber decided to conduct a form of quasi-experiment called switch-back experiments where Uber divides the whole market into clusters of user/drivers and switch back and forth between treatment and control for a period of time and then measure the impact.
A quasi-experiment design can also be used in cases where random assignment of treatment or control is unethical or practically too-expensive to implement.
The biggest plus of using a quasi-experiment design is that it has higher external validity because they are mostly used in real-world cases. But, since these experiments have lower internal validity because of lack of randomness in the experimental design.
Different types of quasi-experiment design:
- Non-equivalent group design
- Regression discontinuity
- Natural experiments
Non-equivalent group design
These experiments are designs are conducted on similar existing groups with one of these groups receiving treatment. Over the period of time, researchers have improved upon the process of conducting such experiments. Different types of non-equivalent group designs are:
- Posttest only non-equivalent group designs: The experimental units are compared after one of the group is exposed to treatment.
- Pretest-Posttest only non-equivalent group designs: The experimental units are compared before and after one of the group is exposed to treatment.
- Interrupted time-series design with non-equivalent group designs: Two or more groups are exposed to treatment and measurements are done at regular intervals, before and after the exposure.
- Pretest-Posttest design with switching replication: The non-equivalent groups are alternated between control and treatment and observations are made before and after the exposure to the treatment.
- Switching replication with treatment removal design: This is similar to Pretest-Posttest design with switching replication but instead we expose the treatment to one group a little earlier than it’s removed from the first group.
Regression discontinuity design (RDD)
This method is employed when there is cut-off point for the eligibility for treatment. In such case, the users just above or below the cut-off point can be examined and compared. There are two types of RDD and they are defined on the basis of cut-off point.
- Sharp design: If the index to decide the cut-off is deterministic.
- Fuzzy design: If the index to decide the cut-off is probabilistic.
Natural experiment
In a natural experiment design, the distribution of treatment is performed by nature. Impact of policy changes or exposure to weather presents a perfect setting to conduct natural experiments.
The most famous natural experiment was conducted by John Snow in the 19th century where he used the principles of quasi-experimental design to infer the cause of cholera epidemic in London.
