The Randomized Experimental Design

Overview

This simulation models a pretest-posttest randomized experiment. We generate 500 participants, randomly assign half to a program group and half to a comparison group, and add a 10-point treatment effect to the program group's posttest scores. Because assignment is random, the two groups start out equivalent on average — any posttest difference is attributable to the program.

We then analyze the data three ways: a simple t-test comparing posttest means, an ANOVA model, and an ANCOVA model that uses the pretest as a covariate. All three should recover the true treatment effect of 10 points. The value of ANCOVA is that it reduces residual variance and narrows the confidence interval around the effect estimate.

Step 1 — Generate Data

Create true scores and error terms, then construct the pretest X and posttest Y.

Step 2 — Random Assignment

Assign participants to groups by drawing a random normal variable and splitting at its median. This creates two groups of approximately equal size. Z = 1 indicates the program group; Z = 0 the comparison group.

Step 3 — Build in the Treatment Effect

Create the posttest with a 10-point treatment effect for the program group. For program cases (Z = 1), the product 10 * Z equals 10; for comparison cases (Z = 0) it equals 0. This is how we "add in" an effect without changing the data-generation process for the comparison group.

Both groups should have nearly identical pretest means (confirming successful randomization), but the program group should have a posttest mean about 10 points higher than the comparison group.

Step 4 — t-Test

A simple independent-samples t-test compares the posttest means of the two groups. The estimated treatment effect is the mean difference; the p-value tests whether that difference could plausibly be zero.

Step 5 — ANOVA

An ANOVA model with group as the only predictor gives the same result as the t-test for a two-group design. The coefficient for Z is the estimated treatment effect.

Step 6 — ANCOVA

Adding the pretest X as a covariate reduces unexplained variance and improves precision. In a randomized experiment ANCOVA is not needed for an unbiased estimate, but it typically narrows the standard error of the treatment effect. Compare the standard error of the Z coefficient between the ANOVA and ANCOVA models.

In the bivariate plot you should see two parallel regression lines (same slope, different intercepts) with a vertical gap of about 10 points between them — the treatment effect.