The Nonequivalent Group Design
Part II

This is the second exercise in the analysis of nonequivalent group designs. In the first exercise you learned how to simulate data for a pretest - posttest two-group design. You put in an initial difference between the two groups (i.e., a selection threat) and also put in a treatment effect. You then analyzed the data with an Analysis of Covariance (ANCOVA) model and, if all went well, you found that the estimate of program effect was biased (i.e., the true effect was not within a 95% confidence interval of the obtained estimate).

In this exercise, you will create data exactly as you did in the last exercise. This time however, you will adjust the data for unreliability on the pretest before conducting the ANCOVA. Because unreliability or measurement error on the pretest causes the bias when using ANCOVA, this correction should result in an unbiased estimate of the program effect.

By now you should be fairly familiar with what most of these MINITAB commands are doing (remember, you should consult the MINITAB Handbook or use the HELP command if you want information about a command) and so the initial command will be presented without explanation. Get into MINITAB and you should see the MINITAB prompt (which looks like this MTB>). Now you are ready to enter the following commands:

MTB> Random 500 C1;
SUBC> Normal 50 5.
MTB> Random 500 C2;
SUBC> Normal 0 5.
MTB> Random 500 C3;
SUBC> Normal 0 5.
MTB> Add C1 C2 C4.
MTB> Add C1 C3 C5.
MTB> Name C1 ='true' C2 ='x error' C3 ='y error' C4 ='pretest' C5 ='posttest'
MTB> Set C6
DATA> 1:500
MTB> Code (1:250) 0 C6 C6
MTB> Code (251:500) 1 C6 C6
MTB> Name C6 = 'Group'
MTB> Let C4 = C4 + (5 * C6)
MTB> Let C5 = C5 + (15 * C6)

You have constructed the data exactly as in Part I. Notice that the data commands from the previous exercise have been condensed here. If you don't understand why you used these commands, go back and read the instructions for Part I again. Be sure that you understand that you now have data for two groups, one of which received a program or treatment (i.e., those with a '1' for C6) and the other a comparison group (i.e., with a '0' for C6). The groups differ on the pretest by an average of five points, with the program group being advantaged. On the posttest, the groups differ by fifteen points on the average. It is assumed that five of this fifteen point difference is due to the initial difference between groups and that the program had a ten point average effect. You should verify this by examining the group means:

MTB> Table C6;
SUBC> means C4 C5.

You should also look at histograms and at the bivariate distribution:

MTB> Histogram C4
MTB> Histogram C5
MTB> Plot C5 * C4
MTB> Plot C5 * C4;
SUBC> Symbol C6.

Recall that the ANCOVA yields biased estimates of effect because there is measurement error on the pretest. You added in the measurement error when you added C2 (x-error) into the pretest score. In fact, you added in about as much error as true score because the standard deviations used in all three Random/Normal commands are the same (and equal to 5). In order to adjust the pretest scores for measurement error or unreliability, we need to have an estimate of how much variance or deviation in the pretest is due to error (we know that it is about half because we set it up that way). Recall that reliability is defined as the ratio of true score variance to total score variance and that we estimate reliability using a correlation. Traditionally, we can use either a split-half reliability (the correlation between two randomly-selected subsets of the same test) or a test-retest correlation. In these simulations we will correct the pretest scores using a test-retest correlation. Since the test-retest correlation may differ between groups it is often advisable to calculate this correlation for the program and comparison groups separately. Before we can do this we have to separate the data for the two groups and put it in separate columns:

MTB> Copy C6 C4 C5 C20-C22;
SUBC> use C6 = 0.
MTB> Copy C6 C4 C5 C30-C32;
SUBC> use C6 = 1.
MTB> Name C21 = 'contpre' C22 = 'contpost' C31= 'progpre' C32 = 'progpost'

The first statement copies all cases having a '0' in C6 (i.e., all the comparison group cases) and their pretest and posttest scores (C4 and C5) and puts these scores in C20 through C22. It is important for you to notice that the order in which you copy the variables (i.e., C6, C4, C5) is the order in which they are put into the other columns. Therefore, C20 should consist entirely of '0' values. The next statement copies all the program group cases and puts them in C30 through C32. We are not interested in C20 or C30 because these consist of the '0' and '1' values so we don't name them. We name C21 'contpre' to stand for control pretest, C31 'progpre' to stand for program group pretest, and so on. Now, we are ready to estimate the test- retest reliabilities for each group. These are simply the correlations between the pretest and posttest for each group:

MTB> Correlation C21 C22 M1
MTB> Correlation C31 C32 M2

Each correlation is stored in a 2x2 matrix variable. In order for us to use these correlations later, we have to copy them from the M matrix into a K-variable constant. We can do this with the following commands:

MTB> copy M1 C41 C42
MTB> copy C41 K1;
SUBC> use 2.
MTB> copy M2 C43 C44
MTB> copy C43 K2;
SUBC> use 2.

Now, we will adjust the pretest scores for the comparison group. All you need to do is the following formula:

MTB> Let C23 = aver (C21) + (K1 * (C21 - aver (C21)))

Be sure to type this statement exactly as is. If you make a mistake, just re-enter the command. Now, let's adjust the scores for the program group. Use the following formula:

MTB> Let C33 = aver (C31) + (K2 * (C31 - aver (C31)))

You should recognize that for each group you are taking each person's pretest score, subtracting out the group pretest average, multiplying this difference by the within-group pre-post correlation (the K-variable), and then adding the group pretest mean back in. For instance, the formula for this reliability correction for the control group is:

We should probably name these new adjusted pretests:

MTB> Name C23='contadj' C33='progadj'

Now, compare the pretest means for each group before and after making the adjustment:

MTB> Describe C21 C22 C23 C31 C32 C33

What do you notice about the difference between unadjusted and adjusted scores? Did the mean change because of the adjustment (compare the means for C21 and C23 for instance)? Did the standard deviations? Remember that about half of the standard deviation of the original scores was due to error. How much have we reduced the standard deviations by adjusting? How big is the pre-post correlation for each group? Is the size of the correlation related to the standard deviations in any way?

Now, we want to combine the pretest scores for the two groups back into one set of scores (i.e., one column of data). To do this we use the Stack command:

MTB> Stack (C23) (C33) (C7).

Which means 'stack the scores in C23 on top of the scores in C33 and put these into C7'. Now let's name the adjusted pretest scores:

MTB> Name C7 = 'adjpre'

Now take a look at the means and standard deviations of the original pretest, adjusted pretest and posttest by group:

MTB> Table C6;
SUBC> means C4 C7 C5;
SUBC> stdev C4 C7 C5.

It should be clear to you that the adjustment for measurement error on the pretest did not affect the means, but did affect the standard deviations. You are now ready to conduct the analysis. First, do the ANCOVA on the original pretest scores (C4) just like in Part I. You should see that the estimate of effect is biased. Then, conduct the analysis using the adjusted pretest scores (C7). This time you should see that the estimate of effect is much closer to the ten points that you originally put in. First, the biased analysis:

MTB> Regress C5 2 C4 C6

Remember that the estimate of the program effect is the COEFFICIENT in the table which is on the C6 GROUP line. Is it near a value of ten? You can construct a 95% confidence interval for this estimate using the ST.DEV. OF COEF. which is on the same line as the estimate. To construct this interval, first multiply the standard deviation of the coefficient by two (remember from statistics that the 95% confidence interval is plus or minus approximately two standard error units). To get the lower limit of the interval subtract this value from the coefficient in the table, to get the upper limit add them. Does the true program effect (10 points) fall within the bounds of this interval. For most of you it will not (just by chance, it may for a few of you. Event if it does fall within the interval it will probably not be near the center of the interval). Now, conduct the same ANCOVA analysis this time substituting the adjusted pretest scores (C7) for the original ones:

MTB> Regress C5 2 C7 C6

Again, look at the coefficient in the table that is on the same line as the C6 GROUP variable. Is this value closer to the true program effect of 10 points than it was for the previous analysis? It should be (again, for some of you this may not be true just by chance alone). Once again, construct the 95% confidence interval for the coefficient using the standard deviation of the coefficient given in the table. You should see that this time the true program effect of 10 points falls within the interval.

You have just conducted a reliability-corrected Analysis of Covariance. In practice, we know that test-retest and split-half reliability estimates will often differ, sometimes considerably. Because of this, we will often conduct two analyses like the one above- once using the test-retest correlations and once using the split-half correlations. Although this will give us two estimates of the program effect, we expect that one is a conservative adjustment while the other is a liberal one. We can therefore be fairly confident that the true program effect probably lies somewhere between the two estimated effects.

At this point, you should stop and consider the steps that were involved in conducting this analysis. You might want to try one or more of the following variations:

Downloading the MINITAB Commands

If you wish, you can download a text file that has all of the MINITAB commands in this exercise and you can run the exercise simply by executing this macro file. To find out how to call the macro check the MINITAB help system on the machine you're working on. You may want to run the exercise several times -- each time will generate an entirely new set of data and slightly different results. Click here if you would like to download the MINITAB macro for this simulation.
Simulation Home Page
Copyright 1996, William M.K. Trochim