## The Nonequivalent Group DesignPart II

This is the second exercise in the analysis of nonequivalent group designs. In the first exercise you learned how to simulate data for a pretest - posttest two-group design. You put in an initial difference between the two groups (i.e., a selection threat) and also put in a treatment effect. You then analyzed the data with an Analysis of Covariance (ANCOVA) model and, if all went well, you found that the estimate of program effect was biased (i.e., the true effect was not within a 95% confidence interval of the obtained estimate).

In this exercise, you will create data exactly as you did in the last exercise. This time however, you will adjust the data for unreliability on the pretest before conducting the ANCOVA. Because unreliability or measurement error on the pretest causes the bias when using ANCOVA, this correction should result in an unbiased estimate of the program effect.

By now you should be fairly familiar with what most of these MINITAB commands are doing (remember, you should consult the MINITAB Handbook or use the HELP command if you want information about a command) and so the initial command will be presented without explanation. Get into MINITAB and you should see the MINITAB prompt (which looks like this MTB>). Now you are ready to enter the following commands:

MTB> Random 500 C1;
SUBC> Normal 50 5.
MTB> Random 500 C2;
SUBC> Normal 0 5.
MTB> Random 500 C3;
SUBC> Normal 0 5.
MTB> Name C1 ='true' C2 ='x error' C3 ='y error' C4 ='pretest' C5 ='posttest'
MTB> Set C6
DATA> 1:500
DATA> End
MTB> Code (1:250) 0 C6 C6
MTB> Code (251:500) 1 C6 C6
MTB> Name C6 = 'Group'
MTB> Let C4 = C4 + (5 * C6)
MTB> Let C5 = C5 + (15 * C6)

You have constructed the data exactly as in Part I. Notice that the data commands from the previous exercise have been condensed here. If you don't understand why you used these commands, go back and read the instructions for Part I again. Be sure that you understand that you now have data for two groups, one of which received a program or treatment (i.e., those with a '1' for C6) and the other a comparison group (i.e., with a '0' for C6). The groups differ on the pretest by an average of five points, with the program group being advantaged. On the posttest, the groups differ by fifteen points on the average. It is assumed that five of this fifteen point difference is due to the initial difference between groups and that the program had a ten point average effect. You should verify this by examining the group means:

MTB> Table C6;
SUBC> means C4 C5.

You should also look at histograms and at the bivariate distribution:

MTB> Histogram C4
MTB> Histogram C5
MTB> Plot C5 * C4
MTB> Plot C5 * C4;
SUBC> Symbol C6.

Recall that the ANCOVA yields biased estimates of effect because there is measurement error on the pretest. You added in the measurement error when you added C2 (x-error) into the pretest score. In fact, you added in about as much error as true score because the standard deviations used in all three Random/Normal commands are the same (and equal to 5). In order to adjust the pretest scores for measurement error or unreliability, we need to have an estimate of how much variance or deviation in the pretest is due to error (we know that it is about half because we set it up that way). Recall that reliability is defined as the ratio of true score variance to total score variance and that we estimate reliability using a correlation. Traditionally, we can use either a split-half reliability (the correlation between two randomly-selected subsets of the same test) or a test-retest correlation. In these simulations we will correct the pretest scores using a test-retest correlation. Since the test-retest correlation may differ between groups it is often advisable to calculate this correlation for the program and comparison groups separately. Before we can do this we have to separate the data for the two groups and put it in separate columns:

MTB> Copy C6 C4 C5 C20-C22;
SUBC> use C6 = 0.
MTB> Copy C6 C4 C5 C30-C32;
SUBC> use C6 = 1.
MTB> Name C21 = 'contpre' C22 = 'contpost' C31= 'progpre' C32 = 'progpost'

The first statement copies all cases having a '0' in C6 (i.e., all the comparison group cases) and their pretest and posttest scores (C4 and C5) and puts these scores in C20 through C22. It is important for you to notice that the order in which you copy the variables (i.e., C6, C4, C5) is the order in which they are put into the other columns. Therefore, C20 should consist entirely of '0' values. The next statement copies all the program group cases and puts them in C30 through C32. We are not interested in C20 or C30 because these consist of the '0' and '1' values so we don't name them. We name C21 'contpre' to stand for control pretest, C31 'progpre' to stand for program group pretest, and so on. Now, we are ready to estimate the test- retest reliabilities for each group. These are simply the correlations between the pretest and posttest for each group:

MTB> Correlation C21 C22 M1
MTB> Correlation C31 C32 M2

Each correlation is stored in a 2x2 matrix variable. In order for us to use these correlations later, we have to copy them from the M matrix into a K-variable constant. We can do this with the following commands:

MTB> copy M1 C41 C42
MTB> copy C41 K1;
SUBC> use 2.
MTB> copy M2 C43 C44
MTB> copy C43 K2;
SUBC> use 2.

Now, we will adjust the pretest scores for the comparison group. All you need to do is the following formula:

MTB> Let C23 = aver (C21) + (K1 * (C21 - aver (C21)))

Be sure to type this statement exactly as is. If you make a mistake, just re-enter the command. Now, let's adjust the scores for the program group. Use the following formula:

MTB> Let C33 = aver (C31) + (K2 * (C31 - aver (C31)))

You should recognize that for each group you are taking each person's pretest score, subtracting out the group pretest average, multiplying this difference by the within-group pre-post correlation (the K-variable), and then adding the group pretest mean back in. For instance, the formula for this reliability correction for the control group is: We should probably name these new adjusted pretests:

Now, compare the pretest means for each group before and after making the adjustment:

MTB> Describe C21 C22 C23 C31 C32 C33

What do you notice about the difference between unadjusted and adjusted scores? Did the mean change because of the adjustment (compare the means for C21 and C23 for instance)? Did the standard deviations? Remember that about half of the standard deviation of the original scores was due to error. How much have we reduced the standard deviations by adjusting? How big is the pre-post correlation for each group? Is the size of the correlation related to the standard deviations in any way?

Now, we want to combine the pretest scores for the two groups back into one set of scores (i.e., one column of data). To do this we use the Stack command:

MTB> Stack (C23) (C33) (C7).

Which means 'stack the scores in C23 on top of the scores in C33 and put these into C7'. Now let's name the adjusted pretest scores:

Now take a look at the means and standard deviations of the original pretest, adjusted pretest and posttest by group:

MTB> Table C6;
SUBC> means C4 C7 C5;
SUBC> stdev C4 C7 C5.

It should be clear to you that the adjustment for measurement error on the pretest did not affect the means, but did affect the standard deviations. You are now ready to conduct the analysis. First, do the ANCOVA on the original pretest scores (C4) just like in Part I. You should see that the estimate of effect is biased. Then, conduct the analysis using the adjusted pretest scores (C7). This time you should see that the estimate of effect is much closer to the ten points that you originally put in. First, the biased analysis:

MTB> Regress C5 2 C4 C6

Remember that the estimate of the program effect is the COEFFICIENT in the table which is on the C6 GROUP line. Is it near a value of ten? You can construct a 95% confidence interval for this estimate using the ST.DEV. OF COEF. which is on the same line as the estimate. To construct this interval, first multiply the standard deviation of the coefficient by two (remember from statistics that the 95% confidence interval is plus or minus approximately two standard error units). To get the lower limit of the interval subtract this value from the coefficient in the table, to get the upper limit add them. Does the true program effect (10 points) fall within the bounds of this interval. For most of you it will not (just by chance, it may for a few of you. Event if it does fall within the interval it will probably not be near the center of the interval). Now, conduct the same ANCOVA analysis this time substituting the adjusted pretest scores (C7) for the original ones:

MTB> Regress C5 2 C7 C6

Again, look at the coefficient in the table that is on the same line as the C6 GROUP variable. Is this value closer to the true program effect of 10 points than it was for the previous analysis? It should be (again, for some of you this may not be true just by chance alone). Once again, construct the 95% confidence interval for the coefficient using the standard deviation of the coefficient given in the table. You should see that this time the true program effect of 10 points falls within the interval.

You have just conducted a reliability-corrected Analysis of Covariance. In practice, we know that test-retest and split-half reliability estimates will often differ, sometimes considerably. Because of this, we will often conduct two analyses like the one above- once using the test-retest correlations and once using the split-half correlations. Although this will give us two estimates of the program effect, we expect that one is a conservative adjustment while the other is a liberal one. We can therefore be fairly confident that the true program effect probably lies somewhere between the two estimated effects.

At this point, you should stop and consider the steps that were involved in conducting this analysis. You might want to try one or more of the following variations:

• Change the reliability of the pretest. Recall that the reliability of the pretest depends on how much true score and error you add into it. Try the simulation with extremely low reliability (high standard deviation for error, C2, low for true score, C1) and high reliability. You might even want to try a perfectly reliable pretest (i.e., don't add C2 in at all). In this case, we would expect that the unadjusted ANCOVA yields an unbiased estimate of the program effect.

• Change the reliability of the posttest. Here you would alter the ratio of the standard deviations for the Y error, C3, and true score, C1 variables. Recall that unreliability on the posttest should not result in biased estimates with unadjusted ANCOVA. However, the estimates will be less precise when you have more error on the posttest. You will see this by looking at the standard deviation of the coefficient and comparing it with the standard deviation of the coefficient which you obtained in this simulation. With higher error variance on the posttest, the 95% confidence interval should be wider.

• As in the previous exercise variations, try to construct a simulation where the program group is disadvantaged relative to the comparison group.

• Conduct the simulation above but put in a negative program effect. To do this, just put in a -15 for the 15 in the LET statement which constructed the posttest. Simulation Home Page