Simulation to investigate Tukey and Fisher's LSD procedures
Tukey's procedure provides strong control of FWE Fishers LSD procedure provides weak control of FWE (strong and weak defined below) Fisher's LSD procedure does not make any adjustment for multiplicity except to carry out the tests only after evidence of some inequality among treatment means, that is, only after carrying out an F-test and rejecting the null hypothesis of all treatment means equal. In the absence of such a rejection, this procedure does not go on to make subsequent multiple comparisons. See the code diets-sim.sas and output diets-sim.lst
  1. Simulate a single dataset from a completely randomized design in which the mean weight gain under five diets to which 30 turkeys are randomized are
    (9,15,15,15,19)
    and the errors are iid normal with mean 0 and std. deviation sigma=4;
  2. Fit these data using the PROC MIXED code provided, including the LSMEANS treatment/TUKEY statement to carry out all multiple comparisons. Go through the output from PROC MIXED carefully.
  3. Note that there are two types of type I error control in multiple testing problems: strong and weak defined as follows:
    strong - under any configuration of treatment means, e.g. (9,15,15,15,19)
    weak - under the configuration of equal treatment means, e.g. (15,15,15,15,15)
    To observe the behavior of these two procedures, consider simulating many datasets. Among the k=10 pairwise comparisons, there are 3 comparing means where there is no effect (2-3,2-4,3-4) and 7 with real effects. The code provided will keep track of the sum of type I errors for each simulation and each MCP.

Questions: (4-6 are intended for graduate students in statistics)

Consider these two configurations of treatment means:
mu' = (12,15,15,15,16)
mu' = (9,15,15,15,19)
1) Which configuration will lead to more significant F-tests?
2) For which configuration is the failure of the LSD procedure to strongly control the FWE more acute:
3a) What effect does varying sigma have on the overall F-test?
3b) What effect does varying sigma have on LSD and HSD performance?
3c) What effect does increasing the number of turkeys have on the overall F-test?

4) Change the seed. Based on your simulation with 500 simulations, determine an estimate of the FWE of Fisher's LSD. Determine the Monte Carlo Standard Error (MCSE). How would the MCSE change if you ran 2000 simulations?
5) Using the configuration (15,15,15,15,15), check whether the simulated data is consistent with the LSD's weak control of FWE.
6) Consider developing the code to simultaneously compare the power of the LSD and Tukey procedures and maybe even the Bonferroni procedure. Intuitively, which will be most powerful? (Coding this in SAS might be somewhat challenging.)