In the ideal situation, specific questions regarding comparisons among the means are posed before the data are collected. We can answer specific questions of this kind and attach a level of confidence to the answers we give. We now explore these ideas through a Facebook study.

Example 12.18 How do users spend their time on Facebook?

An online study was designed to compare the amount of time a Facebook user devotes to reading positive, negative, and neutral Facebook profiles. Each participant was randomly assigned to one of five Facebook profile groups:

1. Positive female
2. Positive male
3. Negative female
4. Negative male
5. Gender neutral with neutral content

Each participant was provided an email link to a survey on Survey Monkey. As part of the survey, the participant was directed to view the assigned Facebook profile page and then answer some questions. The amount of time (in minutes) the participant spent viewing the profile prior to answering the questions was recorded as the response.¹⁰

We should always begin our analysis with a check of the data. Time-to-event data (here, the time until the participant begins to answer the survey questions) is often skewed to the right. Preliminary analysis of the residuals (Figure 12.11) confirms this.

As a result, we consider the square root of time for analysis. These results are summarized in Figures 12.12 and 12.13. The residuals appear approximately Normal (Figure 12.12), and our rule for examining standard deviations indicates that we can assume equal population standard deviations (1.041<2(0.834)).

The F test is significant with a P-value of 0.002. Because the P-value is very small, there is strong evidence against H0. We can conclude that the five population means are not all the same (F(4,100)=4.55 with P=0.002).

Rejecting the null hypothesis

and concluding that the five population means are not the same does not tell us all we’d like to know. We would really like our analysis to provide us with more specific information. For example, the alternative hypothesis is true if

or if

or if

When you reject the ANOVA null hypothesis, additional analyses are required to clarify the nature of the differences between the means.

For this study, the researcher predicted that participants would spend more time viewing the negative Facebook pages compared to the positive or neutral pages because the negative pages would stand out more and thus garner more attention. (This is called cognitive salience.) How do we translate these predictions into testable hypotheses?

In the preceding example, we used H01 and Ha1 to designate the null and alternative hypotheses. The reason for this is that there is an additional set of hypotheses to assess. We use H02 and Ha2 for this set.

Each of H01 and H02 says that a combination of population means is 0. These combinations of means are called contrasts because the coefficients sum to zero. We use ψ, the Greek letter psi, for contrasts among population means. For our first comparison, we have

and for the second comparison

In each case, the value of the contrast is 0 when H0 is true. Note that we have chosen to define the contrasts so that they will be positive when the alternative of interest (what we expect) is true. Whenever possible, this is a good idea because it makes results easier to read.

A contrast expresses an effect in the population as a combination of population means. To estimate the contrast, form the corresponding sample contrast by using sample means in place of population means. Under the ANOVA assumptions, a sample contrast is a linear combination of independent Normal variables and, therefore, has a Normal distribution (page 49). We can obtain the standard error of a contrast by using the rules for variances. Inference is based on t statistics. Here are the details.

Because each x¯i estimates the corresponding μi, the addition rule for means tells us that the mean μc of the sample contrast c is ψ. In other words, c is an unbiased estimator of ψ. Testing the hypothesis that a contrast is 0 assesses the significance of the effect measured by the contrast. A confidence interval provides an estimate of the size of the effect. It is good practice to include a confidence interval with a test of a population contrast, especially when the effect is found statistically significant.

We now look at inference for each of these contrasts in turn.

We use the same method for the second contrast.

The size of the difference can be described with a confidence interval.

JMP output for the two contrasts is given in Figure 12.14. The results agree with the calculations that we performed in Examples 12.22 and 12.23 except for minor differences due to roundoff error in our calculations. Note that the output does not give the confidence interval that we calculated in Example 12.24. This is easily computed, however, from the contrast estimate and standard error provided in the output.

Some statistical software packages report the test statistics associated with contrasts as F statistics rather than t statistics. These F statistics are the squares of the t statistics described previously. As with much other statistical software output, P-values for significance tests are reported for the two-sided alternative.

If the software you are using gives P-values for the two-sided alternative, and you are using the appropriate one-sided alternative, divide the reported P-value by 2. In our example, we argued that a one-sided alternative may be appropriate for the first contrast. The software reported the P-value as 0.836, so we can conclude P=0.418. Dividing this value by 2 has no effect on the conclusion.

Questions about population means are expressed as hypotheses about contrasts. A contrast should express a specific question that we have in mind when designing the study. Because the ANOVA F test answers a very general question, it is less powerful than tests for contrasts designed to answer specific questions.

When contrasts are formulated before seeing the data, inference about contrasts is valid whether or not the ANOVA H0 of equality of means is rejected. Specifying the important questions before the analysis is undertaken enables us to use this powerful statistical technique.

In many studies, specific questions cannot be formulated in advance of the analysis. If H0 is not rejected, we conclude that the population means are indistinguishable on the basis of the data given. On the other hand, if H0 is rejected, we would like to know which pairs of means differ. Multiple-comparisons procedures address this issue. It is important to keep in mind that multiple-comparisons procedures are generally used only after rejecting the ANOVA H0.

These 10 t statistics are very similar to the pooled two-sample t statistic for comparing two population means. The difference is that we now have more than two populations, so each statistic uses the pooled estimator sp from all groups rather than the pooled estimator from just the two groups being compared. This additional information about the common σ increases the power of the tests. The degrees of freedom for all these statistics are DFE=100, those associated with sp.

Because we do not have any specific ordering of the means in mind as an alternative to equality, we must use a two-sided approach to the problem of deciding which pairs of means are significantly different.

One obvious choice for t** is the upper α/2 critical value for the t(DFE) distribution. This choice simply carries out as many separate significance tests of fixed level α as there are pairs of means to be compared. The method based on this choice is called the least-significant differences (LSD) method.

The LSD method has some undesirable properties, particularly if the number of means being compared is large. Suppose, for example, that there are I=20 groups, and we use the LSD method with α=0.05. There are 190 different pairs of means. If we perform 190 t tests, each with an error rate of 5%, our overall error rate will be unacceptably large. We expect about 5% of the 190 to be significant even if the corresponding population means are the same. Because 5% of 190 is 9.5, we expect 9 or 10 false rejections.

Because LSD fixes the probability of a false rejection for each single pair of means being compared, it does not control the overall probability of some false rejection among all pairs. Other choices of t** control possible errors in other ways. The choice of t** is, therefore, a complex problem, and a detailed discussion of it is beyond the scope of this text. Many choices for t** are used in practice. Most statistical packages provide several to choose from.

We will discuss only one of these, called the Bonferroni method. Use of this method with α=0.05, for example, guarantees that the probability of any false rejection among all comparisons made is no greater than 0.05. This is much stronger protection than controlling the probability of a false rejection at 0.05 for each separate comparison.

Of course, we prefer to use software for the calculations.

Example 12.27 Interpreting software output.

The output generated by SPSS for comparisons using the Bonferroni method appears in Figure 12.15. Here, all 10 comparisons are reported. In fact, each comparison is given twice. The software uses an asterisk to indicate that the difference in a pair of means is statistically significant. These results agree with our conclusions for the three comparisons in Example 12.26.

SPSS does not provide the t statistic but rather a Bonferroni-adjusted P-value for the comparisons under the heading “Sig.” The Bonferroni-adjusted P-value is obtained by multiplying the unadjusted P-value by the number of comparisons. When this is greater than 1, the adjusted P-value is reported as 1.000. In the “Sig.” column, there are only three comparisons with a P-value smaller than the overall α=0.05 level. These are the comparisons of Profiles 1, 2, or 4 versus Profile 5. If you computed the t statistic, you’d find that these are the only comparisons with a t statistic greater than t**=2.871.

When there are a large number of groups, it is often difficult to concisely describe the numerous results of multiple comparisons. Instead, researchers often list the means and use letters to label the pairs that are not found to be statistically different.

The conclusions from this table in Example 12.28 appear to be illogical. If μ1 is the same as μ3 and if μ3 is the same as μ5, doesn’t it follow that μ1 is the same as μ5? Logically, the answer must be Yes.

This apparent contradiction points out the nature of the conclusions of statistical tests of significance. In the multiple-comparison setting, these tests ask, “Do we have adequate evidence to distinguish two means?” In describing the results, we should talk about failing to detect a difference or concluding that two groups are different. We also must remember that failing to find strong enough evidence that two means differ doesn’t say that they are equal. Thus, it is not illogical to conclude that we have sufficient evidence to distinguish μ1 from μ5 but not μ1 from μ3 or μ3 from μ5.

One way to deal with the difficulties of interpretation is to give confidence intervals for the differences. The intervals remind us that the differences are not known exactly. We also want to give simultaneous confidence intervals—that is, intervals for all differences among the population means at once. Again, we must face the problem that there are many competing procedures—in this case, many methods of obtaining simultaneous intervals.

The confidence intervals generated by a particular choice of t** are closely related to the multiple-comparisons results for that same method. If one of the confidence intervals includes the value 0, then that pair of means will not be declared significantly different and vice versa.

The power of a statistical test is a measure of the test’s ability to detect deviations from the null hypothesis. In Chapter 7, we described the use of power calculations to ensure adequate sample size for inference about means in one- and two-population studies. In this section, we extend these methods to any number of populations.

Because the one-way ANOVA F test is a generalization of the two-sample t test, it should not be surprising that the procedure for calculating power of the F test is quite similar. In both cases, the following four study specifications are needed to compute power:

• Significance level α
• Common population standard deviation
• Group sample sizes
• Specific alternative Ha

The difference comes in the specific calculations. Instead of using a t and noncentral t distribution to compute the probability of rejecting H0 under a specific alternative, we now use an F and a noncentral F distribution. Calculations involving the noncentral F distributions are again not practical to compute by hand but are quite easy with software.

The last three study specifications in the prior list determine the appropriate noncentral F distribution. Most software assumes a constant sample size n for the group sizes. For the alternative, some software doesn’t request group means but rather the smallest difference between means that is judged practically important. Here is an example using two software packages that approach specifying Ha differently.

Example 12.30 Power of a reading comprehension study.

Suppose that a study on reading comprehension for three different teaching methods has 10 students in each group. How likely is this study to detect differences in means? A previous study, using the same three teaching methods but performed in a different setting, found sample means of 40, 48, and 42, and the pooled standard deviation was 7. We’ll use these values to compute the power for this new study.

Figure 12.16 shows the power calculation output from JMP and Minitab. In both cases, we use α=0.05, and n1=n2=n3=10, and σ=7. The difference is in how we input this information and the group means of the specific Ha.

For JMP, we enter the alternative group means and the total sample size N=30. The power is calculated when the “Continue” button is clicked. It is roughly 61%. For Minitab, we enter the common sample size N=10 and the smallest difference between means that is deemed important. The largest difference among the previous sample means is 8=48−40, so that was entered. The calculated power is 57%.

As in this case, the power is usually lower when only specifying an important difference between means. This is because the other population means are not specified, and so the software considers a worst-case scenario.

If the assumed values of the μi in the previous study describe differences among the groups that the experimenter wants to detect, then we would want to use more than 10 subjects per group. Although H0 is false for these μi, the chance of rejecting it at the 5% level is only about 60%. This chance can be increased to higher levels by increasing the sample sizes.

When you have a level of power in mind, software will allow you to directly solve for the necessary sample size rather than trying different values for n, as we did in this example. Instead of leaving the power blank, you’d enter the desired power as an input (usually 80% or 90%) and leave the sample size blank. Try this out using JMP for the setting of Example 12.31 and verify that the experimenters need n=19 for 90% power.

A study with 90% power means the experimenters have a 90% chance of rejecting H0 and thereby demonstrating that the groups have different means. In the long run, 90 out of every 100 such experiments would reject H0 at the α=0.05 level of significance. This is much better than the original 61 out of every 100 such experiments when n=10.

There is, however, a cost associated with this increased power. The total sample size is almost doubled, which could mean a doubling of the cost of the study. To trade off cost with the risk of not detecting the difference in the means, researchers generally shoot for power in the 80% to 90% range. In most real-life situations, the additional cost of increasing the sample size so that the power is very close to 100% cannot be justified.