The data for one-way ANOVA are collected just as data are collected in the two-sample case. We either draw a random sample from each population or we randomly divide subjects into different treatments. The resulting data are used to test the null hypothesis that the means are all equal.

Consider the following two examples.

These two examples are similar in that

• There is a single quantitative response variable measured on many units; the units are the 60 participants in the first example and the 250 customers in the second.
• The goal is to compare several populations: participants using three different game controller types in the first example and customers of the five coffeehouses in the second.
• There is a single categorical explanatory variable, or factor, that classifies these populations: controller type in the first example and coffeehouse in the second.

There is, however, an important difference. Example 12.1 describes an experiment in which each participant is randomly assigned to a type of game controller. Example 12.2 is an observational study in which customers are selected during a particular time period, and not all agree to provide data. This means that the samples in the second example are not truly SRSs. The reporter will treat them as such because she believes that the selective sampling and nonresponse are ignorable sources of bias and variability, but others may not agree. Always consider the sampling design and various sources of bias in an observational study.

In both examples, one-way ANOVA is used to compare the mean responses. The same ANOVA methods apply to data from randomized experiments (Example 12.1) and to data from random samples (Example 12.2). However, it is important to keep in mind the data-production method when interpreting the results. A strong case for causation is best made by a randomized experiment.

The question we ask in one-way ANOVA is “Do all groups have the same population mean?” (We will often use the term groups for the populations to be compared in a one-way ANOVA.) To answer this question, we compare the sample means. Figure 12.1 displays the sample means for Example 12.1. It appears that a game controller with force feedback has the shortest average completion time. But is the observed difference in sample means just the result of chance variation? We should not expect sample means to be equal even if all groups have the same mean.

The purpose of one-way ANOVA is to assess whether the observed differences among sample means are statistically significant. Could variation among the three sample means this large be plausibly due to chance, or is it strong evidence for a difference among the group means? This question can’t be answered from the sample means alone. Because the standard deviation of a sample mean x¯ is the population standard deviation σ divided by n, the answer also depends upon both the variation within the groups of observations and the sizes of the samples.

Side-by-side boxplots help us see the variation within the groups. Compare Figures 12.2(a) and 12.2(b). The sample medians are the same in both panels, but the large variation within the groups in Figure 12.2(a) suggests that the differences among the sample medians could be due simply to chance variation. The data in Figure 12.2(b) are much more convincing evidence that the populations differ.

Even the boxplots omit essential information, however. To assess the observed differences, we must also know how large the samples are. Nonetheless, boxplots provide a good preliminary display of the data.

Although one-way ANOVA compares means and boxplots display medians, these two measures of center will be close together for distributions that are nearly symmetric. This is something we need to check prior to inference. If the distributions are strongly skewed, the means are not good summaries of the population centers. We may, for example, consider a transformation prior to displaying and analyzing the data.

Two-sample t statistics compare the means of two populations. If the two populations are assumed to have equal but unknown standard deviations and the sample sizes are both equal to n, the t statistic is

The square of this t statistic is

If we use one-way ANOVA to compare two populations, the ANOVA F statistic is exactly equal to this t2. Thus, we can learn something about how ANOVA works by looking carefully at the statistic in this form.

The numerator in the t2 statistic measures the between-group variation in terms of the difference between their sample means x¯1 and x¯2 and the common sample size n. The numerator can be large because of a large difference between the sample means or because the common sample size n is large.

The denominator measures the within-group variation by sp2, the pooled estimator of the common variance. For the same sample means and n, the smaller the within-group variation, the larger the test statistic and, thus, the more significant the result (recall Figure 12.2).

Although the general form of the one-way ANOVA F statistic is more complicated, the idea is the same. To assess whether several populations all have the same mean, we compare the variation among the means of several groups with the variation within groups. Just like with linear regression, we are comparing variation so the method is called analysis of variance.

Now that we have a general idea of how one-way ANOVA works, we will explore the specifics using an example. Because the computations are more lengthy than those for the t test, we generally use computer software to perform the calculations. Automating the calculations frees us from the burden of needing to perform arithmetic and allows us to concentrate on interpretation.

Example 12.3 Task performance with background music.

The effect of music on task performance has long been of interest to researchers. The “Mozart effect” is one of the first, most well-known, and controversial explanations of music’s influence. It suggests a short-term increase of 8 to 9 points in mean spatial IQ score after listening to Mozart’s Sonata for Two Pianos (K448).² Subsequent research has looked at replicating these results as well as looking at the effect with different types of tasks, different types of music, and different levels of volume. Findings have been mixed, with some suggesting music improved performance, others suggesting music hindered performance, and still others finding no evidence of an effect.

One experiment compared the performance of various tasks under three different types of background music conditions: silence, music with lyrics, and music conditions without lyrics.³ For our analysis, we will focus just on the mathematics task, which involved solving as many simple arithmetic problems as possible in a minute.

A total of 452 participants were recruited through Amazon Mechanical Turk (MTurk), which is a crowdsourcing marketplace. Of these, only 447 completed the mathematics task. Here is a summary of the data for the number of completed simple arithmetic problems x in a minute:

Group	n	x¯	s
Silence	152	18.855	4.690
Music without lyrics	149	17.745	4.175
Music with lyrics	146	17.815	4.238

It appears that music results in a decrease in production of about one problem and that there is very little difference between the types of music. ANOVA will allow us to determine whether this observed pattern in sample means can be extended to the group means.

We should always start our ANOVA with a careful examination of the data using graphical and numerical summaries. Complicated computations do not guarantee a valid statistical analysis. This examination gives us an idea of what to expect from the analysis and also enables us to check for unusual patterns in the data. Just as with the two-sample t methods and linear regression, outliers and extreme deviations from Normality can invalidate the computed results.

Example 12.4 A graphical summary of the data.

Let’s use the labels 1, 2, and 3 to represent silence, music without lyrics, and music with lyrics, respectively. Histograms for the three groups are given in Figure 12.3. Note that the heights of the bars are percents rather than counts. Percents are commonly used when the group sample sizes vary. Figure 12.4 gives side-by-side boxplots for these data. We see a lot of spread in the data, with the number of completed problems ranging from 6 to 31.

There do not appear to be any outliers, and the distributions appear relatively symmetric. Even though the data are counts, the fact that the group sample sizes are all more than 45 suggests that we can feel confident that the sample means are approximately Normal. As with the two-sample procedures, it is important for the sample means, rather than the data, to be approximately Normal.

Figure 12.4 also includes the three means adjacently joined with a line. The difference in means pales in comparison to the variability within each group. This suggests that the observed pattern in the means could likely be due to chance variation.

Including the mean in a boxplot is quite common. Joining the means of side-by-side boxplots is less common, and some do not like it. It is done here to help show the differences in sample means relative to the spread in the data. Oftentimes, combined plots like this one—a combination of a mean plot (e.g., Figure 12.1) and side-by-side boxplots—can be more helpful than separate, more traditional ones. Do you think it is more helpful here?

The setting of this example is an experiment in which participants were randomly assigned to one of the three groups. Each of these groups has a mean, and our inference asks questions about these means. The participants were all recruited from MTurk and participated online. They were paid $1.00 in return for completing the study. The authors also state that they were properly motivated to take the tasks seriously using an incentive payment of $5.00 that went to the participant with the highest performance.

Formulating a clear definition of the populations being compared with ANOVA can be difficult. Often some expert judgment is required, and different consumers of the results may have different opinions. Whether we can comfortably generalize our conclusions of this study to the population of MTurk workers or to the general population of adults is open to debate. Using MTurk is quite popular because it supplies a large number of potential participants at very low cost. The trade-off is more difficulty in clearly defining the populations. Regardless, we are more confident in generalizing our conclusions to similar populations when the results are clearly significant than when the level of significance just barely passes the standard of P=0.05.

For inference, we first ask whether or not there is sufficient evidence in the data to conclude that the corresponding group means are not all equal. Our null hypothesis here states that the population mean score is the same for all three groups. The alternative is that they are not all the same. Rejecting the null hypothesis that the means are all the same is not the same as concluding that all the means are different from one another. The one-way ANOVA alternative hypothesis can be true in many different ways. It could be true because all the means are different or simply because one of them differs from the rest.

Because the one-way ANOVA alternative is a more complex situation than the two-population comparison alternative, we need to perform some further analysis if we reject the null hypothesis. This additional analysis allows us to draw conclusions about which population means differ from which others and by how much.

For the music study, the researchers hypothesize that background music should negatively impact performance. Therefore, a reasonable question to ask is whether the mean for the silence group is larger than the others. When there are particular versions of the alternative hypothesis that are of interest, we use contrasts to examine them. Note that to use contrasts, it is necessary that the questions of interest be formulated before examining the data. It is cheating to make up the questions after analyzing the data.

If we have no specific relationships among the means in mind before looking at the data, we instead use a multiple-comparisons procedure to determine which pairs of population, or group, means differ significantly. We explore both contrasts and multiple comparisons in the next section.

When analyzing data, the following equation reminds us that we look for an overall pattern in the data and deviations from it:

In the regression model, the FIT was the population regression equation, and the RESIDUAL represented the deviations of the data from it. We now apply this framework to describe the statistical models used in one-way ANOVA. These models provide a convenient way to summarize the conditions that are the foundation for our analysis. They also give us the necessary notation to describe the calculations needed.

Recall the statistical model for a random sample of observations from a single Normal population with mean μ and standard deviation σ. If the observations are

we can describe this model by saying that the xj are an SRS from the N(μ, σ) distribution. Another way to describe the same model is to think of the x’s varying about their population mean. To do this, write each observation xj as

The ϵj are then an SRS from the N(0, σ) distribution. Because μ is unknown, the ϵ’s cannot actually be observed. This form more closely corresponds to our

way of thinking. The FIT part of the model is represented by μ. It is the systematic part of the model, like the line in a regression. The RESIDUAL part is represented by ϵj. It represents the deviations of the data from the fit and is due to random, or chance, variation.

There are two unknown parameters in this statistical model: μ and σ. We estimate μ by x¯, the sample mean, and σ by s, the sample standard deviation. The differences ej=xj−x¯ are the residuals and correspond to the ϵj in the statistical model.

The model for one-way ANOVA is very similar. We take random samples from each of I different populations. The sample size is ni for the ith population. Let xij represent the jth observation from the ith population. The I population means are the FIT part of the model and are represented by μi. The random variation, or RESIDUAL, part of the model is represented by the deviations ϵij of the observations from the means.

Note that the sample sizes ni may differ (as they do in our example), but the standard deviation σ is assumed to be the same in all the populations. Figure 12.5 pictures this model for I=3. The three population means μi are different, but the shapes of the three Normal distributions are the same, reflecting the assumption that all three populations have the same standard deviation.

The ANOVA model assumes that these deviations ϵij are independent and Normally distributed with mean 0 and standard deviation σ. For our example, we have clear evidence that the data are non-Normal as the data are integer values. However, because our inference is based on the sample means, which will be approximately Normally distributed, we are not overly concerned about this violation of model conditions. We only need the model conditions to be approximately met in order for ANOVA to provide valid results.

The unknown parameters in the statistical model for ANOVA are the I population means μi and the common population standard deviation σ. To estimate μi, we use the sample mean for the ith group:

The residuals eij=xij−x¯i reflect the variation about the sample means that we see in the data and are used in the calculations of the sample standard deviations

In addition to the deviations being Normally distributed, the one-way ANOVA model also states that the population standard deviations are all equal. Before estimating σ, it is important to check this equality condition using the sample standard deviations. Most statistical software provides at least one test for the equality of standard deviations. Unfortunately, many of these tests lack robustness against non-Normality.

Because one-way ANOVA procedures are not extremely sensitive to unequal standard deviations, especially when the group sample sizes are similar, we do not recommend a formal test of equality of standard deviations as a preliminary to the ANOVA. Instead, we will use the following rule of thumb.

If there is evidence that we have unequal group standard deviations, we generally try to transform the data so that they are approximately equal. We might, for example, work with xij or log xij. Fortunately, we can often find a transformation that both makes the group standard deviations more nearly equal and also makes the distributions of observations in each group more nearly Normal. If the standard deviations are markedly different and cannot be made similar by a transformation, inference requires different methods, such as the nonparametric methods described in Chapter 15 and the bootstrap described in Chapter 16.

When we assume that the population standard deviations are equal, each sample standard deviation is an estimate of σ. To combine these into a single estimate, we use a generalization of the pooling method introduced in Chapter 7 (page 423).

Pooling gives more weight to groups with larger sample sizes. If the sample sizes are equal, sp2 is just the average of the I sample variances. Note that sp is not the average of the I sample standard deviations.

Comparison of several means is accomplished by using an F statistic to compare the variation among groups with the variation within groups. We now show how the F statistic expresses this comparison. Calculations are organized in an ANOVA table, which contains numerical measures of the variation among groups and within groups.

First, we must specify our hypotheses for one-way ANOVA. As before, I represents the number of populations to be compared.

Before proceeding to inference, we should verify the conditions necessary for the one-way ANOVA results to be trusted. There is no point in doing this analysis if the data do not, at least approximately, meet these conditions.

Example 12.8 Verifying the conditions for ANOVA.

We’ve discussed each of these conditions already. Here is a summary of our assessments for the music study.

SRSs. This is an experiment in which participants were randomly assigned to groups. We clearly have SRSs, but defining the population from which we obtained the participants is challenging. They were all recruited from MTurk and were paid to participate. Can we act as if they were randomly chosen from the general population of adults? People may disagree on the answer.

Normality. Are the number of problems completed Normally distributed in each group? We’ve already argued they are not Normally distributed, but because inference is based on the sample means and we have very large sample sizes, we are not concerned about violating this condition. The exception would be if there were outliers or extreme skewness. Figure 12.6 displays the Normal quantile plots for each of the three groups. There are no outliers and no strong departures from Normality.

Common standard deviation. Are the population standard deviations equal? Because the largest sample standard deviation is not more than twice the smallest sample standard deviation, we can proceed, assuming that the population standard deviations are equal.

Having determined that the data approximately satisfy these three conditions, we proceed with inference.

Example 12.9 Reading software output.

Figure 12.7 shows descriptive statistics generated by JMP for the music study. Summaries for each background music group are given on the first three lines. In addition to the sample size, the mean, and the standard deviation, this output also gives the minimum and maximum observed values and the standard error of the mean of each group. The three sample means x¯i given in the output are estimates of the three unknown population means μi.

The output gives the estimates of the standard deviations, si, for each group but does not provide sp, the pooled estimate of the common standard deviation, σ. We could perform the calculation using a calculator, as we did in Example 12.7. We will see an easier way to obtain this quantity from the ANOVA table.

Some software packages report sp as part of the standard ANOVA output. Sometimes you are not sure whether or not a quantity given by software is what you think it is. A good way to resolve this dilemma is to do a sample calculation with a simple example to check the numerical results.

Note that sp is not the standard deviation given in the “All” row of Figure 12.7. This quantity is the standard deviation that we would obtain if we viewed the data as a single sample of 447 participants and ignored the possibility that the group means could be different. As we have mentioned many times before, it is important to use care when reading and interpreting software output.

Example 12.10 Reading software output (continued).

Output for the one-way ANOVA of the music study is given in Figure 12.8. We will discuss the construction of this output next. For now, we observe that the results of our significance test are given in the last two columns of the “Analysis of Variance” output. The null hypothesis that the three group means are the same is tested by the statistic F=3.039, and the associated P-value is reported as P=0.0489. The data provide evidence that barely passes the α=0.05 threshold. Although we can conclude that there are some differences among the three group means, we may not be comfortable generalizing the results beyond the population of MTurk workers. We also may question whether a difference of completing just one additional problem is really very meaningful or has much impact, given the level of variation within groups.

For one-way ANOVA, the model is

Much as with linear regression, we can think of the information in the one-way ANOVA table in terms of our

view of statistical models. It summarizes the separation of the total variation and degrees of freedom in the data into two parts: the part due to the fit and the remainder, which we call residual.

The Grp row in the table gives information related to the variation among groups—in particular, their means. In expressing a general one-way ANOVA table, we will use the generic label “Groups.” The Error row in the table gives information related to the variation within groups. Although the term “Error” is frequently used for this source of variation, it is most appropriate for experiments in the physical sciences where the observations within a group differ because of measurement error. In business and the biological and social sciences, the within-group variation is often due to the fact that not all firms or plants or people are the same. This sort of variation is not due to errors and is better described as “residual” or “within-group” variation. Nevertheless, we will use the generic label “Error” for this source of variation in the general one-way ANOVA table.

Finally, the C. Total row in the one-way ANOVA table corresponds to the DATA term in our view of statistical models. So, for one-way analysis of variance,

translates into

The second and third columns in the software output given in Figure 12.8 are labeled “DF” and “Sum of Squares.” As you might expect, each sum of squares is a sum of squared deviations. We use SSG, SSE, and SST for the entries in this column, corresponding to groups, error, and total. Each sum of squares measures a different type of variation. SST measures variation of the data around the overall mean, xij − x¯. SSG measures variation of the group means around the overall mean, x¯i − x¯. Finally, SSE measures variation of each observation around its group mean, xij − x¯i.

Associated with each sum of squares is a quantity called the degrees of freedom, labeled “DF.” Because SST measures the variation of all N observations around the overall mean, its degrees of freedom are DFT=N−1. This is the same as the degrees of freedom for the ordinary sample variance with sample size N. Similarly, because SSG measures the variation of the I sample means around the overall mean, its degrees of freedom are DFG=I−1. Finally, SSE is the sum of squares of the deviations xij − x¯i. Here we have N observations being compared with I sample means, and DFE=N − I.

In this music study, most of the variation is coming from within groups. For a one-way analysis of variance, we define the coefficient of determination as

The coefficient of determination plays the same role as the squared multiple correlation R2 in a multiple regression. We can easily calculate the value from the one-way ANOVA table entries.

Only 1.4% of the variation in the number of completed math problems is explained by the different background music levels. The other 98.6% of the variation is due to participant-to-participant variation within each of the groups. We can see this in the boxplots of Figure 12.4 (page 604). Each of the groups has a large amount of variation, and there is a substantial amount of overlap in their distributions.

To assess whether the observed differences in sample means are statistically significant, some additional calculations are needed. For each source of variation, the mean square is the sum of squares divided by the degrees of freedom. You can verify this by doing the divisions for the values given on the output in Figure 12.8. We compare these mean squares to test whether the population means are all the same.

We can use the mean square for error to find sp, the pooled estimate of the parameter σ of our model. It is true in general that

In other words, the mean square for error is an estimate of the within-group variance, σ2. The estimate of σ is, therefore, the square root of this quantity. So,

If H0 is true, there are no differences among the group means. The ratio MSG/MSE is a statistic that is approximately 1 if H0 is true and tends to be larger if Ha is true. This is the one-way ANOVA F statistic. In our example, MSG=58.2084 and MSE=19.1557, so the ANOVA F statistic is

When H0 is true, the F statistic has an F distribution that depends upon two numbers: the degrees of freedom for the numerator and the degrees of freedom for the denominator. For one-way ANOVA, the degrees of freedom for the numerator are DFG=I−1, and the degrees of freedom for the denominator are DFE=N−I. We use the notation F(I−1,N−I) for this distribution. When determining the P-value, remember that the F test is always one-sided because any differences among the group means tend to make F large.

Even though the group means explain only 1.4% of the total variability, the F test is statistically significant at the α=0.05 level. This example serves as a reminder that the F test is comparing the means. A significant F test does not necessarily mean that the distributions of values are far apart.

Using the One-Way ANOVA applet is an excellent way to see how the value of the F statistic and the P-value depend upon the variability of the data within the groups, the sample sizes, and the differences between the means. See Exercises 12.44, 12.45, and 12.46 (page 642) for use of this applet.

The one-way ANOVA F test shares the robustness of the two-sample t test. It is relatively insensitive to moderate non-Normality and unequal variances, especially when the sample sizes are similar. The constant variance assumption is more important when we compare means using contrasts and multiple comparisons, which are additional analyses that are performed after the ANOVA F test. We discuss these analyses in the next section.

The following display shows the general form of a one-way ANOVA table with the F statistic. The formulas in the sum of squares column can be used for calculations in small problems. There are other formulas that are more efficient for hand or calculator use, but ANOVA calculations are usually done by computer software.

We used JMP to illustrate the analysis of the music study. Other statistical software gives similar output, and you should be able to read it without any difficulty. Here’s an example on which to practice this skill.

Example 12.16 Do eyes affect ad response?

Research from a variety of fields has found significant effects of eye gaze and eye color on emotions and perceptions such as arousal, attractiveness, and honesty. These findings suggest that a model’s eyes may play a role in a viewer’s response to an advertisement.

In one study, students in marketing and management classes of a southern, predominantly Hispanic, university were each presented with one of four portfolios.⁵ Each portfolio contained a target ad for a fictional product, Sparkle Toothpaste. Students were asked to view the ad and then respond to questions concerning their attitudes and emotions about the ad and product. All questions were from advertising-effects questionnaires previously used in the literature. Each response was on a seven-point scale.

Although the researchers investigated nine attitudes and emotions, we will focus on the viewer’s “attitudes toward the brand.” This response was obtained by averaging 10 survey questions.

The target ads were created using two digital photographs of a model. In one picture, the model is looking directly at the camera so the eyes can be seen. This picture was used in three target ads. The only difference was the model’s eyes, which were made to be either brown, blue, or green. In the second picture, the model is in virtually the same pose but looking downward so the eyes are not visible. A total of 222 surveys were used for analysis. Outputs from Excel, JMP, and Minitab are given in Figure 12.9.

There is evidence at the 5% significance level to reject the null hypothesis that the four groups have equal means (P=0.036). In Exercises 12.27 and 12.28 (page 639), you are asked to perform further inference using contrasts.

This test uses a more robust measure of deviation, replacing the mean with the median and replacing squaring with the absolute value. Also, the creation of the transformed response variable is straightforward, so this test can be performed regardless of whether your software specifically has it.

Example 12.17 Are the standard deviations equal?

Figure 12.10 shows output of the modified Levene’s test for the advertisement study. In JMP, the test is called Brown–Forsythe. The P-value is 0.6775, which is much larger than α=0.05, suggesting that we cannot reject the null hypothesis that they are the same. This result further supports the use of ANOVA for the advertisement study.

Remember that our rule of thumb (page 608) is used to assess whether different standard deviations will impact the ANOVA results. It’s not a formal test that the standard deviations are equal. There will be times when the rule and formal test do not agree.