The unbiasedness of an estimator concerns the center of its sampling distribution, but questions about variation are answered by looking at its spread. The central limit theorem says that if the entire population of SATM scores has mean μ and standard deviation σ, then in repeated SRSs of size 500, the sample mean x¯ is approximately N(μ, σ/500). Let us suppose that we know that the standard deviation σ of SATM scores in our California population is σ=100. (We will see in the next chapter how to proceed when σ is not known. For now, we are more interested in statistical reasoning than in details of realistic methods.) This means that in repeated sampling the sample mean x¯ has an approximately Normal distribution centered at the unknown population mean μ and a standard deviation of

Now we are ready to proceed. Consider this line of thought, which is illustrated in Figure 6.2:

• The 68–95–99.7 rule says that the probability is about 0.95 that x¯ will be within 9 points (that is, 2 standard deviations of x¯) of the population mean μ.
• To say that x¯ lies within 9 points of μ is the same as saying that μ is within 9 points of x¯.
• So about 95% of all samples will contain the true μ in the interval from x¯−9 to x¯+9.

We have simply restated a fact about the sampling distribution of x¯. The language of statistical inference uses this fact about what would happen in the long run to express our confidence in the results of any one sample. Our sample gave x¯=505. We say that we are 95% confident that the unknown mean score for all California seniors lies between

and

Be sure you understand the grounds for our confidence. There are only two possibilities for our SRS:

1. The interval between 496 and 514 contains the true μ.
2. The interval between 496 and 514 does not contain the true μ.

We cannot know whether our sample is one of the 95% for which the interval x¯±9 contains μ or one of the unlucky 5% for which it does not contain μ. The statement that we are 95% confident is shorthand for saying, “We arrived at these numbers by a method that gives correct results 95% of the time.”

In the setting of Example 6.3, the interval of numbers between the values x¯±9 is called a 95% confidence interval for μ. Like most confidence intervals we will discuss, this one has the form

The estimate (x¯=505 in this case) is our guess for the value of the unknown parameter. The margin of error (9 here) reflects how accurate we believe our guess is, based on the variability of the estimate, and how confident we are that the procedure will produce an interval that will contain the true population mean μ.

Figure 6.3 illustrates the behavior of 95% confidence intervals in repeated sampling from a Normal distribution with mean μ. The center of each interval (marked by a dot) is at x¯ and varies from sample to sample. The sampling distribution of x¯ (also Normal) appears at the top of the figure to show the long-term pattern of this variation.

The 95% confidence intervals, x¯± margin of error, from 25 SRSs appear below the sampling distribution. The arrows on either side of the dot (x¯) span the confidence interval. All except 1 of the 25 intervals contain the true value of μ. In those intervals that contain μ, sometimes μ is near the middle of the interval and sometimes it is closer to one of the ends. This again reflects the variation of x¯. In practice, we don’t know the value of μ, but we have a method such that, in a very large number of samples, 95% of the confidence intervals will contain μ.

We can construct confidence intervals for many different parameters based on a variety of designs for data collection. We will learn the details of a number of these in later chapters. Two important things about a confidence interval are common to all settings:

1. It is an interval of the form (a, b), where a and b are numbers computed from the sample data.
2. It has a property called a confidence level that gives the probability of producing an interval that contains the unknown parameter.

Users can choose the confidence level, but 95% is the standard for most situations. Occasionally, 90% or 99% is used. We use C to stand for the confidence level in decimal form. For example, a 95% confidence level corresponds to C=0.95.

With the Confidence Intervals applet, you can construct diagrams similar to the one displayed in Figure 6.3. The only differences are that the applet displays both the Normal population distribution and the Normal sampling distribution of x¯ at the top and that at most 20 confidence intervals are displayed, with the data that make up the last SRS displayed below its confidence interval. To use the applet, choose the confidence level C, the sample size n, and how many SRSs (between 1 and 100) to generate. The number (and percent) of intervals that contain μ is displayed.

The spread of the data for the latest SRS reflects the spread of the population distribution. This spread is assumed known, and it does not change with sample size. What does change, as you vary n, is the spread of the sampling distribution of x¯; this is expressed through the margin of error. As you increase n, you’ll find that the span of the confidence interval gets smaller. More on this property soon.

We now construct a level C confidence interval for the mean μ of a population when the data are an SRS of size n. The construction is based on the sampling distribution of the sample mean x¯. This distribution is exactly N(μ, σ/n) when the population has the N(μ, σ) distribution. The central limit theorem says that this same sampling distribution is approximately correct for large samples. For now, we will assume we are in one of these two situations. We discuss what we mean by “large sample” after we briefly study these intervals.

Our construction of a 95% confidence interval for the mean SATM score began by noting that any Normal distribution has probability about 0.95 within ±2 standard deviations of its mean. To construct a level C confidence interval, we first catch the central C area under a Normal curve. That is, we must find the number z* such that any Normal distribution has probability C within ±z* standard deviations of its mean.

Because all Normal distributions have the same standardized form, we can obtain everything we need from the standard Normal curve. Figure 6.4 shows how C and z* are related. Values of z* for many choices of C appear in the row labeled z* at the bottom of Table D at the back of the book. Here are the most important entries from that row:

Notice that for 95% confidence, the value 2 obtained from the 68–95–99.7 rule is replaced with the more precise 1.96.

As Figure 6.4 reminds us, any Normal curve has probability C between the point z* standard deviations below the mean and the point z* standard deviations above the mean. The sample mean x¯ has the Normal distribution with mean μ and standard deviation σ/n, so there is probability C that x¯ lies between

This is exactly the same as saying that the unknown population mean μ lies between

That is, there is probability C that the interval x¯±z*σ/n contains μ. This is our confidence interval. The estimate of the unknown μ is x¯, and the margin of error is z*σ/n.

Since 2008, Sallie Mae, a major provider of education loans and savings programs, has conducted an annual study titled “How America Pays for College.” In the 2019 survey, 2000 randomly selected individuals (1000 parents of undergraduate students and 1000 undergraduate students) were surveyed online.⁴

Many of the survey questions focus on the composition of funding sources used to pay for college, so the undergraduates in the survey are often responding for their family. For example, each participant is asked to report how much of the parents’ current income is used to pay for college. Do you think it is wise to combine responses across the parents and undergraduates? Are you fully aware of how much money your parents are spending and borrowing for college? The authors consider this one population and report overall averages and percents in their report. We will also consider this a sample from one population, but this is certainly debatable.

Example 6.4 Average college savings fund contribution.

One Sallie Mae survey question asked how much money from a college savings fund, such as a 529 plan, is used to pay for college. Of the 2000 who were surveyed, the average amount is $1577. This population distribution is highly skewed to the right, with many zeros and likely a few very large amounts. Nevertheless, because the sample size is quite large, we can rely on the central limit theorem to assure us that the confidence interval based on the Normal distribution will be a good approximation.

Let’s compute an approximate 95% confidence interval for the true mean amount contributed from a college savings fund among all undergraduates. We’ll assume that the standard deviation for the population of college savings fund contributions is $2076. For 95% confidence, we see from Table D that z*=1.960. The margin of error for the 95% confidence interval for μ is, therefore,

m=z*σn=1.96020762000=90.98

We have computed the margin of error with more digits than we really need. Our mean is rounded to the nearest $1, so we will do the same for the margin of error. Keeping additional digits would provide no additional useful information. Therefore, we will use m=91. The approximate 95% confidence interval is

x¯±m=1577±91=(1486, 1668)

We are 95% confident that the mean amount contributed from a college savings fund among all undergraduates is between $1486 and $1668.

Suppose that the researchers who designed this study had used a different sample size. How would this affect the confidence interval? We can answer this question by changing the sample size in our calculations and assuming that the sample mean is the same.

Notice that the margin of error for this example is twice as large as the margin of error that we computed in Example 6.4. The only change that we made was to assume that the sample size is 500 rather than 2000. This sample size is one-fourth of the original 2000. Thus, we double the margin of error when we reduce the sample size to one-fourth of the original value. Figure 6.5 illustrates the effect in terms of the intervals.

The argument leading to the form of confidence intervals for the population mean μ rested on the fact that the sampling distribution of the statistic x¯ used to estimate μ is Normal. Because many sample estimates have Normal (at least approximately), it is useful to notice that the confidence interval has the form

The estimate based on the sample is the center of the confidence interval. The margin of error is z*σestimate. The desired confidence level determines z* from Table D. The standard deviation of the estimate is found from knowledge of the sampling distribution in a particular case. When the estimate is x¯ from an SRS, the standard deviation of the estimate is σestimate=σ/n. We return to this general form numerous times in the following chapters.

The margin of error z*σ/n for the mean of a Normal population illustrates several important properties that are shared by all confidence intervals in common use. The user chooses the sample size and confidence level, and the margin of error follows from these choices.

Both high confidence and a small margin of error are desirable characteristics of a confidence interval. High confidence says that our method almost always gives correct answers. A small margin of error says that we have pinned down the parameter quite precisely.

Suppose that in planning a study, you calculate the margin of error and decide that it is too large. Here are your choices to reduce it:

• Use a lower level of confidence (smaller C).
• Choose a larger sample size (larger n).
• Reduce σ.

For most problems, you would choose a confidence level of 90%, 95%, or 99%, so z* will be 1.645, 1.960, or 2.576, respectively. Figure 6.4 (page 335) shows that z* will be smaller for lower confidence (smaller C). The bottom row of Table D also shows this. If n and σ are unchanged, a smaller z* leads to a smaller margin of error.

Example 6.6 How the confidence level affects the confidence interval.

Suppose that for the college saving fund contribution data in Example 6.4 (page 336), we wanted 99% confidence. Table D tells us that for 99% confidence, z*=2.576. The margin of error for 99% confidence based on 2000 observations is

m=z*σn=2.57620762000=119.58

and the 99% confidence interval is

x¯±m=1577±120=(1457, 1697)

Requiring 99%, rather than 95%, confidence has increased the margin of error from 91 to 120. Figure 6.6 compares the two intervals.

Similarly, choosing a larger sample size n reduces the margin of error for any fixed confidence level. The square root in the formula implies that we must multiply the number of observations by 4 in order to cut the margin of error in half. Likewise, if we want to reduce the standard deviation of x¯ by a factor of 4, we must take a sample 16 times as large.

The standard deviation σ measures the variation in the population. You can think of the variation among individuals in the population as noise that obscures the average value μ. It is harder to pin down the mean μ of a highly variable population; that is why the margin of error of a confidence interval increases with σ. In practice, we can sometimes reduce σ by carefully controlling the measurement process. We also might change the mean of interest by restricting our attention to only part of a large population. Focusing on a subpopulation (e.g., women in rural counties) will typically result in a smaller σ. The trade-off, however, is less generalizable results.

A wise user of statistics never plans data collection without, at the same time, planning the inference. You can arrange to have both high confidence and a small margin of error through the margin of error formula for a population mean

By rearranging this equation, we can find the sample size n for a desired margin of error m. Here is the result.

Recall that the value of σ is known, and the value z* comes from the choice of confidence level C. Notice also that the size of the population (as long as the population is much larger than the sample) does not influence the sample size we need.

A drawback of this formula is that it does not account for collection costs. In practice, taking observations costs time and money. The required sample size may be impossibly expensive. If you face such a situation, you might consider a larger margin of error and/or a lower confidence level to find a workable sample size.

This formula also does not adjust for the fact that the actual number of usable observations is often less than what is planned at the beginning of a study. This is particularly true of data collected in surveys but is an important consideration in most studies. Careful study designers often assume a nonresponse rate or dropout rate that specifies what proportion of the originally planned sample will fail to provide data. This information is used to adjust the sample size to be used at the start of the study.

If past studies suggested that 70% of those contacted will respond, we would need to start with a sample size of 6623/0.7=9642 undergraduates to expect usable information from 6623 of them. Accounting for nonresponse often results in a much larger initial sample size.

The purpose of these calculations is to determine a sample size that is sufficient to provide useful results, but the determination of what is useful is a matter of judgment that requires subject-matter knowledge. In Section 7.3 (page 433), we revisit this issue of sample size determination under the more realistic setting when σ is unknown. Although the calculations are more complex, the design principles are the same. Furthermore, software now does most of these calculations for us.

We have already seen that small margins of error and high confidence can require large numbers of observations. You should also be keenly aware that any formula for inference is correct only in specific circumstances. If the government required statistical procedures to carry warning labels like those on drugs, most inference methods would have long labels. Our formula x¯±z*σ/n for estimating a population mean comes with the following list of warnings for the user:

• The data should be an SRS from the population. However, we are not in great danger if the data can plausibly be thought of as independent observations from a population. That is the case in Examples 6.4 through 6.7, provided that the undergraduates and parents can be considered one population.
• The formula is not correct for probability sampling designs more complex than an SRS. Correct methods for other designs are available. If you plan such samples, be sure that you (or your statistical consultant) know how to carry out the inference you desire.
• There is no correct method for inference from data haphazardly collected with bias of unknown size. Fancy formulas cannot rescue badly produced data.

• Because x¯ is not a resistant measure, outliers can have a large effect on the confidence interval. You should search for outliers and try to correct them or justify their removal before computing the interval. If the outliers cannot be removed, ask your statistical consultant about procedures that are not sensitive to outliers.
• If the sample size is small and the population is not Normal, the true confidence level will be different from the value C used in computing the interval. Prior to doing any calculations, examine your data carefully for skewness and other signs of non-Normality. Remember, though, that the interval relies only on the distribution of x¯, which even for quite small sample sizes is much closer to Normal than is the distribution of the individual observations. When n≥15, the confidence level is not greatly disturbed by non-Normal populations unless extreme outliers or quite strong skewness are present. Our college fund contribution data in Example 6.4 are very skewed, but because of the large sample size, we are confident that the distribution of the sample mean will be approximately Normal.

• The interval x¯±z*σ/n assumes that the standard deviation σ of the population is known. This unrealistic requirement renders the interval of little use in statistical practice. We will learn in the next chapter what to do when σ is unknown. If, however, the sample is large, the sample standard deviation s will be close to the unknown σ. The interval x¯±z*s/n is then an approximate confidence interval for μ.

The most important caution concerning confidence intervals is a consequence of the first of these warnings. The margin of error in a confidence interval covers only random sampling errors. The margin of error is obtained from the sampling distribution and indicates how much error can be expected because of chance variation in randomized data production.

Practical difficulties such as undercoverage and nonresponse in a sample survey cause additional errors. These errors can be larger than the random sampling error. This often happens when the sample size is large (so that σ/n is small). Remember this unpleasant fact when reading the results of an opinion poll or other sample survey. The practical conduct of the survey influences the trustworthiness of its results in ways that are not included in the announced margin of error.

Every inference procedure that we will meet has its own list of warnings. Because many of the warnings are similar to those we have mentioned, we will not print the full warning label each time. It is easy to state (from the mathematics of probability) conditions under which a method of inference is exactly correct. These conditions are never fully met in practice.

For example, no population is exactly Normal. Deciding when a statistical procedure should be used in practice often requires judgment assisted by exploratory analysis of the data. Mathematical facts are, therefore, only a part of statistics. The difference between statistics and mathematics can be stated thusly: mathematical theorems are true; statistical methods are often effective when used with skill.

Finally, you should understand what statistical confidence does not say. Based on our SRS in Example 6.3, we are 95% confident that the mean SATM score for the California students lies between 496 and 514. This says that this interval was calculated by a method that gives correct results in 95% of all possible samples. It does not say that the probability is 0.95 that the true mean falls between 496 and 514. No randomness remains after we draw a particular sample and compute the interval. The true mean either is or is not between 496 and 514. The probability calculations of standard statistical inference describe how often the method, not a particular sample, gives correct answers.