If Student Monitor took a second random sample of 1200 students, the new sample would have different undergraduates in it. It is almost certain that the sample mean x¯ would not again be 15.1. Likewise, we would not expect there to be exactly 1102 students who own a smartphone. In other words, the value of a statistic will vary from sample to sample. This basic fact is called sampling variability: the value of a statistic varies in repeated random sampling.

Random samples eliminate any preferences or favoritism from the act of choosing a sample, but they can still be misleading because of this variability. For example, what if a second random sample of 1200 undergraduates resulted in only 64% of the students owning a smartphone? Do these two results, 92% and 64%, leave you more or less confident in the value of the true population proportion? When sampling variability is too great, we can’t trust the results of any one sample.

We can assess this variability by using the second advantage of random samples (the first advantage being the elimination of bias). Specifically, if we take lots of random samples of the same size from the same population, the variation from sample to sample will follow a predictable pattern. Statistical inference is based on one idea: to see how trustworthy a procedure is, ask what would happen if we repeated it many times.

To understand the sampling variablility of a statistic, we ask, “What would happen if we took many samples?” Here’s how to answer that question for any statistic:

1. Take a large number of random samples of size n from the same population.
2. Calculate the statistic for each sample.
3. Make a histogram of the values of the statistic.
4. Examine the distribution displayed in the histogram for shape, center, and spread, as well as outliers or other deviations.

In practice, it is too expensive to take many samples from a large population such as all undergraduates enrolled in four-year colleges and universities. But we can imitate taking many samples by using random digits from a table or computer software to emulate chance behavior. This is called simulation. It is a very powerful tool for studying chance variation.

Now that we see how simulation works, it is faster to abandon Table B and to use a computer to generate random numbers. This allows us to easily consider a larger sample size, like n=1200 used in Example 5.1. This also allows us to study other statistics, such as the sample mean, when the population cannot be easily imitated by a table of random digits.

Example 5.3 Take many random samples.

Figure 5.1 illustrates the process of choosing many samples and finding the statistic p^ for each one. Follow the flow of the figure from the population at the left, to choosing an SRS of size n=100 and finding the p^ for this sample, to collecting together the p^’s from many samples. The histogram at the right of the figure shows the distribution of the values of p^ from 1000 separate SRSs of size 100 drawn from a population with p=0.9.

Of course, Student Monitor samples 1200 students, not just 100. Figure 5.2 is parallel to Figure 5.1. It shows the process of choosing 1000 SRSs, each of size 1200, from a population in which the true proportion is p=0.9. The 1000 values of p^ from these samples form the histogram at the right of the figure. Figures 5.1 and 5.2 are drawn on the same x scale. Comparing them shows what happens when we increase the size of our samples from 100 to 1200. These histograms display the sampling distribution of the statistic p^ for two sample sizes.

Strictly speaking, the sampling distribution is the ideal pattern that would emerge if we looked at all possible samples of size n (here, 100 or 1200) from our population. A distribution obtained from a fixed number of trials, like the 1000 trials in Figures 5.1 and 5.2, is only an approximation to the sampling distribution. We will see that probability theory, the mathematics of chance behavior, can sometimes describe sampling distributions exactly. The interpretation of a sampling distribution is the same, however, whether we obtain it by simulation or by the mathematics of probability.

We can use the tools of data analysis to describe any distribution. Let’s apply those tools to Figures 5.1 and 5.2.

• Shape: The histograms look Normal. Figure 5.3 is a Normal quantile plot of the values of p^ for our samples of size 100. It confirms that the distribution in Figure 5.1 is close to Normal. The 1000 values for samples of size 1200 in Figure 5.2 are even closer to Normal. The Normal curves drawn through the histograms describe the overall shape quite well.

• Center: In both cases, the values of the sample proportion p^ vary from sample to sample, but the values are centered at 0.9. Recall that p=0.9 is the true parameter value. Some samples have a p^ less than 0.9 and some greater, but there is no tendency to be always lower or always higher. That is, p^ has no bias as an estimator of p. This is true for both large and small samples. (Want the details? The mean of the 1000 values of p^ is 0.9000 for samples of size 100 and 0.9003 for samples of size 1200. The median value of p^ is exactly 0.9 for samples of both sizes.)

• Spread: The values of p^ from samples of size 1200 are much less spread out than the values from samples of size 100. In fact, the standard deviations are 0.0295 for Figure 5.1 and 0.0085 for Figure 5.2.

Although these results describe just two sets of simulations, they reflect facts that are true whenever we use random sampling.

Our simulations show that a sample of size 1200 will almost always give an estimate p^ that is close to the truth about the population. Figure 5.2 illustrates this fact for just one value of the population proportion (p=0.9), but it is true for any proportion. That is a primary reason Student Monitor uses a sample of size of 1200. There is more sampling variability the smaller the sample size. For example, a sample of size 100 (Figure 5.1) might give an estimate of 82% or 98% when the truth is 90%.

Thinking about Figures 5.1 and 5.2 helps us restate the idea of bias when we use a statistic like p^ to estimate a parameter like p. It also reminds us that variability matters as much as bias.

We can think of the true value of the parameter as the bull’s-eye on a target and of the sample statistic as an arrow fired at the bull’s-eye. Bias and variability describe what happens when an archer fires many arrows at the target. Bias means that the aim is off, and the sample values do not center about the population value. High variability means that sample values are widely scattered. In other words, there is a lack of precision, or consistency, among the sample values. Figure 5.4 shows this target illustration of the two types of error.

Notice that low variability (repeated shots are close together) can accompany high bias (the arrows are consistently away from the bull’s-eye in one direction). And low bias (the arrows center on the bull’s-eye) can accompany high variability (repeated shots are widely scattered). A good sampling scheme, like a good archer, must have both low bias and low variability. Here’s how we ensure this.

In practice, Student Monitor takes only one random sample. We don’t know how close to the truth the estimate from this one sample is because we don’t know the value of the true parameter. But large random samples almost always give an estimate that is close to the truth. Looking at the pattern of many samples when n=1200 shows that we can trust the result of one sample.

Similarly, the Current Population Survey’s sample of about 60,000 households estimates the national unemployment rate very accurately. Of course, only probability samples carry this guarantee. Using a probability sampling design and taking care to deal with practical difficulties reduce bias in a sample.

The size of the sample then determines how close to the population truth the sample result is likely to fall. Results from a sample survey usually come with a margin of error that sets bounds on the size of the likely error.

Because the margin of error directly reflects the variability of the sample statistic, it is smaller for larger samples. In later chapters, we will describe the details of its calculation for many different statistics and how, when planning a survey or experiment, it is used to determine the sample size n.

Student Monitor’s sample of 1200 students is only about 1 out of every 9000 full-time undergraduate students in the United States. Does it matter whether we sample 1-in-1000 individuals in the population or 1-in-9000?

Why does the size of the population have little influence on the behavior of statistics from random samples? To see why this is plausible, imagine sampling harvested grain by thrusting a scoop into a lot of grain kernels. The scoop doesn’t know whether it is surrounded by a bag of grain or by an entire truckload. As long as the grain is well mixed (so that the scoop selects a random sample), the variability of the result depends only on the size of the scoop.

The fact that the variability of sample results is controlled by the size of the sample has important consequences for sampling design. An SRS of size 1200 from the 10.8 million undergraduates gives results as precise as an SRS of size 1200 from the roughly 85,000 inhabitants of San Francisco between the ages of 18 and 24. This is good news for designers of national samples but bad news for those who want accurate information about these citizens of San Francisco. If both use an SRS, both must use the same size sample to obtain equally trustworthy results.

Why randomize? The act of randomizing guarantees that the results of analyzing our data are subject to the laws of probability. The behavior of statistics is described by a sampling distribution. The form of the distribution is known and, in many cases, is approximately Normal. Often, the center of the distribution lies at the true parameter value so that the notion that randomization eliminates bias is made more explicit. The spread of the distribution describes the variability of the statistic and can be made as small as we wish by choosing a large enough sample.

These facts are at the heart of formal statistical inference. The remainder of this chapter has much to say in more technical language about sampling distributions. Later chapters describe the way statistical conclusions are based on them. What any user of statistics must understand is that all the technical talk has its basis in a simple question: “What would happen if the sample or the experiment were repeated many times?” The reasoning applies not only to an SRS but also to the complex sampling designs actually used by opinion polls and other national sample surveys. The same conclusions hold as well for randomized experimental designs. The details vary with the design, but the basic facts are true whenever randomization is used to produce data.

Remember that proper statistical design is not the only aspect of a good sample or experiment. The sampling distribution shows only how a statistic varies due to the operation of chance in randomization. It reveals nothing about possible bias due to undercoverage or nonresponse (page 185) in a sample or to lack of realism in an experiment. The actual error in estimating a parameter by a statistic can be much larger than the sampling distribution suggests. What is worse, there is no way to say how large the added error is. The real world is less orderly than statistics textbooks imply.

In the next two sections, we will study the sampling distributions of two common statistics: the sample mean and the sample proportion. The focus will be on the important features of these distributions so that we can quickly describe and use them in the later chapters on statistical inference. We will see that, in each case, the sampling distribution depends on both the population and the way we collect the data from the population.