It is tempting to simply draw conclusions from our own experience, making no use of more broadly representative data. Stories or reports based on the ideas and opinions of individuals or groups do not provide the kind of data that can be used in a statistical analysis to draw conclusions.

These types of stories, or anecdotes, sometimes provide data. However, a common characteristic is that this type of data does not give us a sound basis for drawing conclusions.

Not all anecdotal data are bad. The experiences of an individual or a small group of individuals might suggest an interesting study that could be performed using more carefully collected data. The anecdotal data mentioned in Example 3.2 could lead to a carefully designed study to evaluate potential treatments.

Occasionally, data are collected for a particular purpose but can also serve as the basis for drawing sound conclusions about other research questions. We use the term available data for this type of data.

The library and the Internet can be good sources of available data. Because producing new data is expensive, we all use available data whenever possible. Here are two examples.

Example 3.3 Unemployment rates.

If you visit the U.S. Bureau of Labor Statistics website, bls.gov, you will find many interesting sets of data and statistical summaries (Figure 3.1). A recent posting in the Student Resources gave a graphic of employment data by state.¹

Many nations have a single national statistical office, such as Statistics Canada (statcan.gc.ca) and Mexico’s INEGI (inegi.org.mx). More than 70 different U.S. agencies collect data. You can reach most of them through the U.S. government’s data site (usa.gov/statistics).

The validity of our conclusions from the analysis of data rests on a foundation of carefully collected data. We could use a survey of college athletes to estimate the percent who gamble. An experiment could be used to determine if restaurant patrons give higher tips when their server repeats their order carefully.

In this chapter, we will develop the skills needed to produce trustworthy data and to judge the quality of data produced by others. The techniques for producing data that we will study require no formulas, but they are among the most important ideas in statistics. Statistical designs for producing data rely on either sampling or experiments.

How have the attitudes of Americans, on issues ranging from abortion to work, changed over time? Sample surveys are the usual tool for answering questions like these.

The GSS selects a sample of adults to represent the larger population of all English-speaking and Spanish-speaking adults living in the United States. The idea of sampling is to study a part in order to gain information about the whole. Data are often produced by sampling a population of people or things. Opinion polls, for example, may report the views of the entire country based on interviews with a sample of about 1000 people. Government reports on employment and unemployment are produced from a monthly sample of about 60,000 households. The quality of manufactured items is monitored by inspecting small samples each hour or each shift.

In all our examples, the expense of examining every item in the population makes sampling a practical necessity. Timeliness is another reason for preferring a sample to a census, which is an attempt to contact every individual in the population. We want information on current unemployment and public opinion next week, not next year. Moreover, a carefully conducted sample is often more accurate than a census. Accountants, for example, sample a firm’s inventory to verify the accuracy of the records. Attempting to count every last item in the warehouse would be not only expensive but inaccurate. Bored people do not count carefully.

A sample survey collects information about a population by selecting and measuring a sample from the population. The goal is to get a picture of the population, disturbed as little as possible by the act of gathering information. Sample surveys are one kind of observational study.

What can we conclude from this study? If you start to play baseball, will you have stronger bones in your throwing arm?

We can imagine an experiment that would remove these difficulties. From a large group of subjects, require some to play baseball and forbid the rest from playing. This is an experiment because the condition (playing baseball or not) is imposed on the subjects. Of course, this particular experiment is neither practical nor ethical.

An observational study, even one based on a carefully chosen sample, is a poor way to determine what will happen if we change something. The best way to see the effects of a change is to do an intervention—where we actually impose the change. When our goal is to understand cause and effect, experiments are the only source of fully convincing data.

Confounding occurs when an explanatory variable is related to one or more other variables that have an influence on the response variable. When this happens, we sometimes attribute a relationship to an explanatory variable when the effect is fully or partly due to the confounding variables.

In Example 3.7, the effect of baseball playing on arm bone strength is confounded with (mixed up with) other characteristics of the subjects in the study. Observational studies that examine the effect of a single variable on an outcome can be misleading when the effects of the explanatory variable are confounded with those of other variables.

Because experiments allow us to isolate the effects of specific variables, we generally prefer them. Here is an example.

Experiments usually require some sort of randomization. We begin the discussion of statistical designs for data collection in Section 3.2 with the principles underlying the design of experiments.