In linear regression, the explanatory variable x is quantitative and can have many different values. Imagine, for example, giving different doses of the calcium supplement x to different groups. We can think of the values of x as defining different subpopulations, one for each possible value of x. Each subpopulation consists of all individuals in the population having the same value of x. If we gave an x=0 mg supplement to one group (control), an x=150 mg supplement to a second, an x=300 mg supplement to a third group, and an x=450 mg supplement to the last group, these four groups would be considered samples from the corresponding four subpopulations.

The statistical model for simple linear regression assumes that for each value of x (or subpopulation), the response variable y is Normally distributed, with a mean that depends on x. We use μy to represent these means. In general, the means μy can change as x changes according to any sort of pattern. In simple linear regression, the means all lie on a line when plotted against x. To summarize, this model also has two important parts:

• The mean blood pressure decrease is different for the different subpopulations of x. The means all lie on a straight line. That is, μy=β0+β1x.
• Individual blood pressure decreases y within each subpopulation of x vary according to a Normal distribution. This variation, measured by the standard deviation σ, is the same for all values of x.

The simple linear regression model is pictured in Figure 10.2. The line describes how the mean response μy changes with x. This is the population regression line. The four Normal curves show how the response y will vary for four different values of the explanatory variable x. Each curve is centered at its mean response μy. All four curves have the same spread, measured by their common standard deviation σ.

The data for a linear regression are the n observed (x, y) pairs. The model takes each x to be a known quantity, like the milligrams of the calcium supplement.¹ The response y for a given x is assumed to be a Normal random variable. The linear regression model describes the mean and standard deviation of this random variable.

We use the following example to explain the fundamentals of simple linear regression. Because regression calculations in practice are done by statistical software, we rely on computer output for the arithmetic. In Section 10.2, we show formulas to do the work with a calculator or spreadsheet. These formulas are useful in understanding analysis of variance (Section 10.2) and multiple regression (Chapter 11).

Example 10.1 Relationship between BMI and physical activity.

Decrease in physical activity is considered to be a major contributor to the increase in prevalence of overweight and obesity in the general adult population. Because the prevalence of physical inactivity among college students is similar to that of the adult population, researchers have tried to understand college students’ physical activity perceptions and behaviors.

In several studies, researchers have looked at the relationship between physical activity (PA) and body mass index (BMI).² For this study, each participant wore a FitBit Flex™ for a week, and the average number of steps taken per day (in thousands) was recorded. Various body composition variables, including BMI in kilograms per square meter, kg/m2, were also measured. We consider a sample of 100 female undergraduates.

We start our analysis with a scatterplot of the data. Figure 10.3 is a plot of BMI versus physical activity for our sample of 100 participants. We use the names BMI and PA for the two quantitative variables. The least-squares regression line is included in the plot. There is a negative association between BMI and PA that appears approximately linear. There is also a considerable amount of scatter about this line.

Always start with a graphical display of the data. There is no point in fitting a linear model if the relationship is not approximately linear. A graphical display can also be used to assess the direction and strength of the relationship and to identify outliers and influential observations.

Before continuing our analysis, it is appropriate to consider the extent to which the results can reasonably be generalized. In the original study, undergraduate volunteers were obtained at a large southeastern public university through classroom announcements and campus flyers. The potential for bias should always be considered when obtaining volunteers. In this study, the volunteers were screened, and those with severe health issues, as well as varsity athletes, were excluded. As a result, the researchers considered the participants as an SRS from the population of undergraduates at this university. However, they acknowledged the risks of generalizing further, stating that similar investigations at universities of different sizes and in other climates of the United States are needed.

In this example, subpopulations are defined by the explanatory variable x, physical activity. The considerable amount of scatter about the least-squares regression line suggests a large amount of variation of BMI for each value of x that is in each subpopulation. Why might this occur? Consider sampling women from your university, each averaging the same number of steps per day—say, 9000. Even though the average number of steps per day is the same, you would not expect all these women to have the same BMI. Variation in other factors, such as genetic makeup, lifestyle, and diet should all contribute to the variation of BMI.

The statistical model for linear regression assumes that these BMI values are Normally distributed, with a mean μy that depends upon x in a linear way. Specifically,

This was displayed in Figure 10.2 with the line and the four Normal curves. The line is the population regression line, which gives the average BMI for all values of x. The Normal curves provide a description of the variation of BMI about these means.

The following equation expresses this idea:

The FIT part of the model consists of the subpopulation means, given by the expression β0+β1x. The RESIDUAL part represents deviations of the data from the line of population means. We assume that these deviations are Normally distributed with mean 0 and standard deviation σ.

We use ε (the lowercase Greek letter epsilon) to stand for the RESIDUAL part of the statistical model. A response y is the sum of its mean and a chance deviation ε from the mean. These model deviations ε represent “noise”—that is, variation in y due to other causes that prevent the observed (x, y)-values from forming a perfectly straight line on the scatterplot.

Before moving on to parameter estimation, there is one additional issue that often deserves attention. The simple linear regression model considers that the x-values are known exactly but, in practice, they are often estimated. For example, in our study, the physical activity level x is estimated using a FitBit Flex worn over a one-week period. If there is error in measuring x, and it is large relative to the spread of the x’s, more advanced inference methods using errors-in-variables models are needed. In this case, the measurement error is expected to be relatively small. Subjects were instructed to continue normal activities and to notify the researchers of any deviations. Also, this estimate of physical activity represents an average over seven days.

The method of least squares presented in Chapter 2 fits a line to summarize a relationship between the observed values of an explanatory variable and a response variable. Now we want to use the least-squares regression line as a basis for inference about a population from which our observations are a sample. In that setting, the slope b1 and intercept b0 of the least-squares line

estimate the slope β1 and the intercept β0 of the population regression line.

This inference should only be done when the linear regression model conditions are reasonably met. Various checks are needed, and some judgment is required. Because additional methods to check these conditions rely on first fitting the model to the data, let’s briefly review the estimation methods of Chapter 2.

Using the formulas from Chapter 2, the slope of the least-squares line is

and the intercept is

Here, r is the correlation between the observed values of y and x, sy is the standard deviation of sample y’s, and sx is the standard deviation of the sample x’s.

Recall that the least-squares regression line always passes through the point (x¯,y¯). Notice also that if the slope b1 is 0, so is the correlation, and vice versa. We discuss this latter connection in more depth later in this chapter.

The remaining parameter to be estimated is σ, which measures the variation of y about the population regression line. More precisely, σ is the standard deviation of the Normal distribution of the deviations εi in the regression model. We don’t observe these εi, so how can we estimate σ?

Recall that the vertical deviations of the points in a scatterplot from the fitted regression line are the residuals. We use ei for the residual of the ith observation:

The residuals ei correspond to the linear regression model deviations εi. The ei sum to 0, and the εi come from a population with mean 0. Because we do not observe the εi, we use the residuals to estimate σ and check the model conditions of the εi.

To estimate σ, we work first with the variance and take the square root to obtain the standard deviation. For simple linear regression, the estimate of σ2 is the average squared residual

We average by dividing the sum by n-2 in order to make s2 an unbiased estimate of σ2. (The sample variance of n observations uses the divisor n-1 for this same reason.) The quantity n-2 is the degrees of freedom for s2. The estimate of the model standard deviation σ is given by

We call s the regression standard error.

We now use statistical software to calculate the regression for predicting BMI from physical activity for Example 10.1. In entering the data, we chose the names PA for the explanatory variable and BMI for the response. It is good practice to use names, rather than just x and y, to remind yourself which variables the output describes.

Example 10.2 Statistical software output for BMI and physical activity.

Figure 10.4 gives the outputs from two commonly used statistical software packages and Excel. Other software will give similar information. The JMP output reports estimates of our three parameters as b0=29.578, b1=-0.655, and s=3.6549. Be sure that you can find these values in this output and the corresponding values in the other outputs.

The least-squares regression line is the straight line that is plotted in Figure 10.3 (page 518). We would report it as

BMI^=29.578-(0.655×PA)

with an estimated model standard deviation of s=3.655. Because PA is coded in thousands of steps, the estimated model says that for each increase of 1000 steps per day, we expect the average BMI to be 0.655 smaller. The large estimated model standard deviation, however, suggests that there is a lot of variability about these means.

Note that the number of digits provided varies with the software used, and we have rounded the values to three decimal places. It is important to avoid cluttering up your report of the results of a statistical analysis with many digits that are not relevant. Software often reports many more digits than are meaningful or useful.

Computer outputs often give more information than we want or need. This is done to reduce user frustration when a software package does not print out the particular statistics wanted for an analysis. As you gain experience interpreting output, you learn to ignore the parts of the output that are not needed for the current analysis.

Because the means μy lie on the line μy=β0+β1x, they are all determined by β0 and β1. Thus, once we have estimates of β0 and β1, the linear relationship determines the estimates of μy for all values of x. Linear regression allows us to do inference not only for subpopulations for which we have data but also for those corresponding to x’s not present in the data. These x-values can be both within and outside the range of observed x’s. However, extreme caution must be taken when performing inference for an x-value outside the range of the observed x’s, also known as extrapolation, because there is no assurance that the same linear relationship between μy and x holds.

Now that we have fitted a line, we can further check the conditions that the simple linear regression model imposes on this fit. These checks are very important but often overlooked. There is no point in trying to do statistical inference if the data do not, at least approximately, meet the conditions that are the foundation for the inference. Misleading or incorrect conclusions can result.

These conditions concern the population, but we can observe only our sample. Thus, in doing inference, we act as if the sample is an SRS from the population. When the data are collected through some sort of random sampling, this condition is often easy to justify. In other settings, this condition requires more thought, and the justification is often debatable. For example, in Example 10.1 (page 518), the sample was a collection of volunteers, and the researchers argued that they could be considered an SRS from the population of college students at that university.

The remaining model conditions can be checked through a visual examination of the residuals. The first condition is that there is a linear relationship in the population, and the second is that the standard deviation of the responses about the population line is the same for all values of the explanatory variable. It is common to plot the residuals both against the case number (especially if this reflects the order in which the observations were collected) and against the explanatory variable. These residual plots are preferred to scatterplots of y versus x because they better magnify patterns. We often add a scatterplot smoother to these plots for further assistance in detecting patterns. The final condition is that the response varies Normally about the population regression line. That is, the model deviations vary Normally about 0.

Example 10.4 Checking the model conditions for BMI and physical activity.

Figure 10.5 is a plot of the residuals versus physical activity with a smooth-function fit. This smooth function suggests that the average residual is greater than 0 at both low and high physical activity levels. This could mean that a curved relationship between BMI and physical activity would better fit the data. It also could just be the result of chance variation. Notice that there is a large positive residual near each end of the physical activity range that is likely pulling up each end of the smoothed curve. Because the effect does not appear large, we attribute this pattern to chance variation. We do, however, investigate this further in Exercise 11.35 (page 593).

To check the assumption of a common standard deviation, we look at the spread of the residuals across the range of x. In Figure 10.5, the spread is roughly uniform across the range of PA, suggesting that this condition is reasonably met. There also do not appear to be any outliers or influential observations.

Figure 10.6 is a Normal quantile plot of the residuals. Because the plot looks reasonably straight, we are confident that the model deviations are approximately Normally distributed.

Note that the model does not require the distributions of y and x to be Normal; only the distribution of the model deviations is assumed Normal. It is a common mistake to assess the Normality of y when checking model conditions. Even though the examination of the x and y distributions can be helpful in terms of identifying potential outliers and influential observations, the focus should be on the residuals, which account for the differing means, when checking Normality.

In settings where the choices of x are controlled—for example, assigning subjects different milligrams of calcium supplement—we consider the subjects to be an SRS from the population and that they are randomly assigned to the different choices of x. In other words, the y’s for each value of x are an SRS from its subpopulation.

Provided our check of conditions gives no reason to question the use of the simple linear regression model, we can comfortably proceed to inference about parameters, or functions of parameters, such as

• The slope β1 and the intercept β0 of the population regression line.
• The mean response μy for a given value of x.
• An individual future response y for a given value of x.

If these condition checks raise doubts, it is best to consult an expert, as a more sophisticated regression model is likely needed. There is, however, one relatively simple remedy that may be worth investigation: transforming variables. An example using this remedy is provided at the end of this section.

Chapter 7 presented confidence intervals and significance tests for means and differences in means. In each case, inference rested on the standard errors of estimates and on t distributions. Inference in simple linear regression is similar in principle. For example, the confidence intervals have the form

where t* is a critical point of a t distribution. It is only the formulas for the estimate and standard error that are different.

As a consequence of the model assumptions about the deviations εi, the sampling distributions of b0 and b1 are Normally distributed with means β0 and β1 and standard deviations that are multiples of σ, the model parameter that describes the variability about the true regression line. In fact, even if the εi are not Normally distributed, a general form of the central limit theorem tells us that the distributions of b0 and b1 will be approximately Normal.

Because we do not know σ, we use the estimated model standard deviation s, which measures the variability of the data about the least-squares line. When we do this, we again move from the Normal distribution to t distributions but now with degrees of freedom n-2, the degrees of freedom of s. We give formulas for the standard errors SEb1 and SEb0 in Section 10.2. For now, we concentrate on the basic ideas and let the computer do the computations.

Formulas for confidence intervals and significance tests for the intercept β0 are exactly the same, replacing b1 and SEb1 by b0 and its standard error SEb0, respectively. Although computer outputs often include a test of H0: β0=0, this information usually has little practical value. From the equation for the population regression line, μy=β0+β1x, we see that β0 is the mean response corresponding to x=0. In many practical situations, this subpopulation does not exist or is not interesting. That is the case for the physical activity study, but Check-in question 10.1 (page 520) is a setting where this information is meaningful.

The test of H0: β1=0 is always quite useful. When we substitute β1=0 in the model, the x term drops out, and we are left with

This equation says that the mean of y does not vary with x. In other words, all the y’s come from a single population with mean β0, which we would estimate by y¯. The hypothesis H0: β1=0 therefore says that there is no straight-line relationship between y and x and that linear regression of y on x is of no value for predicting y.

We have found a statistically significant linear relationship between physical activity and BMI. The estimated slope is more than 4 standard deviations away from zero. Because this is highly unlikely to happen if the true slope is zero, we have strong evidence for our claim.

Note, however, that this is not the same as concluding that we have found a strong linear relationship between the response and explanatory variables in this example. We saw in Figure 10.3 that there is a lot of scatter about the regression line. A very small P-value for the significance test for a zero slope does not necessarily imply that we have found a strong relationship.

A confidence interval provides additional information about the linear relationship. For most statistical software, these intervals are optional output and must be requested. We can also construct them by hand from the default output.

We mentioned earlier that the intercept in this example is not of practical interest. It estimates average BMI when the activity level is 0, a value that isn’t realistic. For this reason, we do not compute a confidence interval for β0 or discuss the significance test available in the software.

Besides performing inference about the slope (and sometimes the intercept) in a linear regression, we may want to use the estimated regression line to make predictions about the response y at certain values of x. We may be interested in the mean response for different subpopulations or in the response of future observations at different values of x. In either case, we would want an estimate and associated margin of error.

For any specific value of x, say x*, the mean of the response y in this subpopulation is given by

To estimate this mean from the sample, we substitute the estimates b0 and b1 for β0 and β1:

A confidence interval for μy adds and subtracts to this estimate a margin of error based on the standard error SEμ^. The formula for the standard error is given in Section 10.2.

Many computer programs calculate confidence intervals for the mean response corresponding to each of the x-values in the data. Some can calculate an interval for any value x* of the explanatory variable. We will use a plot to illustrate these intervals.

Example 10.7 Confidence intervals for the mean response.

Figure 10.7 shows the upper and lower confidence limits on a graph with the data and the least-squares line. The 95% confidence limits appear as dashed curves. For any x*, the confidence interval for the mean response extends from the lower dashed curve to the upper dashed curve. The intervals are narrowest for values of x* near the mean of the observed x’s and widen as x* moves away from x¯.

Some software will do these calculations directly if you input a value for the explanatory variable. Other software will calculate the intervals for each value of x in the data set. Creating a new data set with an additional observation with x equal to the value of interest and y missing will often work.

If we sampled many women who averaged 9000 steps per day, we would expect their average BMI to be between 23.0 and 24.4 kg/m2. Note that many of the observations in Figure 10.7 lie outside the confidence bands. These confidence intervals do not tell us what BMI to expect for a single observation at a particular average steps per day. We need a different kind of interval, a prediction interval, for this purpose.

In the last example, we predicted the average BMI for x*=9000 steps per day. Suppose that we now want to predict an observation of BMI for a woman averaging 9000 steps per day. The predicted response y for an individual case with a specific value x* of the explanatory variable x is

This is the same as the expression for μ^y. That is, the fitted line is used both to estimate the mean response when x=x* and to predict a single future response. We use the two notations μ^y and y^ to remind ourselves of these two distinct uses.

This means our best guess for the BMI of this woman averaging 9000 steps per day is what we obtained using the regression equation, 23.7 kg/m2. A useful prediction, however, also needs a margin of error (or interval) to indicate its precision. The interval used to predict a future observation is called a prediction interval. Although the response y that is being predicted is a random variable, the interpretation of a prediction interval is similar to that for a confidence interval.

Consider doing the following many times:

• Draw a sample of n observations (xi, yi) and then one additional observation (x*, y).
• Calculate the 95% prediction interval for y when x=x* using the sample of size n.

Then, 95% of the prediction intervals will contain the value of y for the additional observation. In other words, the probability that this method produces an interval that contains the value of a future observation is 0.95.

The form of the prediction interval is very similar to that of the confidence interval for the mean response. The difference is that the standard error SEy^ used in the prediction interval includes both the variability due to the fact that the least-squares regression line is not exactly equal to the true regression line and the variability of the future response variable y around the subpopulation mean. The formula for SEy^ appears in Section 10.2.

Again, we use a graph to illustrate the results.

Example 10.9 Prediction intervals for BMI.

Figure 10.8 shows the upper and lower prediction limits, along with the data and the least-squares line. The 95% prediction limits are indicated by the dashed curves. Compare this figure with Figure 10.7, which shows the 95% confidence limits drawn to the same scale. The upper and lower limits of the prediction intervals are much farther away from the least-squares line than are the confidence limits. This results in most, but not all, of the observations in Figure 10.8 lying within the prediction bands.

The comparison of Figures 10.7 and 10.8 reminds us that the interval for a single future observation must be larger than an interval for the mean of its subpopulation.

Although a larger sample would better estimate the population regression line, it would not reduce the degree of scatter about the line. This means that prediction intervals for BMI, given activity level, will always be wide. This example clearly demonstrates that a very small P-value for the significance test for a zero slope does not necessarily imply that we have found a strong predictive relationship.

We started our analysis of Example 10.1 with a scatterplot to check whether the relationship between BMI and physical activity could be summarized with a straight line. We followed the least-squares fit with a residual plot (Figure 10.5) and a Normal quantile plot (Figure 10.6) to check that Normality, constant standard deviation, and linearity were approximately met. A check of model conditions should always be done prior to inference.

When there is a violation, it is best to consult an expert. However, there are times when a transformation of one or both variables will remedy the situation. In Chapter 2, we discussed the use of the log transformation to describe a curved relationship between x and y. Here is an example where the log transformation has a greater impact on other model assumptions.

Example 10.11 The relationship between income and education for entrepreneurs.

Numerous studies have shown that better-educated employees have higher incomes. Is this also true for entrepreneurs? Do more years of formal education translate into higher incomes? One study explored this question using the National Longitudinal Survey of Youth (NLSY), which followed a large group of individuals aged 14 to 22 for roughly 10 years.³ We consider a random sample of 100 entrepreneurs.

The researchers defined entrepreneurs to be those who were self-employed or who were the owner/director of an incorporated business. For each of these individuals, they recorded the education level and income. The education level was defined as the years of schooling completed prior to starting the business. The income level was the average annual total earnings since starting the business.

Figure 10.9 is a scatterplot of income versus education for our sample of 100 entrepreneurs, along with a smoothed curve. We use the variable names Inc and Educ.

The smoothed curve looks roughly linear, but the distributions of income subpopulations are skewed to the right. At each education level, there are many small incomes and just a few very large incomes. It also looks like the smoothed curve is being pulled toward those very large incomes, suggesting that those observations could be influential.

A common remedy for a skewed variable is to consider transforming it prior to fitting the model. Here, the researchers considered the natural logarithm of income (LogInc).

Example 10.12 Is this linear regression model reasonable?

Figure 10.10 is a scatterplot of these transformed data with the least-squares line and smoothed curve. Notice that the smoothed curve is practically the same as the least-squares line. More importantly, the observations are now more equally dispersed above and below the line, and those very large incomes don’t look unusual anymore.

A complete check of the residuals is still needed (see Exercise 10.13), but it appears that transforming y results in a data set that satisfies the linear regression model. Not only is the relationship more linear but the distribution of the observations about the regression line is more Normal. This is not always the end result of a transformation. In other cases, transforming a variable may help linearity and harm the Normality and constant variance assumptions. Always check the residuals before proceeding with inference.

Example 10.13 The accumlation of bone mass in young women.

As we age, our bones become weaker and are more likely to break. Osteoporosis (or weak bones) is the major cause of bone fractures in older women. Various researchers have studied this problem by looking at how and when bone mass is accumulated by young women. They’ve determined that up to 90% of a person’s peak bone mass is acquired by age 18 in girls.⁴ This makes youth the best time to invest in stronger bones.

Figure 10.11 displays data for a measure of bone strength called “total body bone mineral density” (TBBMD) and age for a sample of 256 young women.⁵ TBBMD is measured in grams per square centimeter (g/cm2), and age is recorded in years. The solid curve is the nonlinear fit, and the dashed curves are 95% prediction limits. As with our example of BMI and activity level, there is a large amount of scatter about the fitted curve. Although prediction intervals may be useless in this case, the researchers can draw some conclusions regarding the relationship.

The fitted nonlinear equation is

y^=1.162e-1.162+0.28x1+e-1.162+0.28x

In this equation, y^ is the predicted value of TBBMD, the response variable, and x is age, the explanatory variable. A straight line would not do a very good job of summarizing the relationship between TBBMD and age. At first, TBBMD increases with age, but then it levels off as age increases. The value of the function where it is level is called “peak bone mass”; it is a parameter in the nonlinear model. The estimate is 1.162, and the standard error is 0.008. Software gives the 95% confidence interval as (1.146, 1.178). Other calculations could be done to determine the age by which 90% of this peak bone mass is acquired.

The long-range goals of the researchers who conducted this study include developing intervention programs (such as exercise and increasing calcium intake) for young women to increase their TBBMD.