Confidence Intervals, Prediction Intervals & Examining Lack of Fit (Methods II, Week of 1/25)
Confidence Intervals and Prediction Intervals with Least Squares Regression
One of the goals of regression is to estimate a value of y for a given value of x. In addition to a point estimate of y we can develop a range of y values, basically a confidence interval for an expected value of y.
E(y) is the parameter we are using the confidence interval to estimate, with the predicted value of y (Yi-hat) serving as the point estimate for E(y). The standard error of the CI is the sqrt(MSE)*sqrt((1/n)+((Xi-x-bar)^2)/Sxx))). We can develop a range of CIs for a range of x values, which then can form confidence bands around the regression line on the scatterplot. The typical pattern of a confidence band (shaded portion shown below) is a narrowing around the mean of x and then a fanning out on both ends. This fact taken together with the std error of the CI (specifically the (Xi-Xbar)^2 term) seems to say something like we are reasonably confident about a range of y values for an x value around the mean of x, and less confident elsewhere.
Seeing the confidence band spread out as x increases gives some intuition of why extrapolation can be dangerous with linear regression, as a really wide confidence interval is basically no good.
A prediction interval (band) is a similar idea to a confidence interval (band), and is used to estimate intervals on variables. PI are used to tell you where you can expect to see the next data point sampled. A PI says if you develop many prediction intervals, you'd expect that next value to lie within that prediction interval in 95% of the samples. Since we are predicting a value of a random variable, a Prediction band is always wider than a Confidence band (dotted line in scatterplot above). The formula is basically the same, but with a 1 added under the sqrt in the std error term.
Increasing sample size, alpha level (confidence coefficient) or the range of x-values sampled will all decrease both the CI and PI.
Examining Lack of Fit with Least Squares Regression
Obviously when you develop a regression line, you want it to be a good "fit" to the data. You want the regression to explain the relationship between the independent and dependent variable (without over-fitting).
One way to examine lack of fit is to check out a residual plot, which is done after developing a regression model. A residual is simply the observed value of y minus the predicted value of y at given value of the x-variable. A scatterplot of residuals vs y-hat shows us a picture we can use to evaluate the fit of a model. Since one of the assumptions we made when first developing a regression model is that we assume the errors (residuals) have equal variance, examining a residual plot allows us to check this.
A residual plot that would indicate the assumption of homogeneous variance has not been violated looks something like this:
There is no definite pattern here. If there were a pattern, like the residuals spread out as the values of Yi-hat increased or if the residuals looked like a parabola, the model is not a good fit, and indicates some other model (weighted or a model of higher order) would need to be used instead.
A formal test for lack of fit involves partitioning SSE (variance not explained by the model) into 2 pieces: pure error variability (SSPE) and sum of squares due to lack of fit (SSLOF). SSPE are the errors due to natural variability. In order to do this test you need at least one x value to have more than 1 observation. Writing the formula out here would be too tedious, but the basic idea here is that for each value of x where you have more than one observation:
take the mean of the associated y values
subtract the mean from the observation, square it, and sum them all, this is SSPE
SSE-SSPE=SSLOF
divide SSPE and MSLOF by their respective degrees of freedom to get mean square pure error (MSPE) and mean square lack of fit (MSLOF)
df for SSPE is the sum of the number of y values for which you have more than 1 x observation minus 1. So if you have 5 x-values each with 2 observations, the df would be (2-1)+(2-1)+(2-1)+(2-1)+(2-1)=5
df for SSLOF is (n-2)-df(SSPE)
The null here is that the linear model is a good fit while the alternative is the linear model is not a good fit. The F-stat here is the ratio of lack of fit to pure error, MSLOF/MSPE. The interpretation and conclusion follows the typical steps of an F-test. Concluding the alternative the null would indicate the model is not appropriate and some other method should be used.