#stat 2000

[lawsofemotion]

Regression

When a relation appears to be linear in nature, a researcher will often wish to estimate this relationship between two variables with a straight line.

Regression Lines A regression line is a straight line that describes how a response variable y changes as an explanatory variable x changes. A regression line is often used to predict values of Y for given values of X.

With correlation, an explanatory and response variable were not necessary. Usually in regression, there is an explanatory variable X and a response variable Y, but this is also unnecessary.

True Line The theory behind regression analysis relies on the assumption that there is a true line that describes the relationship between X and Y.

Not all points will lie exactly on the line due to the variation of variables. Instead, the line describes an equation for the average value of y for any given value of x, denoted as μy. The following is the equation for this theoretical line:

μy = β₀ + β₁x

β₀ is the true intercept and β₁ is the true slope.

Simple Linear Regression Model The following are three assumptions made when using the simple linear regression model:

1. For any given value of x, the response variable y follows a normal distribution. 2. Observed responses yᵢ are independent of each other. 3. The standard deviation σ of Y is the same for all values of X, and it is unknown.

From these assumptions, the simple linear regression equation is the following:

yᵢ = β₀ + β₁xᵢ + Ꮛᵢ

yᵢ is the response variable for the ith individual in the population.

β₀ and β₁ are parameters.

xᵢ is the value of the explanatory variable for the ith individual in the population.

Ꮛᵢ is a random error term, where Ꮛᵢ ~ N(0, σ).

Least Squares Regression Given a value of X, a researcher would like to predict the corresponding value of Y.

Unless the relationship is perfect, the value of y will never be exact.

A sample is used to estimate the equation of the true line.

ŷ = b₀ + bᵢx

This line is called the least squares regression line.

ŷ is the estimated value of μy for a given value of X.

The value of b₀ is called the intercept and estimates β₀. The value of b₁ is called the slope and estimates β₁.

A researcher would like to find the line that fits their data the best, and so they need to find the appropriate values of b₀ and b₁. There are infinitely many possible lines, and so, since they are using x to predict y, they would like a line that lies close to the points as possible in the vertical direction.

Therefore, the least squares regression line minimizes the sum of the squared deviations in the vertical direction.

Slope and Intercept The slope of the regression line b₁ is the predicted increase in y when x increases by one unit.

Another formula for b₁ that does not include correlation is the following:

The intercept of the regression line b₀ is the predicted value of y when x = 0.

Predicted Value of Y When the least squares regression line is determined, the predicted value of Y when X is equal to some value can be found using this equation.

Coefficient of Determination The coefficient of determination r² is the fraction of variation in Y that is accounted for by its regression on X.

If r = ±1, then r² = 1. Y can be predicted exactly for any value of X, since the regression on X accounts for all of the variation in Y.

If r = 0, then r² = 0. The regression on X says nothing about the value of y.

Other than these possible values, r² is in between 0 and 1.

Coefficient of Determination Interpretation The following is the general interpretation of the coefficient of determination:

"(100r²)% of the variation in (Y context) is accounted for by (X context)."

Consider the study of whether missing class affects a student's final grade. The following is the data of 14 STAT 1000 students with their total missed classes and final grades:

From this data, the least squares regression line is found to be ŷ = 82.84 – 2.24x and r² = 0.3716. a) Interpret this line. b) Interpret r². c) What would be the predicted final grade for a student who missed 7 classes?

a)

The slope b₁ = -2.24 indicates that, for each additional class missed, it can be predicted that a student's final grade will decrease by 2.24%.

The intercept b₀ = 82.84 indicates that, if a student does not miss any classes, the predicted final grade for that student would be 82.84%.

b)

37.16% of the variation in a student's final grade is accounted for by the number of classes they miss.

c)

If x = 7, then ŷ = 82.84 – 2.24(7) = 67.16

[lawsofemotion]

Residuals

The residual eᵢ for the ith observation compares the observed value of y with the predicted value of y.

eᵢ = observed value of y – predicted value of y = yᵢ - ŷᵢ

The residual for any value of X reflects the error of the prediction.

A positive residual indicates that the observation falls above the regression line. A negative residual indicates that the observation falls below the regression line.

Extrapolation Mathematically, there is no problem with trying to predict a value of Y given a value of X that is outside of the range of the data. However, there is a statistical problem.

The process of predicting a value of Y for a value of X outside of the range of data is called extrapolation and should be avoided.

Outliers The regression line is strongly affected by outliers, especially outliers in the x-direction. Additionally, outliers also affect the coefficient of determination r².

With outliers included in the scatterplot, the regression line is less accurate in describing the relationship.

The following are three types of outliers:

The blue point is an outlier n the y-direction. It generally has little effect on the regression line.

The green point is a bivariate outlier that is neither in the x- or y-directions. It falls generally outside of the pattern of points and generally has little effect on the regression line.

The red point is an outlier in the x-direction. It has a strong effect on the regression line.

Influential Observations An observation is influential if removing it from the data would dramatically change the position of the regression line and the value of r².

Generally, outliers in the x-direction are influential observations.

[lawsofemotion]

Correlation

Consider the following two scatterplots:

Is the second scatterplot considered more stronger linearly than the first? It does not matter, because these scatterplots are actually the same data but in a different scale.

This shows that one's eyes is not the best tool to assess the strength of a relationship between two variables. Instead, a numerical measurement called correlation is used.

Correlation The correlation r measures the direction and strength of a linear relationship between two quantitative variables.

Suppose the values of two quantitative variables X and Y have been measured for n individuals. Then the following formula is used to measure its correlation:

Correlation and Association Despite a linear relation having a very strong positive or negative correlation, it cannot be concluded that the explanatory variable caused the response variable. This is because there may be other variables, called lurking variables, that factor in the changes of the response variable.

Correlation only measures the linear association between two quantitative variables, giving no information about the cause of the relationship. Association does not imply causation.

Properties of Correlation The following are seven properties of correlation r:

1. If r is positive, this indicates a positive association. If r is negative, this indicates a negative association.

2. Correlation r has a range between -1 and 1, inclusive. If r is very close to -1 or 1, the linear association is strong. Values of r that are close to zero indicate a weak linear association. Only a perfect linear relationship where r = -1 or 1 is when all points in the scatterplot lie on the line.

3. Correlation r has no units.

4. Correlation does not distinguish between X and Y, and so explanatory and response variables are unnecessary.

5. Changing the units of X or Y does not change the correlation r.

6. Correlation r only measures linear relationships.

7. Correlation r is affected by outliers.

[lawsofemotion]

ANOVA for Regression

ANOVA was previously used to assess the equality between several population means for some variable. This was done by partitioning the variation in the response variable into two components. The same idea will be used for the application of ANOVA in the regression setting.

Consider the variation of an observation on the response variable yᵢ from its sample mean ȳ:

yᵢ - ȳ

There are two reasons why yᵢ and ȳ differ. The first reason is that part of the variation in Y about its sample mean can be attributed to its regression on the explanatory variable X. The second reason is that there is some variability in y due to Y being a random variable that follows some distribution with a standard deviation σ.

The deviation of yᵢ from its mean can be partitioned into two components:

yᵢ differs from the predicted value ŷ due to chance variation, and the predicted value ŷ differs from ȳ because of the relationship between X and Y. Therefore, there are two sources of variation: error and regression.

Sums of Squares It can be shown that the sums of squares is additive:

The quantity Σ(yᵢ - ȳ)² is the total sum of squares, denoted as SST, with n – 1 degrees of freedom.

The quantity Σ(ŷᵢ - ȳ)² is the regression sum of squares, denoted as SSR, with one degree of freedom.

The quantity Σ(yᵢ - ŷᵢ)² is the error sum of squares, denoted as SSE, with n – 2 degrees of freedom.

Then SST = SSR + SSE.

Mean Squares The mean square for regression, denoted as MSR, is the regression sum of squares divided by its degrees of freedom.

The mean square for error, denoted as MSE, is the error sum of squares divided by its degrees of freedom.

Although the sums of squares are additive, the mean squares are not additive.

ANOVA Testing Previously, testing the existence of a linear relationship between X and Y was done using the t statistic. ANOVA can also be used to conduct this test, where the test statistic F has numerator degrees of freedom of one and denominator degrees of freedom of n – 2.

ANOVA Table Then, with all values known, the ANOVA table can be constructed.

Coefficient of Determination The coefficient of determination r² is also equal to the following:

[lawsofemotion]

Estimating the Standard Deviation

So far, β₀ and β₁ were found to be estimated using b₀ and b₁, respectively. Now the focus is on estimating the third parameter σ.

The standard deviation σ measures the variability of the response variable about the true regression line for any given value of X.

The residuals eᵢ measure the variability of the response variable about the estimated regression line for a given value of X.

Therefore,the estimator for the unknown parameter σ is a function of the residuals.

For a simple linear regression, the estimate of σ is the standard error sₑ with n – 2 degrees of freedom.

Notice that sₑ² is the average of the squared residuals.