Simple Linear Regression

Linear Statistical Models: Regression

Simple Linear Regression

In simple linear regression there is only one predictor variable and takes the form y = b₀ + b₁x + e. Where y is the response variable, x the predictor variable and e the error or residual. Other terms for the predictor variable include independent variable, explanitory variable or right-hand variable. The response variable is also know as the dependent variable, criterion variable, outcome variable or left-hand variable.

Simple Linear Regression Output

use http://www.philender.com/courses/data/hsbdemo, clear

regress write read

      Source |       SS       df       MS              Number of obs =     200
-------------+------------------------------           F(  1,   198) =  109.52
       Model |  6367.42127     1  6367.42127           Prob > F      =  0.0000
    Residual |  11511.4537   198  58.1386552           R-squared     =  0.3561
-------------+------------------------------           Adj R-squared =  0.3529
       Total |   17878.875   199   89.843593           Root MSE      =  7.6249

------------------------------------------------------------------------------
       write |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |   .5517051   .0527178    10.47   0.000     .4477446    .6556656
       _cons |   23.95944   2.805744     8.54   0.000     18.42647    29.49242
------------------------------------------------------------------------------

Interpretation

_cons = 23.95944 -- The predicted value for observation when read scores equal zero.

_b[read] = .5517051 -- For every one unit increase in read, the predicted value for write increases by .5517051.

regress, beta

      Source |       SS       df       MS              Number of obs =     200
-------------+------------------------------           F(  1,   198) =  109.52
       Model |  6367.42127     1  6367.42127           Prob > F      =  0.0000
    Residual |  11511.4537   198  58.1386552           R-squared     =  0.3561
-------------+------------------------------           Adj R-squared =  0.3529
       Total |   17878.875   199   89.843593           Root MSE      =  7.6249

------------------------------------------------------------------------------
       write |      Coef.   Std. Err.      t    P>|t|                     Beta
-------------+----------------------------------------------------------------
        read |   .5517051   .0527178    10.47   0.000                 .5967765
       _cons |   23.95944   2.805744     8.54   0.000                        .
------------------------------------------------------------------------------

Variance

Covariance

Standard Deviation

Sum of Squared Deviations (SS)

Sum of Cross Products (SSCP)

Conditional Expectation

Population Regression Model

Y_i = β₀ + β₁X_i + ε_i

where:
Y_i is the value of the dependent, response or outcome variable for the ith case
β₀ and β₁ are parameters
X_i is the value of the predictor, independent or explanitory variable for the ith case
ε_i is a random eror term with:

Regression Equations

Scatterplot with Regression Line

Partitioning the Sums of Squares

Residuals Illustrated

Correlation Coefficient

Degree of association between variables.

Squared Correlation Coefficient
aka -- Coefficient of Determintion

Proportion of variance accounted for by the predictor.

Coefficient of Alienation

Proportion of variance not accounted for by the predictor.

Residual

Sum of Squares Residuals

Variance of Estimate

Standard Error of Estimate

Alternative formula

Regression Coefficients

Standard Error of Regression Coefficient

Test of Regression Coefficient

Test of Regression Model

Standardized Regression Coefficients
where β = standardized regression coefficient

Standardized regression coefficients are what you would get if all the variables in the regression were first converted to standard scores (z-scores).

Sums of Squares Regression