OLS

Up to this point, we have been focusing on OLS considering:

• how we model regressions with this estimator
• how the estimator is derived and what properties it demonstrates
• how the classical assumptions make the estimator BLUE

OLS

Thus far in this class we’ve fit an OLS model the following questions:

• How much does an additional year of schooling increase earnings?
• Does the number of police officers affect campus crime rates?

Classical Assumptions

A1. Linearity: The population relationship is linear in parameters with an additive error term.

A2. Sample Variation: There is variation in \(X\).

A3. Exogeniety: The \(X\) variable is exogenous

Previously we used the first 3 assumptions to show that OLS is unbiased:

\[ \mathop{\mathbb{E}}\left[ \hat{\beta} \right] = \beta \]

We used the first 5 assumptions to derive a formula for the variance of the OLS estimator:

\[ \mathop{\text{Var}}(\hat{\beta}) = \frac{\sigma^2}{\sum_{i=1}^n (X_i - \bar{X})^2} \]

Sampling distribution

The probability distribution of the OLS estimators obtained from repeatedly drawing random samples of the same size from a population and fitting point estimates each time.

Provides information about their variability, accuracy, and precision across different samples.

Sampling distribution properties

1. Unbiasedness: If the Gauss-Markov assumptions hold, the OLS estimators are unbiased (i.e., \(E(\hat{\beta}_0) = \beta_0\) and \(E(\hat{\beta}_1) = \beta_1\))

Sampling distribution

The sampling distribution of \(\hat{\beta}\) to conduct hypothesis tests.

Use all 6 classical assumptions to show that OLS is normally distributed:

\[ \hat{\beta} \sim \mathop{N}\left( \beta, \frac{\sigma^2}{\sum_{i=1}^n (X_i - \bar{X})^2} \right) \]

Plotting the distributions of the point estimates in a histogram

Simulating 1,000 draws

Plotting the distributions of the point estimates in a histogram

Simulating 10,000 draws

Inference

Our current workflow:

1. Get data (points with \(X\) and \(Y\) values).

2. Regress \(Y\) on \(X\).

3. Plot the point estimates (i.e., \(\hat{Y_i} = \hat{\beta}_0 + \hat{\beta}_1X_i\)) and report.

Estimating error variance

We can estimate the variance of \(u_i\) (\({\color{#BF616A} \sigma^2}\)) using the sum of squared residuals (RSS):

\[ s^2_u = \dfrac{\sum_i \hat{u}_i^2}{n - k} \]

where \(n\) is the number of observations and \(k\) is the number of regression parameters. (In a simple linear regression, \(k=2\).)

OLS Variance

With \(s^2_u = \dfrac{\sum_i \hat{u}_i^2}{n - k}\), we can calculate the estimated variance of \(\hat{\beta}_1\)

\[ \mathop{\text{Var}}(\hat{\beta}_1) = \frac{s^2_u}{\sum_{i=1}^n (X_i - \bar{X})^2} \]

Inference

After deriving the distribution of \(\hat{\beta}_1\)¹, we have two (related) options for formal statistical inference (learning) about our unknown parameter \(\beta_1\):

• Hypothesis testing: Determine whether there is statistically significant evidence to reject a hypothesized value or range of values.
• Confidence intervals: Use the estimate and its standard error to create an interval that will generally² contain the true parameter.

Hypothesis Tests

Systematic procedure that gives us evidence to hang our hat on. Starting with a Null hypothesis (\(H_0\)) and an Alternative hypothesis (\(H_1\))

\[ \begin{align*} H_0:& \beta_1 = 0 \\ H_1:& \beta_1 \neq 0 \end{align*} \]

Possible outcomes

Within this structure, four possible outcomes exist:

Possible outcomes

Within this structure, four possible outcomes exist:

1. We fail to reject the null hypothesis and the null is true.

2. We reject the null hypothesis and the null is false.

Possible outcomes

Within this structure, four possible outcomes exist:

1. We fail to reject the null hypothesis and the null is true.

2. We reject the null hypothesis and the null is false.

3. We reject the null hypothesis, but the null is actually true.

Possible outcomes

Within this structure, four possible outcomes exist:

1. We fail to reject the null hypothesis and the null is true.

2. We reject the null hypothesis and the null is false.

3. We reject the null hypothesis, but the null is actually true.

4. We fail to reject the null hypothesis, but the null is actually false.

Possible outcomes

Within this structure, four possible outcomes exist:

1. We fail to reject the null hypothesis and the null is true.

2. We reject the null hypothesis and the null is false.

3. We reject the null hypothesis, but the null is actually true.¹

4. We fail to reject the null hypothesis, but the null is actually false.²

Or… from the golden age of textbook illustrations

Hypothesis Tests

Goal: Make a statement about \(\beta_1\) using information on \(\hat{\beta}_1\).

Hypothesis Tests

To conduct the test, we calculate a \(t\)-statistic¹:

\[ t = \frac{\hat{\beta}_1 - \beta_1^0}{\mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right)} \]

Distributed by a \(t\)-distribution with \(n-2\) degrees of freedom².

Hypothesis Testing

Normal distribution vs. \(t\) distribution

• A normal distribution has the same shape for any sample size.
• The shape of the t distribution depends the degrees of freedom.

• Degrees of freedom = 5.

Hypothesis Testing

Normal distribution vs. \(t\) distribution

• A normal distribution has the same shape for any sample size.
• The shape of the t distribution depends the degrees of freedom.

• Degrees of freedom = 50.

Hypothesis Testing

Normal distribution vs. \(t\) distribution

• A normal distribution has the same shape for any sample size.
• The shape of the t distribution depends the degrees of freedom.

• Degrees of freedom = 500.

Hypothesis Testing

Two sided t Tests

To conduct a t test, compare the \(t\) statistic to the appropriate critical value of the t distribution.

• To find the critical value in a t table, we need the degrees of freedom and the significance level \(\alpha\).

Reject (\(\text{H}_0\)) at the \(\alpha \cdot 100\)-percent level if

\[ \left| t \right| = \left| \dfrac{\hat{\mu} - \mu_0}{\mathop{\text{SE}}(\hat{\mu})} \right| > t_\text{crit}. \]

Hypothesis Tests

Next, we use the \(\color{#434C5E}{t}\)-statistic to calculate a \(\color{#B48EAD}{p}\)-value.

Describes the probability of seeing a \(\color{#434C5E}{t}\)-statistic as extreme as the one we observe if the null hypothesis is actually true.

Hypothesis Tests

We worry mostly about false positives, so we conduct hypothesis tests based on the probability of making a Type I error¹.

Type I vs Type II

Hypothesis Tests

We then compare \(\color{#434C5E}{\alpha}\) to the \(\color{#B48EAD}{p}\)-value of our test.

• If the \(\color{#B48EAD}{p}\)-value is less than \(\color{#434C5E}{\alpha}\), then we reject the null hypothesis at the \(\color{#434C5E}{\alpha}\cdot100\) percent level.

• If the \(\color{#B48EAD}{p}\)-value is greater than \(\color{#434C5E}{\alpha}\), then we fail to reject the null hypothesis at the \(\color{#434C5E}{\alpha}\cdot100\) percent level.¹

Ex. Are campus police associated with campus crime?

lm(crime ~ police, data = campus) %>% tidy()

# A tibble: 2 × 5
  term        estimate std.error statistic  p.value
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    18.4       2.38      7.75 1.06e-11
2 police          1.76      1.30      1.35 1.81e- 1

Significance level: \(\color{#434C5E}{\alpha} = 0.05\) (i.e., 5 percent test)

Test Condition: Reject \(H_0\) if \(p < \alpha\)

What is the \(\color{#B48EAD}{p}\)-value? \(p = 0.18\)

Do we reject the null hypothesis? No.

Hypothesis Tests

\(\color{#B48EAD}{p}\)-values are difficult to calculate by hand.

Alternative: Compare \(\color{#434C5E}{t}\)-statistic to critical values from the \({\color{#434C5E} t}\)-distribution.

Hypothesis Tests

Notation: \(t_{1-\alpha/2, n-2}\) or \(t_\text{crit}\).

• Find in a \(t\)-table using \(\color{#434C5E}{\alpha}\) and \(n-2\) degrees of freedom.

Compare the the critical value to your \(t\)-statistic:

• If \(|t| > |t_{1-\alpha/2, n-2}|\), then reject the null.
• If \(|t| < |t_{1-\alpha/2, n-2}|\), then fail to reject the null.

Two-sided tests

Based on a critical value of \(t_{1-\alpha/2, n-2} = t_{0.975, 100} =\) 1.98, we can identify a rejection region on the \(\color{#434C5E}{t}\)-distribution.

Ex.¹ \(\alpha = 0.05\)

lm(y ~ x, data = pop_df) %>% tidy()

# A tibble: 2 × 5
  term        estimate std.error statistic  p.value
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    2.53     0.422       6.00 3.38e- 8
2 x              0.567    0.0793      7.15 1.59e-10

Notice that the \(\color{#434C5E}{t}\)-statistic is 7.15. The critical value is \(\color{#434C5E}{t_{\text{0.975, 28}}} = 2.05\).

Which implies that \(p < 0.05\). Therefore, we reject \(H_0\) at the 5% level.

Ex. Are campus police associated with campus crime? (\(\alpha = 0.1\))

lm(crime ~ police, data = campus) %>% tidy()

# A tibble: 2 × 5
  term        estimate std.error statistic  p.value
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    18.4       2.38      7.75 1.06e-11
2 police          1.76      1.30      1.35 1.81e- 1

The \(\color{#434C5E}{t \text{-stat}} = 1.35\). The critical value is \(\color{#434C5E}{t_{\text{0.95, 94}}} = 1.66\).

|\(\color{#434C5E}{t \text{-stat}}| < |\color{#434C5E}{t_{\text{crit}}}|\) implies that \(p > 0.05\). Therefore, we fail to reject \(H_0\) at the 10% level.

One-sided tests

We might be confident in a parameter being non-negative/non-positive.

One-sided tests assume that the parameter of interest is either greater than/less than \(H_0\).

• Option 1 \(H_0\): \(\beta_1 = 0\) vs. \(H_1\): \(\beta_1 > 0\)

• Option 2 \(H_0\): \(\beta_1 = 0\) vs. \(H_1\): \(\beta_1 < 0\)

One-sided tests

Left-tailed: Based on a critical value of \(t_{1-\alpha, n-2} = t_{0.95, 100} =\) 1.66, we can identify a rejection region on the \(t\)-distribution.

One-sided tests

Right-tailed: Based on a critical value of \(t_{1-\alpha, n-2} = t_{0.95, 100} =\) 1.66, we can identify a rejection region on the \(t\)-distribution.

Ex. Do campus police deter campus crime? (\(\alpha = 0.1\))

Suppose we rule out the possibility that police increase crime, but not that they have no effect.

lm(crime ~ police, data = campus) %>% tidy()

# A tibble: 2 × 5
  term        estimate std.error statistic  p.value
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    18.4       2.38      7.75 1.06e-11
2 police          1.76      1.30      1.35 1.81e- 1

Notice that the \(\color{#434C5E}{t \text{-stat}} = 1.35\). The critical value is \(\color{#434C5E}{t_{\text{0.9, 94}}} = 1.29\).

Which implies that \(p > 0.05\). Therefore, we reject \(H_0\) at the 5% level.

Confidence intervals

Until now, we have considered point estimates of population parameters.

• Sometimes a range of values is more interesting/honest.

Confidence intervals

Q: Where does the confidence interval formula come from?

Confidence intervals

Replacing \(t\) with its formula gives:

Fail to reject \(H_0\) if

\[-t_\text{crit} \leq \frac{\hat{\beta}_1 - \beta_1^0}{\mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right)} \leq t_\text{crit} \]

Confidence intervals

Subtracting \(\hat{\beta}_1\) yields

Fail to reject \(H_0\) if \[ -\hat{\beta}_1 -t_\text{crit} \mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right) \leq - \beta_1^0 \leq - \hat{\beta}_1 + t_\text{crit} \mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right) \]

Confidence intervals

Replacing \(\beta_1^0\) with \(\beta_1\) and dropping the test condition yields the interval:

\[ \hat{\beta}_1 - t_\text{crit} \mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right) \leq \beta_1 \leq \hat{\beta}_1 + t_\text{crit} \mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right) \]

which is equivalent to

\[ \hat{\beta}_1 \pm t_\text{crit} \, \mathop{\hat{\text{SE}}} \left( \hat{\beta}_1 \right) \]

Confidence intervals

Main insight:

• If a 95 percent confidence interval contains zero, then we fail to reject the null hypothesis at the 5 percent level.
• If a 95 percent confidence interval does not contain zero, then we reject the null hypothesis at the 5 percent level.

Generally, a \((1- \alpha) \cdot 100\) percent confidence interval embeds a two-sided test at the \(\alpha \cdot 100\) level.

Confidence intervals Ex.

lm(y ~ x, data = pop_df) %>% tidy()

# A tibble: 2 × 5
  term        estimate std.error statistic  p.value
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)    2.53     0.422       6.00 3.38e- 8
2 x              0.567    0.0793      7.15 1.59e-10

# find degrees of freedom
dof <- summary(lm(y ~ x, data = pop_df))$df[2]
# return critical value
qt(0.975, dof)

[1] 1.984467

Confidence intervals

We have a confidence interval for \(\beta_1\), i.e., \(\left[ 0.410,\, 0.724 \right]\)

What does it mean?

Confidence intervals

Going back to our simulation…

Confidence intervals

From our previous simulation, 97.7% of 95% confidence intervals contain the true parameter value of \(\beta_1\).

Ex. Association of police with crime

You can instruct tidy to return a 95 percent confidence interval for the association of campus police with campus crime:

lm(crime ~ police, data = campus) %>% 
  tidy(conf.int = TRUE, conf.level = 0.95)

# A tibble: 2 × 7
  term        estimate std.error statistic  p.value conf.low conf.high
  <chr>          <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
1 (Intercept)    18.4       2.38      7.75 1.06e-11   13.7       23.1 
2 police          1.76      1.30      1.35 1.81e- 1   -0.830      4.34

Ex. Association of police with crime

Four confidence intervals for the same coefficient.

Hypothesis testing

Though it sounds silly, this research question is important

• Malaria is transmitted via mosquito
• Most model for malaria transmission historically assume equal risk of mosquito bites across individuals
• Though, good evidence of heterogenous propensity bites exist
• Understanding which people might have higher propensity for bites may allow for interventions that reduce the impact of malaria

Here is the data. Treatment group in blue.

Plot the true difference

Suppose the difference is coincidental. Then the labels don’t matter

Suppose the difference is coincidental. Then the labels don’t matter

Plot the “fake” difference

Labels don’t matter. Assign treatment randomly. Find the difference.

Plot the difference

Labels don’t matter. Assign treatment randomly. Find the difference.

Plot the differences

Labels don’t matter. Assign treatment randomly. Find the difference.

Plot the differences

Plot all 2,500 differences.