Utilities data set

The data set utilities contains information on the natural gas bills for a house in Minnesota. We’ll focus on two variables:

• dailyspend is the average amount of money spent on natural gas (e.g. heating) for each day during the month
• temp is the average temperature outside for that month

What assumption violation(s) do you see here?

Linearity and homoscedasticity are violated

Using polynomial regression to fix problems

• If a polynomial curve (e.g., quadratic, cubic, etc) would be a better fit for the data than a line, we can fit a curve to the data.
• The way we do this is by adding X^2 to the model as a second predictor variable.

Writing out the equation

Writing out the equation:

\widehat{Y} = 9.4723 + -0.2116(\text{temp}) + 0.0012(\text{temp}^2)

Even though we’re fitting a parabola, it’s still a linear model since Y is a linear function of X and X^2!

Adding an X^2 term fits a parabola to the data

It solves the linearity problem!

Too high a degree “overfits” the data

Overfitting means the model follows the data too closely—including the random noise—so it doesn’t generalize well to new data. Think of memorizing answers for a test instead of understanding the concept: it works for those exact questions but fails on new ones.

Too high a degree creates dangers with extrapolation

Extrapolation = using the model to predict outside the range of X values we actually observed. For high-degree polynomials, the curve can swing wildly once you leave that range, so predictions become unreliable.

From Perusall:

I understand that polynomial regression can capture curvature in the data, but why is extrapolation particularly risky in polynomial regression?

Too high a degree creates dangers with extrapolation

Building polynomial models

From Perusall:

Because linear regression models can have errors in the Residual vs. Fitted plot, how can we be sure that these models with curvature will provide a more accurate representation, besides examining the plot?

Building polynomial models

• Start simple: only add higher-degree terms to the extent it gives you a substantial increase in R^2 (and is statistically significant), or satisfies an assumption hold that wasn’t satisfied before
• You must include lower-order terms: e.g., if you add X^3, you must also include X and X^2
• Be careful about overfitting and extrapolating when adding higher-order terms!
• An X^2 term is an interaction of X with itself: in a parabola, the slope depends on the value of X!

Using transformations to fix problems

• Sometimes, a violation of regression assumptions can be fixed by transforming one or the other of the variables (or both).
• When we transform a variable, we have to also transform our interpretation of the equation.

Moore’s Law

• Moore’s Law was a prediction made by Gordon Moore in 1965 (!) that the number of transistors on computer chips would double every 2 years.
• This implies exponential growth, so a linear model won’t fit well (and neither will any polynomial).

Moore’s Law

A linear model fails the “L” assumption

The log transformation

• The log transformation is frequently useful in regression, because many nonlinear relationships are naturally exponential.
• In R, exp(x) calculates e^x and log(x) calculates \ln x — the natural log of x.
• When we write \log x, we mean \ln x = \log_e x.

Modeling exponential growth

• If Y = ae^{bX}, then: \begin{align*} \log Y &= \log(ae^{bX}) \\ &= \log(a) + \log(e^{bX}) \\ &= \underbrace{\log(a)}_{\beta_0} + \underbrace{b}_{\beta_1}X \end{align*}
• In other words: if we think Y is an exponential function of X, predict \log Y as a linear function of X

Transistors does NOT have a linear relationship with year

But log(Transistors) does have a linear relationship with year

A log-linear model satisfies the linearity assumption

Making predictions using the log-linear model

When making predictions, we have to remember that our equation gives us predictions for \log Y, not Y!

Example: To make a prediction for the number of transistors in 2025:

\log(\widehat{Y}) = -681.21 + 0.35(2025) = 25.83

To make a prediction, we have to solve for \hat Y:

\widehat{Y} = e^{\log(\widehat{Y})} = e^{25.83} = 1.64\times 10^{11}

Making predictions using the log-linear model

exp(predict(log.model, list(Date.of.introduction=2025), 
            interval="prediction"))

           fit         lwr           upr
1 164341865197 19321959793 1397800686137

Interpreting the log-linear model

From Perusall:

Because taking the log of this function only impacts the relationship between the logarithms of the variables, does this impact how we interpret the predictor values as well? Would using this model mean that our interpretations are only applicable to the logarithmic confidence interval etc.?

Interpreting the log-linear model

When is the log transformation useful?

• You can transform X\to\log(X), Y\to\log(Y), or both.
• Anytime you need to “squash” one of the variables (logs make huge numbers, like transistors, not so big!), try transforming it with a log.
• You may need to do a little bit of trial and error to see what works best.
• Other transformations are possible! For example, X \to \sqrt{X}, or X \to 1/X.

An elasticity interpretation for log-log models

• The beer data set contains sales data for cases of beer sold at a supermarket chain
• PRICE18PK is the price of 18-packs, and CASES18PK is the number of 18-packs sold
• There are also variables for 12-packs and 30-packs

What is price elasticity?

• Price elasticity answers: “If we raise price by 1%, by what percent does quantity sold change?” It’s the ratio: \begin{align*} \text{elasticity} &= \frac{\text{\% change in demand}}{\text{\% change in price}} \end{align*}
• In a log-log model, the slope coefficient tells us exactly that: the (approximate) percent change in Y when X increases by 1%. So the slope is the elasticity.

Fitting a demand curve

We essentially need to fit a demand curve—will a linear model work?

Fitting a demand curve

A log-log model will work:

Price elasticity of 18 packs

A 1% increase in price is associated with about a 6.7% decrease in predicted sales. When quantity responds a lot to a small price change, we say the good is elastic (people can switch to other drinks); when quantity barely changes, we say inelastic (e.g., necessities with few substitutes).

Call:
lm(formula = log(CASES18PK) ~ log(PRICE18PK), data = beer)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.8986 -0.2043 -0.0013  0.2665  0.5948 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)      23.831      0.946    25.2   <2e-16 ***
log(PRICE18PK)   -6.705      0.337   -19.9   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.36 on 50 degrees of freedom
Multiple R-squared:  0.888, Adjusted R-squared:  0.886 
F-statistic:  397 on 1 and 50 DF,  p-value: <2e-16

Log transformations can also fix violations of the “E” assumption!

Compare the scale-location plots for the linear model (left) vs the log-log model (right):