STA 235 Week 10
University of Texas at Austin
2026-10-28
How we evaluate a model depends on what our use for it is.
In week 6 we kept adding polynomial terms, and R^2 kept going up.
But a very wiggly curve overfits: it follows the random noise in this data set, so it does badly on new data.
It’s like memorizing the answers to a practice exam:
Today: how to catch overfitting before it costs us.
Like Moore’s law in week 6, Apple’s revenue grows by a roughly steady percentage each quarter. So we model log revenue with a straight-line trend:
(Period counts quarters: 1, 2, 3, …)
Its R^2 is 0.75. Does that tell us how well it will forecast next year?
R^2 and the residual standard error (RSE) measure how well the model fits past data — the same data we used to build it.
That’s the practice exam: the model has already seen the answers.
If you were an analyst, how impressed would your boss be if you claimed to be able to predict the past? (Not very — they want you to be able to predict the future!)
We can mimic predicting on future data by holding back the most recent observations.
Fit the model to the “training” data only (blue) and test the model by forecasting the most recent year of data (red).
We want to split our data into a test set (last 4 quarters) and a training set (everything else).
Then we fit the model to the training data only. (This is the same model as before.)
Now we can see how well the model predicts the test set.
These are predictions for log revenue. exp() undoes log(), so we use it to get back to revenue in $B:
We want one number that says how far off our 4 predictions were.
Root Mean Squared Prediction Error (RMSPE) — read the name backwards:
Interpret RMSPE just like the residual standard error: it’s the size of a typical miss. Smaller is better!
For reference, here are the four steps written in math:
\text{RMSPE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (Y_i - \hat Y_i)^2}
| Quarter | Actual Y | Predicted \hat Y | Miss Y - \hat Y | Miss squared |
|---|---|---|---|---|
| 2025 Q1 | 95.40 | 111.05 | -15.65 | 244.86 |
| 2025 Q2 | 94.00 | 113.32 | -19.32 | 373.35 |
| 2025 Q3 | 102.50 | 115.64 | -13.14 | 172.74 |
| 2025 Q4 | 143.80 | 118.01 | 25.79 | 665.04 |
| Total | 1456.00 |
Mean: 1456.00 \div 4 = 364.00
Root: \sqrt{364.00} = 19.08
When forecasting the future, a typical miss for this model is about $19.08 billion.
Is that big? Apple’s revenue in these quarters was about $94–144 billion, so a typical miss is roughly 18% of revenue.
Notice the pattern: the model guesses too high in Q1–Q3 and too low in Q4, the holiday quarter. A straight-line trend can’t see seasons.
[1] 19.08
Read it from the inside out: miss \to square \to mean \to square root.
When we care about prediction, RMSPE is the one that matters.
Why shuffle? If the rows were sorted (say, by month), the test set might be all summer days. A model trained on fall–spring would be tested on weather it never saw.
Let’s randomly split the utilities data into a training and test set.
library(tidyverse)
set.seed(9529)
# Ask R to shuffle all 117 observations
randomized <- slice_sample(utilities, n = 117)
# 24 is about 20% of 117, so we'll use the first 24 rows as our test set
utilities_test <- randomized[1:24,]
# The rest are our training set
utilities_train <- randomized[25:117,]set.seed() makes the “random” shuffle the same every time you run the code. Any number works.
Black = training data, Red = testing data
The red points are scattered across all temperatures — that’s what shuffling buys us.
When comparing models:
If we fit the model and evaluate it using the same data, we might overfit and get an overly optimistic estimate of how well we’ll predict new data.
Linear (fit to all the data)
Quadratic
9th degree
| Linear | Quadratic | 9th degree | |
|---|---|---|---|
| R^2 | 0.841 | 0.880 | 0.889 |
| RSE | 0.866 | 0.755 | 0.750 |
Judged by R^2 and RSE, the 9th-degree model “wins.” Do you trust it?
Now let’s do it properly. Fit the models using the training set:
Get their predictions on the test set:
Now let’s calculate RMSPE on the testing data:
[1] 1.014
[1] 0.8535
[1] 0.9483
| Model | RSE (training data) | RMSPE (test data) |
|---|---|---|
| Linear | 0.83 | 1.01 |
| Quadratic | 0.73 | 0.85 |
| 9th Degree | 0.71 | 0.95 |
Our result depends on which rows landed in the test set. Here is the same comparison with different random splits:
| Split | Linear | Quadratic | 9th degree | Winner |
|---|---|---|---|---|
| Ours (seed 9529) | 1.01 | 0.85 | 0.95 | Quadratic |
| Seed 1 | 0.73 | 0.54 | 0.64 | Quadratic |
| Seed 2 | 1.07 | 0.96 | 0.91 | 9th degree |
Out of 200 different random splits, the quadratic won 132 times, the 9th degree won 63 times, and the linear model won 5 times.
One split can be “unlucky.” Fix: give several different test sets a turn, and average.
First we need to split the data into 3 random groups of roughly the same size (39\times 3=117):
Now we need to build the model 3 times:
| Train on | Test on | |
|---|---|---|
| Model A | fold2 and fold3 |
fold1 |
| Model B | fold1 and fold3 |
fold2 |
| Model C | fold1 and fold2 |
fold3 |
rbind() stacks two data sets on top of each other.
Each model predicts the fold it was not trained on:
We now get 3 different RMSPE estimates:
[1] 0.9898
[1] 0.745
[1] 0.8463
Our final estimate of RMSPE is the average of these 3 estimates:
\text{RMSPE} = \frac{ 0.99 + 0.75 + 0.85 }{3} = 0.86
| Model | RSE (all data) | RMSPE (3-fold CV) | RMSPE (single train/test split) |
|---|---|---|---|
| Linear | 0.866 | 0.860 | 1.014 |
| Quadratic | 0.755 | 0.765 | 0.853 |
| 9th Degree | 0.750 | 0.907 | 0.948 |