Building models for prediction · 117 monthly utility bills

Fold by fold

One train/test split can be lucky or unlucky. Cross-validation fixes that by giving every month a turn in the test set. Shuffle the data, cut it into k folds, test on each fold once, and average the errors.

CV RMSPE = 0.765 Quadratic model, 3-fold cross-validation
Folds (k)
Model
Round Round 1 of 3

Step 1

Shuffle, then cut into folds

Each square is one month, in shuffled order. The shuffle mixes hot and cold months, so every fold looks like the full data set.

Test fold this roundTraining foldsClick a fold to give it the test-set turn

Step 2

One round: train on the rest, test on one fold

The blue curve is fit to the training folds only. The orange rings are the test fold, months this model never saw. The orange lines are its misses, and RMSPE boils them down to one number.

Training monthsTest monthsModel fit to training folds
This round
Round RMSPETypical miss on the test fold, in $ per day
Running average

Step 3

Every fold takes a turn, then average

After k rounds, every month has been tested exactly once. The fold errors bounce around because some folds are harder to predict than others. Their average is the cross-validation estimate, and it is steadier than any single split.

RoundTrain onTest onTest monthsRMSPE
Average

Step 4

Use it to pick a model

Run the same folds for each model and compare their averages. Each dot is one fold's RMSPE; the black bar is the average.

ModelRSE (all data)CV RMSPEBest and worst fold

Step 5

How many folds?

Switch Folds (k) in the bar at the top and watch the table above.

Five or 10 folds are common in practice. Whatever k you choose, the quadratic model comes out on top here, and the 9th-degree model never beats it.

In R

3-fold cross-validation by hand

set.seed(9529) randomized <- slice_sample(utilities, n = 117) # shuffle fold1 <- randomized[1:39, ] fold2 <- randomized[40:78, ] fold3 <- randomized[79:117, ] # Each fold takes one turn as the test set modelA <- lm(dailyspend ~ temp, data = rbind(fold2, fold3)) modelB <- lm(dailyspend ~ temp, data = rbind(fold1, fold3)) modelC <- lm(dailyspend ~ temp, data = rbind(fold1, fold2)) rmspeA <- sqrt(mean((fold1$dailyspend - predict(modelA, fold1))^2)) rmspeB <- sqrt(mean((fold2$dailyspend - predict(modelB, fold2))^2)) rmspeC <- sqrt(mean((fold3$dailyspend - predict(modelC, fold3))^2)) mean(c(rmspeA, rmspeB, rmspeC)) # 0.86 for the linear model

Data: 117 monthly bills from the mosaicData Utilities set; dailyspend = gas bill ÷ billing days, temperature in °F. The shuffle is the one R makes with set.seed(9529) and slice_sample(), so the 3-fold numbers match lecture exactly. For k = 5 and 10, the same shuffled order is cut into k nearly equal pieces. Companion lab: Train, Test, Repeat.