Two types of validity
- Internal validity is the ability of an experiment to establish cause-and-effect of the treatment within the sample studied.
- External validity is the ability of an experimental result to generalize to a larger context or population.
Two types of validity
Example: Randomly assign salespeople to one of two different compensation plans, and measure their sales performance over the next year.
- Internal validity: We can conclude that the choice of compensation plan caused the difference in sales performance.
- External validity: We can conclude that the same choice of compensation plan would have a similar impact in other salespeople outside of our sample.
What is a threat to validity?
A threat to validity is something that, if true, would cause us to doubt the conclusion that a particular study comes to.
Internal validity
Internal validity is the ability of an experiment to establish cause-and-effect of the treatment within the sample studied.
Examples of threats to internal validity:
- Failure to randomize
- Failure to follow the treatment protocol / attrition
- Small sample sizes
External validity
External validity is the ability of an experimental result to generalize to a larger context or population.
Examples of threats to external validity:
- Nonrepresentative samples
- Nonrepresentative protocol/policy
Using blocking to improve validity
Why do we need blocking if we are already randomizing?
- Randomization works “on average” (i.e., if we were able to run the experiment many times) but we only get one shot at creating treatment and control groups, and there might be imbalances in “nuisance” variables that could affect the outcome.
- For example, what will happen if the treatment group for the Moderna trial happens to get younger people in it than the control group?
- We can solve this by blocking: randomly assigning to treatment/control within groups
Without blocking: gender imbalance, due to random chance
With blocking: perfect gender balance, by design
Blocking in the Moderna vaccine trial
In the Moderna vaccine trial, they identified two possible variables that could impact COVID outcomes:
- Age (65+ vs under 65)
- Underlying health condition (for those under 65): at increased risk of severe COVID or not
Blocking design for Moderna trial
Everyone
├── Age 65+
│ ├── Treatment (Randomize)
│ └── Control (Randomize)
└── Age <65
├── Healthy
│ ├── Treatment (Randomize)
│ └── Control (Randomize)
└── Underlying condition
├── Treatment (Randomize)
└── Control (Randomize)
Analyzing experiments using regression
- For non-blocked design, use a simple regression (T = dummy variable that is 1 for treatment group and 0 for control group): \hat Y = \hat\beta_0 + \hat\beta_1 T
- For a blocked design, use a regression that controls for the blocking variable B: \hat Y = \hat\beta_0 + \hat\beta_1 T + \hat\beta_2 B
In both cases, the coefficient of T (\hat\beta_1) represents the estimated average treatment effect.
Example 2: Does online tutoring improve student achievement?
- Kraft, List, Livingston, and Sadoff (2022) study a pilot program in which volunteer college tutors met 1-on-1 online with middle-school students during the school day.
- Students from Chicago Heights Middle School in spring 2021, during the transition from fully remote instruction to hybrid learning.
- Eligible students were those who chose hybrid learning; a total of 560 students were randomized.
- The treatment was online tutoring two days per week for 30 minutes during advisory period.
- The main outcomes Y are standardized math and reading test scores
Example 2: Online tutoring by college volunteers
The study used blocking because treatment was rolled out across three waves, and randomization was done by grade level (to ensure proportional allocation).
- Wave 1: March 8 (n=268)
- Wave 2: March 22 (n=209)
- Wave 3: April 5 (n=83)
- Within each wave, students were randomized separately by grade level
In other words, there were 9 blocks (3 waves \times 3 grade levels); randomization was done within each block. So the analysis controls for these wave-by-grade randomization blocks.
Blocking design for tutoring experiment
Hybrid-learning students eligible for study
├── Wave 1 entrants
│ ├── Grade 6: Treatment vs Control (Randomize)
│ ├── Grade 7: Treatment vs Control (Randomize)
│ └── Grade 8: Treatment vs Control (Randomize)
├── Wave 2 entrants
│ ├── Grade 6: Treatment vs Control (Randomize)
│ ├── Grade 7: Treatment vs Control (Randomize)
│ └── Grade 8: Treatment vs Control (Randomize)
└── Wave 3 entrants
├── Grade 6: Treatment vs Control (Randomize)
├── Grade 7: Treatment vs Control (Randomize)
└── Grade 8: Treatment vs Control (Randomize)
Analyzing the tutoring experiment
Call:
lm(formula = diagmath_spring_s ~ treatment + block, data = tutoring)
Residuals:
Min 1Q Median 3Q Max
-4.2969 -0.7375 0.0947 0.6832 2.9048
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.026375 0.120069 0.220 0.8262
treatment 0.072226 0.091737 0.787 0.4315
blockgrade6.wave2 -0.096008 0.166356 -0.577 0.5641
blockgrade6.wave3 -0.004213 0.221596 -0.019 0.9848
blockgrade7.wave1 -0.036033 0.162855 -0.221 0.8250
blockgrade7.wave2 -0.136260 0.181348 -0.751 0.4528
blockgrade7.wave3 -0.197810 0.263887 -0.750 0.4539
blockgrade8.wave1 0.125621 0.156424 0.803 0.4223
blockgrade8.wave2 0.148972 0.165704 0.899 0.3691
blockgrade8.wave3 -0.631193 0.231357 -2.728 0.0066 **
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.017 on 487 degrees of freedom
(63 observations deleted due to missingness)
Multiple R-squared: 0.02939, Adjusted R-squared: 0.01145
F-statistic: 1.638 on 9 and 487 DF, p-value: 0.1016
Analyzing the tutoring experiment
2.5 % 97.5 %
(Intercept) -0.2095431 0.2622932
treatment -0.1080235 0.2524762
blockgrade6.wave2 -0.4228718 0.2308554
blockgrade6.wave3 -0.4396153 0.4311888
blockgrade7.wave1 -0.3560179 0.2839517
blockgrade7.wave2 -0.4925814 0.2200607
blockgrade7.wave3 -0.7163081 0.3206884
blockgrade8.wave1 -0.1817274 0.4329692
blockgrade8.wave2 -0.1766120 0.4745559
blockgrade8.wave3 -1.0857748 -0.1766115
Analyzing the tutoring experiment
- The estimated effect of offering tutoring is about 0.072 standard deviations on the math test.
- The 95% confidence interval for the effect ranges from about -0.108 to 0.252 standard deviations.
- Adjusting for other controls (e.g., pre-treatment test scores) does not change the estimated effect much, but it does reduce the standard error and make the confidence interval narrower.
Adjusting for other controls
Call:
lm(formula = diagmath_spring_s ~ treatment + math_s + reading_s +
block, data = tutoring)
Residuals:
Min 1Q Median 3Q Max
-1.8353 -0.4003 0.0164 0.3714 2.2897
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.07609 0.07501 1.014 0.3109
treatment 0.06534 0.05734 1.140 0.2550
math_s 0.63254 0.04198 15.067 < 2e-16 ***
reading_s 0.23412 0.04271 5.482 6.77e-08 ***
blockgrade6.wave2 -0.09506 0.10424 -0.912 0.3622
blockgrade6.wave3 -0.01693 0.13831 -0.122 0.9026
blockgrade7.wave1 -0.08878 0.10173 -0.873 0.3833
blockgrade7.wave2 -0.19445 0.11320 -1.718 0.0865 .
blockgrade7.wave3 -0.25919 0.16480 -1.573 0.1164
blockgrade8.wave1 -0.01475 0.09812 -0.150 0.8806
blockgrade8.wave2 0.09017 0.10345 0.872 0.3838
blockgrade8.wave3 -0.27572 0.14504 -1.901 0.0579 .
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.6345 on 485 degrees of freedom
(63 observations deleted due to missingness)
Multiple R-squared: 0.6235, Adjusted R-squared: 0.615
F-statistic: 73.03 on 11 and 485 DF, p-value: < 2.2e-16
Adjusting for other controls
2.5 % 97.5 %
(Intercept) -0.07129314 0.223463530
treatment -0.04732170 0.178005515
math_s 0.55005314 0.715035119
reading_s 0.15020978 0.318035248
blockgrade6.wave2 -0.29987156 0.109748559
blockgrade6.wave3 -0.28869419 0.254840126
blockgrade7.wave1 -0.28866194 0.111106703
blockgrade7.wave2 -0.41687069 0.027970289
blockgrade7.wave3 -0.58299449 0.064613389
blockgrade8.wave1 -0.20753259 0.178037576
blockgrade8.wave2 -0.11308702 0.293427166
blockgrade8.wave3 -0.56070294 0.009265699
Results from the tutoring experiment
- Intent-to-treat: about 0.07 SDs on the math test
- Effects are positive but statistically insignificant in this pilot study
- Estimates are larger in Wave 1 than in Waves 2 and 3, consistent with a possible dosage effect
- Students attended about 3.1 hours of tutoring on average; later waves had fewer chances to attend
Wave 1 results
Call:
lm(formula = diagmath_spring_s ~ treatment + factor(grade), data = tutoring %>%
filter(wave == 1))
Residuals:
Min 1Q Median 3Q Max
-2.41944 -0.66727 0.09577 0.66891 2.22510
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.01991 0.12275 -0.162 0.871
treatment 0.16165 0.12425 1.301 0.195
factor(grade)7 -0.03446 0.15440 -0.223 0.824
factor(grade)8 0.12507 0.14829 0.843 0.400
Residual standard error: 0.9639 on 237 degrees of freedom
(27 observations deleted due to missingness)
Multiple R-squared: 0.01233, Adjusted R-squared: -0.0001743
F-statistic: 0.9861 on 3 and 237 DF, p-value: 0.4
Wave 1 results
Call:
lm(formula = diagmath_spring_s ~ treatment + math_s + factor(grade),
data = tutoring %>% filter(wave == 1))
Residuals:
Min 1Q Median 3Q Max
-1.78406 -0.35996 0.00794 0.35805 1.61793
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.03310 0.07746 0.427 0.670
treatment 0.09835 0.07843 1.254 0.211
math_s 0.79521 0.04191 18.972 <2e-16 ***
factor(grade)7 -0.06430 0.09738 -0.660 0.510
factor(grade)8 0.03609 0.09364 0.385 0.700
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.6078 on 236 degrees of freedom
(27 observations deleted due to missingness)
Multiple R-squared: 0.6089, Adjusted R-squared: 0.6022
F-statistic: 91.85 on 4 and 236 DF, p-value: < 2.2e-16
Threats to validity of the tutoring experiment?
- What are some possible threats to validity here?
Imperfect uptake
- About 18% of treated students attended no tutoring at all
- Is this a threat to internal or external validity?
Imperfect uptake
- It is a threat to internal validity if we want the effect of tutoring itself
- But we can still get an internally valid estimate of the effect of being offered tutoring (the intent-to-treat effect)
- This often answers a more policy-relevant question anyway
The tutor pool
- Tutors were unpaid volunteers, many from highly selective colleges (including UT!)
- To scale up, the program would need many more tutors – might need to offer pay, or be less selective
- Is this a threat to internal or external validity?
The tutor pool
- It is a threat to external validity – how would the treatment work in a broader population?
- Not a threat to internal validity – the estimated effect of offering tutoring is still valid for the sample studied, even if the tutors were unusually good
The student pool
- This was a single pilot program in one Chicago middle school during the COVID-19 pandemic
- Would we see the same effect in rural schools in 2026?
The student pool
- This is a threat to external validity – the results may not generalize to other settings, grades, districts, or non-pandemic environments
- Not a threat to internal validity
- It is common for pilot studies to have limited external validity before conducting larger-scale trials with more representative samples
The limitations of RCTs
Although they are powerful for inferring causation, RCTs are hard to pull off:
- They can be incredibly expensive (e.g., Phase 3 clinical trial)
- Compliance with the treatment protocol isn’t perfect
- It can be hard to generalize beyond the participants involved in the study, if they aren’t representative (e.g., psychology experiments conducted on college students)
- They can be impractical (e.g., effect of salary on performance) or unethical to conduct (e.g., seatbelts, parachutes, even some medical trials)
Source: Beth Mole, Ars Technica, October 9, 2026