Basic causal inference and randomized controlled trials

STA 235 Week 12

Emre Yucel

University of Texas at Austin

2026-11-11

First strategy for inferring causation: controlling for confounders

Chance, confounders, and colliders

If we run a regression predicting Y from X and find that X is a significant predictor of Y, we might like to conclude that X causes Y. But it might be the case that:

  • X causes Y.
  • Y actually causes X.
  • A common variable Z (a confounder) causes both X and Y.
  • A common variable W (a collider) is caused by both X and Y.
  • X and Y are not actually related in the population; they happen to be correlated in the sample just by chance.

Spurious (chance) correlations

Confounders (lurking variables)

A confounder is a third variable that causes both X and Y and explains the observed correlation between X and Y:

  • Days on which more ice cream is consumed tend to also have higher crime.
  • People who are vaccinated against COVID-19 are 34% less likely to die from non-COVID causes, compared to unvaccinated people.
  • French villages with more storks also have more babies.

Colliders

A collider is a third variable that is caused by both X and Y.

  • Talent and attractiveness are negatively correlated among celebrities even if uncorrelated in general
    • You become a celebrity if you are really good at something or really good-looking (maybe, but not usually, both), so most celebrities have one but not the other

Colliders

A collider is a third variable that is caused by both X and Y.

  • Among hospitalized patients, smokers are less likely to have COVID than non-smokers

    • Both COVID and smoking make you more likely to land in the hospital, so if you are a smoker you probably got there because of smoking, not COVID

Selecting the sample using a collider (or controlling for a collider) can change the correlation!

Colliders

Using regression to try to infer causation (last week)

  • Control for potential confounders so the slope of X expresses the “partial” relationship between X and Y without the effect of the confounder mixed in.
    • Example: Predict crime rate from ice cream consumption, but control for temperature.
  • Do not control for colliders because doing so might mislead us about the nature of the relationship between X and Y.
  • But this strategy is not perfect: we cannot be sure that we have controlled for all possible confounders!

A better strategy for inferring causation: Randomized Controlled Trials

Randomized controlled trials (RCTs)

  • Often called the “gold standard” for establishing causality
  • Randomly assign the value of X to participants
  • Now, any observed relationship between X and Y must be due to X (or chance), since the only reason an individual had a particular value of X was the random assignment

Why does randomization work?

  • Randomization ensures that the treatment and control groups are, on average, identical in all respects except for the treatment itself.
  • Put another way: if T is a binary treatment dummy, then it’s (approximately) uncorrelated with any other variable in or out of the dataset
  • The regression coefficient on T – difference in averages between treatment and control groups – is about the same whether or not we include additional variables!

Common business applications of RCTs

  • Marketing: A/B testing to see which version of an email marketing campaign performs better
  • Management: Testing out a sales bonus structure by randomly assigning stores to bonus plans
  • MIS: Randomly segmenting a small % of web traffic to a new server that is being tested

The fundamental problem of causal inference

Suppose you have a headache, and you take an aspirin. Then you don’t have a headache. Did the aspirin work?

The fundamental problem of causal inference

For any given person, we can only observe one outcome or the other, depending on whether the person took an aspirin or not:

Person Took aspirin Didn’t take aspirin
1 no headache ?
2 no headache ?
3 no headache ?
4 no headache ?
5 ? no headache
6 ? headache
7 ? headache
8 ? headache

Solving the fundamental problem of causal inference

The best we can do is compute an average treatment effect: the difference in the statistic of interest between the treatment vs control group:

\begin{align*} \text{Average treatment effect} &= (\% \text{ headache among aspirin-takers}) - {} \\ &\qquad (\% \text{ headache among non-aspirin-takers}) \\ &= 0/4 - 3/4 \\ &= -0.75 \end{align*}

Example 1: Phase 3 Clinical Trial for the Moderna COVID-19 vaccine

  • X = got the vaccine, Y = got COVID-19
  • Randomly assign study participants to get either the vaccine (a treatment group of 14,134 people) or a placebo (a control group of 14,073 people)
  • 11 vaccine recipients got COVID; 185 of placebo recipients got COVID

Threats to validity

Two types of validity

  • Internal validity is the ability of an experiment to establish cause-and-effect of the treatment within the sample studied.
  • External validity is the ability of an experimental result to generalize to a larger context or population.

Two types of validity

Example: Randomly assign salespeople to one of two different compensation plans, and measure their sales performance over the next year.

  • Internal validity: We can conclude that the choice of compensation plan caused the difference in sales performance.
  • External validity: We can conclude that the same choice of compensation plan would have a similar impact in other salespeople outside of our sample.

What is a threat to validity?

A threat to validity is something that, if true, would cause us to doubt the conclusion that a particular study comes to.

Internal validity

Internal validity is the ability of an experiment to establish cause-and-effect of the treatment within the sample studied.

Examples of threats to internal validity:

  • Failure to randomize
  • Failure to follow the treatment protocol / attrition
  • Small sample sizes

External validity

External validity is the ability of an experimental result to generalize to a larger context or population.

Examples of threats to external validity:

  • Nonrepresentative samples
  • Nonrepresentative protocol/policy

Using blocking to improve validity

Why do we need blocking if we are already randomizing?

  • Randomization works “on average” (i.e., if we were able to run the experiment many times) but we only get one shot at creating treatment and control groups, and there might be imbalances in “nuisance” variables that could affect the outcome.
  • For example, what will happen if the treatment group for the Moderna trial happens to get younger people in it than the control group?
  • We can solve this by blocking: randomly assigning to treatment/control within groups

Without blocking: gender imbalance, due to random chance

With blocking: perfect gender balance, by design

Blocking in the Moderna vaccine trial

In the Moderna vaccine trial, they identified two possible variables that could impact COVID outcomes:

  • Age (65+ vs under 65)
  • Underlying health condition (for those under 65): at increased risk of severe COVID or not

Blocking design for Moderna trial

Everyone
├── Age 65+
│   ├── Treatment (Randomize)
│   └── Control (Randomize)
└── Age <65
    ├── Healthy
    │   ├── Treatment (Randomize)
    │   └── Control (Randomize)
    └── Underlying condition
        ├── Treatment (Randomize)
        └── Control (Randomize)

Analyzing experiments using regression

  • For non-blocked design, use a simple regression (T = dummy variable that is 1 for treatment group and 0 for control group): \hat Y = \hat\beta_0 + \hat\beta_1 T
  • For a blocked design, use a regression that controls for the blocking variable B: \hat Y = \hat\beta_0 + \hat\beta_1 T + \hat\beta_2 B

In both cases, the coefficient of T (\hat\beta_1) represents the estimated average treatment effect.

Example 2: Does online tutoring improve student achievement?

  • Kraft, List, Livingston, and Sadoff (2022) study a pilot program in which volunteer college tutors met 1-on-1 online with middle-school students during the school day.
  • Students from Chicago Heights Middle School in spring 2021, during the transition from fully remote instruction to hybrid learning.
  • Eligible students were those who chose hybrid learning; a total of 560 students were randomized.
  • The treatment was online tutoring two days per week for 30 minutes during advisory period.
  • The main outcomes Y are standardized math and reading test scores

Example 2: Online tutoring by college volunteers

The study used blocking because treatment was rolled out across three waves, and randomization was done by grade level (to ensure proportional allocation).

  • Wave 1: March 8 (n=268)
  • Wave 2: March 22 (n=209)
  • Wave 3: April 5 (n=83)
  • Within each wave, students were randomized separately by grade level

In other words, there were 9 blocks (3 waves \times 3 grade levels); randomization was done within each block. So the analysis controls for these wave-by-grade randomization blocks.

Blocking design for tutoring experiment

Hybrid-learning students eligible for study
├── Wave 1 entrants
│   ├── Grade 6: Treatment vs Control (Randomize)
│   ├── Grade 7: Treatment vs Control (Randomize)
│   └── Grade 8: Treatment vs Control (Randomize)
├── Wave 2 entrants
│   ├── Grade 6: Treatment vs Control (Randomize)
│   ├── Grade 7: Treatment vs Control (Randomize)
│   └── Grade 8: Treatment vs Control (Randomize)
└── Wave 3 entrants
    ├── Grade 6: Treatment vs Control (Randomize)
    ├── Grade 7: Treatment vs Control (Randomize)
    └── Grade 8: Treatment vs Control (Randomize)

Analyzing the tutoring experiment


Call:
lm(formula = diagmath_spring_s ~ treatment + block, data = tutoring)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.2969 -0.7375  0.0947  0.6832  2.9048 

Coefficients:
                   Estimate Std. Error t value Pr(>|t|)   
(Intercept)        0.026375   0.120069   0.220   0.8262   
treatment          0.072226   0.091737   0.787   0.4315   
blockgrade6.wave2 -0.096008   0.166356  -0.577   0.5641   
blockgrade6.wave3 -0.004213   0.221596  -0.019   0.9848   
blockgrade7.wave1 -0.036033   0.162855  -0.221   0.8250   
blockgrade7.wave2 -0.136260   0.181348  -0.751   0.4528   
blockgrade7.wave3 -0.197810   0.263887  -0.750   0.4539   
blockgrade8.wave1  0.125621   0.156424   0.803   0.4223   
blockgrade8.wave2  0.148972   0.165704   0.899   0.3691   
blockgrade8.wave3 -0.631193   0.231357  -2.728   0.0066 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 1.017 on 487 degrees of freedom
  (63 observations deleted due to missingness)
Multiple R-squared:  0.02939,   Adjusted R-squared:  0.01145 
F-statistic: 1.638 on 9 and 487 DF,  p-value: 0.1016

Analyzing the tutoring experiment

                       2.5 %     97.5 %
(Intercept)       -0.2095431  0.2622932
treatment         -0.1080235  0.2524762
blockgrade6.wave2 -0.4228718  0.2308554
blockgrade6.wave3 -0.4396153  0.4311888
blockgrade7.wave1 -0.3560179  0.2839517
blockgrade7.wave2 -0.4925814  0.2200607
blockgrade7.wave3 -0.7163081  0.3206884
blockgrade8.wave1 -0.1817274  0.4329692
blockgrade8.wave2 -0.1766120  0.4745559
blockgrade8.wave3 -1.0857748 -0.1766115

Analyzing the tutoring experiment

  • The estimated effect of offering tutoring is about 0.072 standard deviations on the math test.
  • The 95% confidence interval for the effect ranges from about -0.108 to 0.252 standard deviations.
  • Adjusting for other controls (e.g., pre-treatment test scores) does not change the estimated effect much, but it does reduce the standard error and make the confidence interval narrower.

Adjusting for other controls


Call:
lm(formula = diagmath_spring_s ~ treatment + math_s + reading_s + 
    block, data = tutoring)

Residuals:
    Min      1Q  Median      3Q     Max 
-1.8353 -0.4003  0.0164  0.3714  2.2897 

Coefficients:
                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)        0.07609    0.07501   1.014   0.3109    
treatment          0.06534    0.05734   1.140   0.2550    
math_s             0.63254    0.04198  15.067  < 2e-16 ***
reading_s          0.23412    0.04271   5.482 6.77e-08 ***
blockgrade6.wave2 -0.09506    0.10424  -0.912   0.3622    
blockgrade6.wave3 -0.01693    0.13831  -0.122   0.9026    
blockgrade7.wave1 -0.08878    0.10173  -0.873   0.3833    
blockgrade7.wave2 -0.19445    0.11320  -1.718   0.0865 .  
blockgrade7.wave3 -0.25919    0.16480  -1.573   0.1164    
blockgrade8.wave1 -0.01475    0.09812  -0.150   0.8806    
blockgrade8.wave2  0.09017    0.10345   0.872   0.3838    
blockgrade8.wave3 -0.27572    0.14504  -1.901   0.0579 .  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.6345 on 485 degrees of freedom
  (63 observations deleted due to missingness)
Multiple R-squared:  0.6235,    Adjusted R-squared:  0.615 
F-statistic: 73.03 on 11 and 485 DF,  p-value: < 2.2e-16

Adjusting for other controls

                        2.5 %      97.5 %
(Intercept)       -0.07129314 0.223463530
treatment         -0.04732170 0.178005515
math_s             0.55005314 0.715035119
reading_s          0.15020978 0.318035248
blockgrade6.wave2 -0.29987156 0.109748559
blockgrade6.wave3 -0.28869419 0.254840126
blockgrade7.wave1 -0.28866194 0.111106703
blockgrade7.wave2 -0.41687069 0.027970289
blockgrade7.wave3 -0.58299449 0.064613389
blockgrade8.wave1 -0.20753259 0.178037576
blockgrade8.wave2 -0.11308702 0.293427166
blockgrade8.wave3 -0.56070294 0.009265699

Results from the tutoring experiment

  • Intent-to-treat: about 0.07 SDs on the math test
  • Effects are positive but statistically insignificant in this pilot study
  • Estimates are larger in Wave 1 than in Waves 2 and 3, consistent with a possible dosage effect
  • Students attended about 3.1 hours of tutoring on average; later waves had fewer chances to attend

Wave 1 results


Call:
lm(formula = diagmath_spring_s ~ treatment + factor(grade), data = tutoring %>% 
    filter(wave == 1))

Residuals:
     Min       1Q   Median       3Q      Max 
-2.41944 -0.66727  0.09577  0.66891  2.22510 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)
(Intercept)    -0.01991    0.12275  -0.162    0.871
treatment       0.16165    0.12425   1.301    0.195
factor(grade)7 -0.03446    0.15440  -0.223    0.824
factor(grade)8  0.12507    0.14829   0.843    0.400

Residual standard error: 0.9639 on 237 degrees of freedom
  (27 observations deleted due to missingness)
Multiple R-squared:  0.01233,   Adjusted R-squared:  -0.0001743 
F-statistic: 0.9861 on 3 and 237 DF,  p-value: 0.4

Wave 1 results


Call:
lm(formula = diagmath_spring_s ~ treatment + math_s + factor(grade), 
    data = tutoring %>% filter(wave == 1))

Residuals:
     Min       1Q   Median       3Q      Max 
-1.78406 -0.35996  0.00794  0.35805  1.61793 

Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)     0.03310    0.07746   0.427    0.670    
treatment       0.09835    0.07843   1.254    0.211    
math_s          0.79521    0.04191  18.972   <2e-16 ***
factor(grade)7 -0.06430    0.09738  -0.660    0.510    
factor(grade)8  0.03609    0.09364   0.385    0.700    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.6078 on 236 degrees of freedom
  (27 observations deleted due to missingness)
Multiple R-squared:  0.6089,    Adjusted R-squared:  0.6022 
F-statistic: 91.85 on 4 and 236 DF,  p-value: < 2.2e-16

Threats to validity of the tutoring experiment?

  • What are some possible threats to validity here?

Imperfect uptake

  • About 18% of treated students attended no tutoring at all
    • Very common!
  • Is this a threat to internal or external validity?

Imperfect uptake

  • It is a threat to internal validity if we want the effect of tutoring itself
  • But we can still get an internally valid estimate of the effect of being offered tutoring (the intent-to-treat effect)
  • This often answers a more policy-relevant question anyway

The tutor pool

  • Tutors were unpaid volunteers, many from highly selective colleges (including UT!)
  • To scale up, the program would need many more tutors – might need to offer pay, or be less selective
  • Is this a threat to internal or external validity?

The tutor pool

  • It is a threat to external validity – how would the treatment work in a broader population?
  • Not a threat to internal validity – the estimated effect of offering tutoring is still valid for the sample studied, even if the tutors were unusually good

The student pool

  • This was a single pilot program in one Chicago middle school during the COVID-19 pandemic
  • Would we see the same effect in rural schools in 2026?

The student pool

  • This is a threat to external validity – the results may not generalize to other settings, grades, districts, or non-pandemic environments
  • Not a threat to internal validity
  • It is common for pilot studies to have limited external validity before conducting larger-scale trials with more representative samples

The limitations of RCTs

Although they are powerful for inferring causation, RCTs are hard to pull off:

  • They can be incredibly expensive (e.g., Phase 3 clinical trial)
  • Compliance with the treatment protocol isn’t perfect
  • It can be hard to generalize beyond the participants involved in the study, if they aren’t representative (e.g., psychology experiments conducted on college students)
  • They can be impractical (e.g., effect of salary on performance) or unethical to conduct (e.g., seatbelts, parachutes, even some medical trials)

Source: Beth Mole, Ars Technica, October 9, 2026