Total core time: ~150 minutes (about 2.5 hours). A natural break point is after Example 1 (Table 1 and adjusted analysis in PBC)—cover Example 2 (confounding simulation) and the exercises in a second sitting. Exercise 3 is open-ended and its time will vary.
Section
Time
Type
The Key Idea: No Analysis Can Fix a Broken Design
~10 min
Concept
Background: Study Design
~15 min
Concept
Background: Bias in Clinical Research
~15 min
Concept
Example 1: Table 1 and Adjusted Analysis in PBC
~30 min
Walkthrough
Example 2: Observational Study (Confounding)
~15 min
Walkthrough
What Can Go Wrong
~10 min
Reading
Exercise 1 (Guided)
~15 min
Practice
Exercise 2 (Semi-guided)
~20 min
Practice
Exercise 3 (Open-ended)
15–30 min
Practice
Comprehension Check
~10 min
Self-test
If you are short on time, focus on the pbc-table1 chunk in Example 1: even in this genuinely randomised trial, one of six baseline comparisons (age) comes out “significant” at p = 0.020 by chance. This is the session’s central lesson in one table - randomisation does not guarantee every p-value will be reassuring, and a single low p-value among several baseline comparisons is not evidence that randomisation failed.
TipHow to Use the Code in This Session
Every code chunk below is written so you can copy it into your R console (or an R script) and run it yourself, in the order shown. Where you see a “Run It Yourself” box, stop and run the code above it before reading the explanation that follows.
21.1 When Do You Use This?
Tip
You are designing a clinical or translational study and need to make foundational decisions before collecting any data: What study design is appropriate? What biases might invalidate your conclusions? How do you translate a research question into a clearly defined estimand? These choices determine what statistical analysis is appropriate and what your results can legitimately claim. For example: whether a study of a novel intervention should be an RCT or an observational cohort, and how that choice affects analysis and interpretation.
NoteTeacher Note
Ask learners to briefly describe a study they are involved in or planning - and to classify it using the design table in “Types of Study Design” below before they reach that table. Many learners are surprised to find their “obviously observational” project actually has elements of more than one design (e.g., a cross-sectional survey nested within a prospective cohort).
21.2 Learning Objectives
After completing this session you will be able to:
Distinguish study designs (RCT, cohort, case-control, cross-sectional) and identify their strengths and limitations
Define and identify selection bias, information bias, and confounding in clinical research
Describe the Bradford Hill criteria for causal inference from observational data
Translate a PICO clinical question into an analysis plan
Produce a Table 1 and interpret the results of an adjusted analysis
21.3 The Key Idea: No Analysis Can Fix a Broken Design
Estimated time: ~10 minutes (Concept)
Every other session in this course starts from the assumption that you already have a dataset, and asks: which test, which model, which diagnostic? This session asks the question that comes before all of that: was the study designed in a way that your data can actually answer your question?
Here is the uncomfortable truth: a sophisticated multivariable model applied to a poorly designed study does not rescue it. If your controls were recruited from a population that systematically differs from your cases (selection bias), if your exposure was measured differently in cases and controls (information bias), or if you failed to measure the one variable that explains both your exposure and your outcome (confounding), no regression coefficient, however precisely estimated, will give you the right answer. Get the design right first; the analysis comes second.
21.4 Background: Study Design
Estimated time: ~15 minutes (Concept)
21.4.1 From Question to Design
Every clinical study starts with a PICO question:
Element
Description
Example
P
Population
Adults with primary biliary cholangitis (PBC)
I
Intervention / Exposure
D-penicillamine treatment
C
Comparator
Placebo
O
Outcome
Death or liver transplantation at 10 years
21.4.2 Types of Study Design
Design
Key feature
Strengths
Limitations
RCT
Randomised allocation
Eliminates confounding; gold standard for causality
Expensive; ethical limits; short follow-up
Prospective cohort
Follow exposed/unexposed forward in time
Natural exposure; long follow-up
Confounding; dropout; expensive
Retrospective cohort
Use existing records
Fast; cheap
Data quality; selection bias
Case-control
Start from outcome; look back at exposure
Efficient for rare outcomes
Recall bias; selection of controls
Cross-sectional
One-time snapshot
Fast; cheap
Cannot determine temporality
Systematic review / meta-analysis
Synthesise multiple studies
Highest level of evidence
Publication bias; heterogeneity
21.4.3 Hierarchy of Evidence
Randomised evidence sits above observational evidence because randomisation distributes confounders equally across groups. However, observational designs are essential for: - Rare diseases (small RCT is underpowered) - Long-term outcomes - Questions where randomisation is unethical (e.g., smoking causes cancer)
21.5 Background: Bias in Clinical Research
Estimated time: ~15 minutes (Concept)
21.5.1 Three Core Biases
Selection bias occurs when study participants are not representative of the target population, or when selection into the study depends on both exposure and outcome.
Example: A hospital-based case-control study of smoking and MI; hospital controls may include more smokers (Berkson’s bias), inflating the apparent association.
Information bias arises from systematic errors in measuring exposure or outcome.
Types: - Recall bias: Cases remember exposures more vividly than controls (common in case-control studies) - Misclassification: Non-differential (random, attenuates association) vs differential (one group mismeasured, can bias any direction)
Confounding occurs when a variable is associated with both the exposure and outcome, and is not on the causal pathway.
Classic confound: Alcohol is associated with lung cancer, but the real confounders are smoking (associated with both heavy drinking and cancer). Adjusting for smoking removes the confounded association.
21.5.2 The Bradford Hill Criteria
Nine criteria for judging whether an observed association is causal:
Strength - strong associations are less likely to be entirely explained by bias
Consistency - replicated across settings and populations
Analogy - similar associations exist for similar factors
21.6 Example 1: Table 1 and Adjusted Analysis in PBC
Estimated time: ~30 minutes (Walkthrough)
NoteTeacher Note
Before running pbc-table1, remind learners that survival::pbc is from a genuinely randomised trial - so on average, baseline characteristics should be balanced across treatment arms by design. Ask: if you compute six p-values comparing baseline characteristics between two randomly assigned groups, each truly equal in the population, what is the chance that at least one of those six p-values is below 0.05, just by chance? (Answer: roughly 1 - 0.95^6 ≈ 26%.) Keep this in mind when the table appears.
We use the survival::pbc dataset (418 patients, primary biliary cholangitis trial).
2 Wilcoxon rank sum test; Pearson’s Chi-squared test
TipRun It Yourself
You should see a table with six rows comparing D-penicillamine (n = 158) and Placebo (n = 154):
Characteristic
D-penicillamine
Placebo
p-value
Age (years)
51 (11)
49 (10)
0.020
Sex (f)
137 (87%)
139 (90%)
0.3
Serum bilirubin (mg/dL)
2.9 (3.6)
3.6 (5.3)
0.8
Albumin (g/dL)
3.52 (0.44)
3.52 (0.40)
>0.9
Histologic stage
(distribution)
(distribution)
0.2
Died during follow-up
65 (41%)
60 (39%)
0.7
Five of the six comparisons are comfortably non-significant (p ≥ 0.2), as expected from randomisation. The exception is age (p = 0.020): the D-penicillamine group is on average about 2 years older. As discussed in the Teacher Note, with six comparisons there is roughly a 1-in-4 chance that at least one would fall below p = 0.05 purely by chance, even with perfect randomisation. A 2-year age difference is also small in absolute terms and unlikely to be clinically meaningful on its own.
This is not evidence that randomisation failed - it is exactly the kind of baseline imbalance that a well-randomised trial can still produce by chance, and exactly why the adjusted analysis below includes age as a covariate: adjustment can account for it regardless.
21.6.1 Unadjusted Analysis
Estimated time: ~5 minutes (Walkthrough)
fit_unadj <-glm(status_bin ~ trt_label, data = pbc_clean, family = binomial)tidy(fit_unadj, exponentiate =TRUE, conf.int =TRUE) %>%filter(term !="(Intercept)")
The unadjusted odds of death are slightly lower for Placebo (OR = 0.913), but the confidence interval (0.580 to 1.44) is wide and includes 1, and p = 0.695 is far from significant. This matches the Survival and Time-to-Event session’s log-rank result for the same trial (p = 0.7): there is no detectable difference in outcomes between treatment arms.
21.6.2 Adjusted Analysis
Estimated time: ~10 minutes (Walkthrough)
fit_adj <-glm(status_bin ~ trt_label + age + bili + albumin +factor(stage),data = pbc_clean, family = binomial)tidy(fit_adj, exponentiate =TRUE, conf.int =TRUE) %>%filter(term !="(Intercept)") %>%select(term, estimate, conf.low, conf.high, p.value)
The treatment odds ratio barely moves (0.913 → 0.819) and remains non-significant (p = 0.482) - exactly what “adjustment should not materially change the estimate” predicts for a randomised trial, despite the baseline age imbalance flagged above. Meanwhile, age itself is a highly significant predictor of death (OR = 1.05 per year, p < 0.001), as is bili (OR = 1.40 per unit, p < 0.001). Notice the familiar near-miss pattern for factor(stage)4 (OR = 7.62, p = 0.0652) - this echoes the borderline Stage 4 hazard ratio seen in the survival session’s Cox model, and for the same reason: relatively few patients reach Stage 4 with an event recorded, so the estimate is imprecise (wide CI from 1.28 to 148) even though the point estimate suggests a large effect.
Compare the treatment odds ratio before and after adjustment. In a well-randomised trial, adjustment should not materially change the estimate, but it can improve precision.
21.7 Example 2: Observational Study (Confounding)
Estimated time: ~15 minutes (Walkthrough)
NoteTeacher Note
This simulation is constructed so that exercise has no true effect on HbA1c - the only reason they are related is that both depend on SES. Before running confounding-sim, ask learners to predict the sign and significance of the unadjusted exercise coefficient, and then predict what will happen to it once SES is added to the model in confounding-adjusted.
In this example we simulate an observational study where the exposure-outcome association is entirely confounded.
set.seed(2024)n <-1000# Confounder: socioeconomic status (SES; high = 1)SES <-rbinom(n, 1, 0.4)# Exposure: physical exercise - SES causes exerciseexercise <-rbinom(n, 1, 0.3+0.4* SES)# Outcome: HbA1c - SES causes HbA1c (not exercise)HbA1c <-7.5-0.5* SES +rnorm(n, 0, 0.8)# Exercise has NO direct effect on HbA1c in this simulationobs <-tibble(SES, exercise, HbA1c)# Unadjusted: exercise appears protectivefit_unconf <-lm(HbA1c ~ exercise, data = obs)tidy(fit_unconf, conf.int =TRUE) %>%filter(term =="exercise")
This looks like a clear, highly significant result: people who exercise have HbA1c about 0.2 mg/dL lower (p < 0.001), and the confidence interval doesn’t come close to including zero. If you stopped here, you would conclude that exercise reduces HbA1c.
Adding SES to the model collapses the exercise effect to essentially zero (0.0216, p = 0.689, CI spanning both positive and negative values). This is exactly what the simulation built in: exercise has no direct effect on HbA1c (see the comment in confounding-sim). The unadjusted “effect” of -0.199 was entirely an artefact of SES driving both exercise (richer people exercise more) and HbA1c (richer people have lower HbA1c). Without a DAG or prior knowledge of the data-generating process, you cannot tell from the unadjusted result alone that this is what’s happening - see the Causal Inference session for the general framework.
In the unadjusted model, exercise appears to lower HbA1c, but this is entirely explained by SES (richer people exercise more and have lower HbA1c). Adjusting for SES removes the confounded association.
21.8 Where to Find Data Like This
Note
survival::pbc: An RCT in primary biliary cholangitis: 418 patients, 17 variables, continuous follow-up. Ideal for survival analysis and clinical regression.
MASS::birthwt: Birth weight data with maternal risk factors (smoking, hypertension, race). Good for logistic regression teaching.
NHANES: US population health survey, accessible via the nhanesA package. A rich observational dataset for confounding and epidemiology examples.
Epidemiological methods textbooks: Often provide study datasets. Rothman (2012) includes examples from case-control and cohort designs.
21.9 What Can Go Wrong
Estimated time: ~10 minutes (Reading)
Warning
Immortal time bias In retrospective cohort studies, if patients must survive long enough to receive a treatment, the treatment group has immortal follow-up time before exposure begins. Ignoring this biases results in favour of the treated group.
Collider bias Adjusting for a variable on the causal pathway (a collider) can open a spurious association between exposure and outcome. See Causal Inference for a worked example.
Ecological fallacy Associations observed at the group level (e.g., country-level sodium intake and stroke mortality) do not necessarily hold at the individual level.
Over-adjusting for intermediates Adjusting for a mediator (a variable on the causal pathway from exposure to outcome) removes the effect you are trying to estimate. For example, adjusting for BMI when studying the effect of a dietary intervention on diabetes.
21.10 Exercises
NoteTeacher Note
Exercise 1 is a worked example of identifying bias from a study description alone - the kind of critical appraisal skill that Exercise 3 asks learners to apply to a real published paper. Exercise 2 returns to Table 1 and adjustment, but this time with observational (non-randomised) data, where you should expect - and find - genuine baseline imbalances, unlike the chance imbalance in Example 1.
21.10.1 Exercise 1 (Guided): Identify Bias
Estimated time: ~15 minutes (Practice)
Read the following study description and identify the likely bias:
“A case-control study of dietary fat intake and breast cancer. Cases were women diagnosed with breast cancer in the past 2 years. Controls were recruited from the same hospital with other conditions. Diet was assessed by interview.”
What type of study is this?
What biases are most likely? (Hint: think about control selection and exposure measurement.)
If cases over-report fat intake because they are searching for an explanation for their illness, what type of bias is this? Would it bias the OR upward or downward?
TipExercise 1: Answers
Case-control study.
Selection bias (Berkson’s bias): Hospital controls may not represent the general population; they may have conditions related to diet that are common in the hospital (e.g., cardiovascular disease), which could distort the apparent fat-cancer association. Population-based controls would be better. (b) Recall bias: Cases who know they have cancer may search their memory more carefully for dietary risk factors - they may over-report fat intake compared with controls who are not “searching for a cause.”
Differential recall bias (or differential misclassification). Because misclassification differs between cases and controls, the OR will be biased; if cases over-report fat intake, the association between fat and cancer will be inflated (biased upward).
21.10.2 Exercise 2 (Semi-guided): Table 1 and Confounding
Real output. The Table 1 for birthwt shows genuine baseline imbalance between birth-weight groups - unlike the chance Age imbalance in Example 1, these differences reflect real associations in observational data: smoke (p = 0.026), race (p = 0.082), lwt (p = 0.013), and age (p = 0.2, not significant).
The unadjusted model gives smokeSmoker OR = 2.02 (95% CI 1.08-3.80, p = 0.0276) - smokers appear roughly twice as likely to have a low-birth-weight baby. After adjusting for age, race, and maternal weight, the picture is:
term
estimate (OR)
conf.low
conf.high
p.value
smokeSmoker
2.87
1.38
6.19
0.00552
age
0.978
0.913
1.04
0.511
raceBlack
3.43
1.25
9.63
0.0172
raceOther
2.57
1.15
5.94
0.0234
lwt
0.988
0.974
0.999
0.0498
The smoking odds ratio increases after adjustment (2.02 -> 2.87), the opposite of the typical “confounding toward the null” pattern in Example 2’s confounding-sim/confounding-adjusted pair. This is negative confounding (or “confounding masking”): smokers in this sample tend to have characteristics (e.g. lower lwt, race distribution) that are themselves protective against low birth weight, so those factors were partly cancelling out smoking’s true effect in the unadjusted model. Once age, race, and lwt are held constant, smoking’s full effect on birth weight becomes visible - and the adjusted association is more significant (p = 0.00552) than the unadjusted one (p = 0.0276). The lesson generalises: adjustment can move an estimate in either direction, and “the odds ratio changed after adjustment” does not by itself tell you whether confounding was inflating or masking the true effect - you have to think about the direction of each confounder’s association with both exposure and outcome.
21.10.3 Exercise 3 (Open-ended)
Estimated time: 15–30 minutes (Practice)
Identify a published observational study in your research area. Write a one-page critical appraisal covering:
Study design and PICO question
The main exposure and outcome
Three specific biases that could affect the results
Whether the statistical analysis adequately adjusted for confounders
Whether the Bradford Hill criteria support a causal interpretation
21.11 Comprehension Check
Estimated time: ~10 minutes (Self-test)
What is the key advantage of an RCT over a prospective cohort study?
A study finds that coffee drinking is associated with reduced risk of Parkinson’s disease. A confounder is proposed: smokers drink more coffee and are less likely to develop Parkinson’s (due to nicotine’s neuroprotective effects). Is this a confounder? Explain why or why not.
In a case-control study, cases and controls are recruited from different hospitals. What bias might this introduce?
What does “non-differential misclassification” mean, and which direction does it typically bias the association?
A cohort study uses data from 2010–2020. Patients are classified as “treated” if they received a statin at any point. However, many patients only started statins after having a cardiovascular event. What bias does this introduce?
NoteAnswers
Randomisation distributes measured and unmeasured confounders equally across treatment groups; something observational studies cannot achieve. In an RCT, any observed difference in outcome can be attributed to the treatment with much greater confidence. Cohort studies, even with careful adjustment, can only control for measured confounders; unknown or unmeasured confounders may still distort the estimate.
Yes, this is a confounder. Smoking satisfies all three criteria: (1) it is associated with the exposure (coffee drinking; smokers drink more coffee); (2) it is independently associated with the outcome (smoking is associated with reduced Parkinson’s risk via nicotine); (3) it is not on the causal pathway from coffee to Parkinson’s. Failure to adjust for smoking would make coffee drinking appear protective, when some of the apparent protection actually comes from smoking. Note: this is a real phenomenon and a genuine methodological challenge in coffee/Parkinson’s research.
Berkson’s (hospital admission) bias. Patients admitted to different hospitals may differ systematically by severity, comorbidities, or healthcare-seeking behaviour. Using controls from a different hospital means they may not be representative of the same source population as the cases, potentially creating a spurious association (or masking a true one).
Non-differential misclassification means the measurement error does not differ between exposure groups or between cases and controls - it is equally likely to misclassify anyone. This typically biases the association toward the null (i.e., it attenuates the observed effect size). A true association is made to look smaller. Differential misclassification (which differs between cases and controls) can bias in any direction.
Immortal time bias. Patients who start statins after a cardiovascular event were at risk of that event before starting treatment, but they are classified as “treated” throughout the follow-up period. This means the treated group has a period of follow-up before exposure starts (an immortal period when they could not have had the event). This artefactually makes the treated group appear healthier. The correct design is time-varying exposure analysis (Cox model with time-varying statin use) or an intention-to-treat design from a clear index date.
21.12 How to Report
NoteReporting Study Design and Methods
Use the appropriate reporting checklist for your study design; reviewers and journals now require these:
Study design
Checklist
Where to find it
Randomised controlled trial
CONSORT
consort-statement.org
Observational (cohort, case-control)
STROBE
strobe-statement.org
Diagnostic accuracy study
STARD
stard.equator-network.org
Systematic review / meta-analysis
PRISMA
prisma-statement.org
In a Methods section (observational study):
“This was a [prospective/retrospective] [cohort/case-control/cross-sectional] study. [Exposure] was defined as [definition]. [Outcome] was defined as [definition and ascertainment method]. The analysis adjusted for [list confounders], which were identified a priori as potential confounders based on [clinical rationale/DAG]. We followed the STROBE guidelines for reporting.”
Common reporting errors: - Describing an observational study as showing that an exposure “causes” an outcome - Not reporting the time period of data collection - Failing to describe how missing data were handled - Not specifying whether the analysis was pre-specified or exploratory
TipChoosing and reporting an effect measure
Which effect measure you report matters as much as the p-value. For a binary outcome, give a relative measure (relative risk or odds ratio) and an absolute one (risk difference, or number needed to treat for trials) – see Section 8.1. If your study evaluates a diagnostic test or a risk model, also report its diagnostic accuracy (sensitivity, specificity, and prevalence-dependent predictive values) – see Section 13.1.
21.13 Further Reading
Rothman (2012): Epidemiology: An Introduction - authoritative text on study design and confounding
Hernan and Robins (2020): Causal Inference: What If - free online textbook on observational causal inference
Szklo and Nieto (2014): Epidemiology: Beyond the Basics - bias, confounding, effect modification
Grimes and Schulz (2002): Cohort studies: marching towards outcomes (Lancet series on study design)
Grimes, David A., and Kenneth F. Schulz. 2002. “Cohort Studies: Marching Towards Outcomes.”Lancet 359 (9303): 341–45.