Total core time: ~145 minutes (about 2.5 hours). This session is structured as a reference guide as much as a tutorial - the decision tables in particular are designed for you to come back to whenever you face a new analysis. A natural break point is after Example 2.
| Section | Time | Type |
|---|---|---|
| The Key Idea: Match Method to Question | ~10 min | Concept |
| The Decision Framework | ~5 min | Concept |
| Decision Tables by Question Type | ~15 min | Reference |
| Example 1: Navigating to the Right Test | ~15 min | Walkthrough |
| Example 2: Binary Outcome | ~15 min | Walkthrough |
| Assumption Checking & Method Summary | ~10 min | Reference |
| What Can Go Wrong | ~10 min | Reading |
| Exercise 1 (Guided) | ~15 min | Practice |
| Exercise 2 (Semi-guided) | ~20 min | Practice |
| Exercise 3 (Open-ended) | 15–30 min | Practice |
| Comprehension Check | ~10 min | Self-test |
If you are short on time, work through Example 2 (binary-example): it shows a chi-square test returning p = 0.18 even though the simulation builds in a real 15-percentage-point difference (35% vs 20%) between groups. This is the session’s central warning in miniature - a non-significant result does not mean “no test was needed” or “pick a different test until something is significant”; it may simply mean the sample size was too small to detect a real effect.
Every code chunk below is written so you can copy it into your R console (or an R script) and run it yourself, in the order shown. Where you see a “Run It Yourself” box, stop and run the code above it before reading the explanation that follows.
You have data from a clinical study and need to choose the right test. Should you use a t-test or Mann-Whitney? Logistic regression or a chi-square test? This guide helps you navigate from your research question and data type to the appropriate method, and links you to the session where you can learn it properly.
Before starting, ask learners to write down one analysis they are currently planning or have recently completed (their own research question, outcome type, and comparison). Keep this written down - at the end of the session, in Exercise 3, they will use the decision tables to check whether the test they chose (or are planning to choose) is the appropriate one.
After completing this session you will be able to:
Estimated time: ~10 minutes (Concept)
Every statistical method in this course was designed to answer a specific type of question. Use the wrong method and your answer is biased or meaningless, even if R runs without errors.
The two most common mistakes are: picking a test based on which gives the smallest p-value (outcome-dependent test selection, which inflates false positives), and defaulting to a t-test when the design calls for something else (paired measurements, a binary outcome, or clustered observations). Before you open R, answer three questions: What type is your outcome? What are you comparing or estimating? Are the assumptions for your first-choice method met? The tables below take you from those three answers to the right method, every time.
Estimated time: ~5 minutes (Concept)
Every statistical test choice starts with three questions:
Estimated time: ~15 minutes (Reference)
| Outcome | Assumptions met? | Method | Session |
|---|---|---|---|
| Continuous | Normal, equal variance | Independent t-test | Section 5.1 |
| Continuous | Normal, unequal variance | Welch t-test | Section 5.1 |
| Continuous | Non-normal or n < 30 | Mann-Whitney U | Section 6.1 |
| Continuous, paired | Normal differences | Paired t-test | Section 5.1 |
| Continuous, paired | Non-normal differences | Wilcoxon signed-rank | Section 6.1 |
| Binary | Counts = 5 per cell | Chi-square / Fisher’s | Section 8.1 |
| Ordinal | - | Mann-Whitney U | Section 6.1 |
| Outcome | Assumptions met? | Method | Session |
|---|---|---|---|
| Continuous | Normal, equal variance | One-way ANOVA | Section 7.1 |
| Continuous | Non-normal or unequal variance | Kruskal-Wallis | Section 6.1 |
| Continuous, two factors | Normal | Two-way ANOVA | Section 7.1 |
| Continuous, repeated | Normal | Repeated-measures ANOVA / LMM | Section 16.1 |
| Binary | Counts = 5 | Chi-square (k × 2 table) | Section 8.1 |
| Outcome | Method | Session |
|---|---|---|
| Continuous–Continuous, linear | Pearson r | Section 10.1 |
| Continuous–Continuous, non-linear / ordinal | Spearman ρ | Section 10.1 |
| Binary–Binary | Phi coefficient / chi-square | Section 8.1 |
| Time series / autocorrelated | Check session on multiple regression | Section 12.1 |
| Outcome | Model | Session |
|---|---|---|
| Continuous | Linear regression | Section 11.1 |
| Continuous, multiple predictors | Multiple regression | Section 12.1 |
| Binary | Logistic regression | Section 13.1 |
| Ordered categorical | Ordinal logistic regression | Section 13.1 |
| Count | Poisson / negative binomial regression | Section 12.1 |
| Time-to-event | Cox proportional hazards | Section 17.1 |
| Correlated / hierarchical | Mixed models (LMM / GLMM) | Section 16.1 |
After fitting a binary model, evaluate it as a diagnostic test – sensitivity, specificity, PPV/NPV, and likelihood ratios – in Section 13.1.
| What you want | Method | Session |
|---|---|---|
| One proportion vs known value | One-sample z-test / binomial test | Section 4.1 |
| Two proportions | Chi-square / Fisher’s exact | Section 8.1 |
| Relative risk / odds ratio | Logistic regression | Section 13.1 |
| Risk difference / NNT (effect size) | Compute from the 2×2 table | Section 8.1 |
Estimated time: ~15 minutes (Walkthrough)
The simulation below builds in a real difference: males have a 35% chance of NAFLD vs 20% for females, a 15-percentage-point gap. Before running binary-example, ask learners to predict whether chisq.test() will find this difference statistically significant with n = 200 split roughly evenly between sexes. Most learners expect “yes, obviously” because the built-in difference is large. The actual result is a useful surprise.
Scenario: You want to know if NAFLD diagnosis (yes/no) differs between males and females in a cohort of 200 patients.
set.seed(2024)
n <- 200
sex <- sample(c("Male", "Female"), n, replace = TRUE)
# Males have ~35% NAFLD, females ~20%
nafld <- rbinom(n, 1, ifelse(sex == "Male", 0.35, 0.20))
dat <- tibble(sex, nafld = factor(nafld, labels = c("No", "Yes")))
# Chi-square for unadjusted association
chisq.test(table(dat$sex, dat$nafld))
Pearson's Chi-squared test with Yates' continuity correction
data: table(dat$sex, dat$nafld)
X-squared = 1.7904, df = 1, p-value = 0.1809You should see:
X-squared = 1.7904, df = 1, p-value = 0.1809Despite the 15-percentage-point true difference built into the simulation (35% vs 20%), the chi-square test gives p = 0.18 - not significant at the conventional 0.05 threshold. With only 200 patients split between two sexes, the observed counts in this particular random sample are not extreme enough to rule out chance. This is not a reason to try a different test until you find a significant one (see “What Can Go Wrong” below); it may simply reflect limited power. The Power and Sample Size session covers how to plan a study large enough to detect an effect of this size.
If you need to adjust for age and BMI, switch to logistic regression:
age <- rnorm(n, 50, 10)
bmi <- rnorm(n, 27, 4)
dat2 <- dat %>% mutate(nafld_num = as.integer(nafld) - 1, age, bmi, sex = factor(sex))
glm(nafld_num ~ sex + age + bmi, data = dat2, family = binomial) %>%
broom::tidy(exponentiate = TRUE, conf.int = TRUE) %>%
select(term, estimate, conf.low, conf.high, p.value)# A tibble: 4 × 5
term estimate conf.low conf.high p.value
<chr> <dbl> <dbl> <dbl> <dbl>
1 (Intercept) 0.608 0.0486 7.44 0.697
2 sexMale 1.60 0.885 2.92 0.122
3 age 1.01 0.976 1.04 0.665
4 bmi 0.972 0.899 1.05 0.467You should see (rounded):
| term | estimate (OR) | conf.low | conf.high | p.value |
|---|---|---|---|---|
| (Intercept) | 0.608 | 0.0486 | 7.44 | 0.697 |
| sexMale | 1.60 | 0.885 | 2.92 | 0.122 |
| age | 1.01 | 0.976 | 1.04 | 0.665 |
| bmi | 0.972 | 0.899 | 1.05 | 0.467 |
The adjusted odds ratio for sexMale (1.60, 95% CI 0.885 to 2.92, p = 0.122) tells the same story as the chi-square test, just on a different scale: males appear to have higher odds of NAFLD, but the confidence interval is wide and includes 1, and the p-value does not reach significance. Logistic regression adds the ability to adjust for age and bmi (neither of which shows an association here) - this is the key reason to move beyond chi-square, not because chi-square “failed” to find significance.
Decision path: Binary outcome → Association with one predictor → Chi-square (unadjusted). Add covariates → Logistic regression → Section 13.1
Estimated time: ~10 minutes (Reference)
| Assumption | Test / Plot | What to do if violated |
|---|---|---|
| Normality | Shapiro-Wilk, QQ plot | Non-parametric alternative |
| Equal variance | Levene’s / Bartlett’s | Welch t-test / Welch ANOVA |
| Independence | Study design | Mixed model for clustering |
| Proportional hazards (Cox) | cox.zph() |
Stratified Cox or time-varying HR |
| Linearity (regression) | Residual vs fitted | Polynomial or log transform |
| No influential outliers | Cook’s distance | Robust regression |
tribble(
~Method, ~Outcome, ~Key_Assumption, ~Non_parametric_Alternative,
"Independent t-test", "Continuous", "Normal, equal variance", "Mann-Whitney U",
"Paired t-test", "Continuous", "Normal differences", "Wilcoxon signed-rank",
"One-way ANOVA", "Continuous", "Normal, equal variance", "Kruskal-Wallis",
"Pearson r", "Continuous", "Bivariate normal, linear", "Spearman rho",
"Chi-square", "Categorical", "Expected count >= 5", "Fisher's exact",
"Linear regression", "Continuous", "Normal residuals", "Robust regression",
"Logistic regression", "Binary", "Sufficient events/predictor", "None standard",
"Cox regression", "Time-to-event", "Proportional hazards", "Stratified Cox"
) %>%
knitr::kable()| Method | Outcome | Key_Assumption | Non_parametric_Alternative |
|---|---|---|---|
| Independent t-test | Continuous | Normal, equal variance | Mann-Whitney U |
| Paired t-test | Continuous | Normal differences | Wilcoxon signed-rank |
| One-way ANOVA | Continuous | Normal, equal variance | Kruskal-Wallis |
| Pearson r | Continuous | Bivariate normal, linear | Spearman rho |
| Chi-square | Categorical | Expected count >= 5 | Fisher’s exact |
| Linear regression | Continuous | Normal residuals | Robust regression |
| Logistic regression | Binary | Sufficient events/predictor | None standard |
| Cox regression | Time-to-event | Proportional hazards | Stratified Cox |
Estimated time: ~10 minutes (Reading)
Choosing based on p-value alone Never choose the test that gives the most significant result. Test choice must be determined by the research question and data type, not by the outcome.
Ignoring paired structure Using a two-sample t-test when data are paired wastes statistical power (you ignore the within-subject correlation). Always account for the study design.
Multiple testing without correction Running 20 tests and reporting the significant ones inflates the type I error. Apply FDR or Bonferroni correction when testing multiple outcomes. See resources on multiple-testing corrections for high-dimensional data.
Treating ordinal as continuous Likert scales (1–5) are ordinal. Means are interpretable but formal inference should use ordinal tests or models unless scales are well-validated and items are summed.
For each scenario in Exercise 1, ask learners to work through the three decision-framework questions explicitly (outcome type, comparison, assumptions) before consulting the answer - this is the habit the whole session is trying to build, and it transfers directly to Exercise 3, where they apply it to their own research.
Estimated time: ~15 minutes (Practice)
For each scenario below, name the appropriate statistical test and state why:
Estimated time: ~20 minutes (Practice)
You have expression levels of a biomarker in healthy (n = 15) vs disease (n = 18) patients. The data look right-skewed with a few large values.
Estimated time: 15–30 minutes (Practice)
Take a published paper from your field. For each analysis reported: 1. Identify the outcome type and comparison being made. 2. Assess whether the correct test was used based on the decision tables above. 3. Note one assumption that the authors did or did not check, and state how it could affect the conclusions.
Estimated time: ~10 minutes (Self-test)
In a methods section: “The choice of statistical test was based on the outcome data type (continuous / binary / time-to-event), the number of groups compared, and whether the data met parametric assumptions (normality assessed by Shapiro-Wilk test and QQ-plot; homogeneity of variance assessed by Levene’s test).”
What to always state:
Template sentences:
Avoid: Choosing a test after seeing the results (outcome-dependent test selection inflates the false-positive rate). Pre-specify the primary test and document any post-hoc changes.