A framework for statistical power

• Classical statistical framework: use evidence from the data to try to reject a null hypothesis
• We can never be 100% certain of the truth
• We might be wrong because of natural variation, measurement error, biases in our sample, or any/all of those

POWER: If an effect exists, the chance that our study will find it

• Power is important for both experimental and observational studies!

Definition of statistical power

• Power of a statistical test: the probability that the test will detect a phenomenon if the phenomenon is true
• Power is the probability of declaring a true positive if the null hypothesis is false
• In contrast, the p-value is the probability of declaring a false positive if the null hypothesis is true

You can’t be right all the time

Magic number: false positive rate

Magic number 2: false negative rate

What’s wrong with an underpowered study: false negatives

• An underpowered study is one that has a low probability of detecting a significant effect, if the effect truly exists in the world
• Imagine we do an underpowered study and get a negative result: we can’t reject the null hypothesis
• Because the power is low, we don’t know if we got a true negative or a false negative
• No new knowledge was gained, we might as well not have done the study at all

What’s wrong with an underpowered study: false positives

• But what if the underpowered study gives us a positive (significant) result?
• Some people say that means the study was powerful enough after all, and power only matters if we don’t get a significant result

False positives in underpowered studies, continued

• At small sample sizes when power is low, natural variation/measurement error/sampling bias can cause overestimates of the effect size
• At low power, even though false positive rate is fixed at 0.05, the ratio of false positives:true positives increases
• A single positive result is not very convincing if it comes from an underpowered study
• Publication bias in favor of positive results from small studies makes the problem worse

Power analysis is a ballpark figure

Err on the side of caution

What do we need to know to do a power analysis?

• Significance level
• Power
• Sample size
• Effect size

(Un)known quantity 1: Significance level

• Many stat analyses in ag science assume \(\alpha = 0.05\)
• But \(\alpha\) may need to be adjusted if you have multiple comparisons
• This adjustment is often (wrongly) neglected when doing power analysis

(Un)known quantity 2: Power

• Sometimes power analysis is done assuming a desired level of power (often 80%, or \(\beta = 0.20\))
• Or we may estimate the power to detect a given effect at a given sample size

(Un)known quantity 3: Sample size

• We may know how many samples/replicates we can (afford to) collect or measure
• Or we may calculate the minimum sample size needed to get a certain power
• Often there is a range of feasible sample sizes, not just a single number

Sample size: the more the better, up to a point

(Un)known quantity 4: Effect size

• The size of a practically meaningful effect that you want to be able to detect statistically
• Often not a single number but a range
• Probably the most important quantity for power analysis
• Also the one that takes the most effort to determine

Statistical power increases as effect size increases

Standardized effect sizes

• Effect sizes are almost always in standardized units
• Different studies measure response variables in different units (biomass in kg or g or lb, height in cm, m, or in)
• If we convert to standard deviation units, we can compare results that are on different scales

How to get information about effect size

• Data from your own lab
• The literature
• Prior external information
• Effect size benchmarks

Effect sizes from data

• Find datasets that are roughly comparable to what you expect in your proposed study
  • Datasets where your response of interest is measured in a similar system
  • Datasets where a different variable is measured in your study system
• Calculate effect sizes from the prior datasets, and design your study around that effect size

Effect sizes from the literature

• Find studies on a similar topic or done in a similar system
• Even without their raw data, you can still calculate effect size from summary statistics, tables, or figures
• Essentially a quick and informal meta-analysis
• Quality/relevance of the studies is more important than quantity

Example: impact of transgenic Bt crops on abundance of soil organisms

Distribution of Cohen’s d from the published tables

What effect size to use?

• There is no single correct answer; it depends what you can justify
• We could use the median (\(d = 0.46\), a medium effect size), or a lower quantile
• Or you could use this distribution to get an idea of the power for a predetermined design
  • For example, if your study has 80% power to detect \(d \geq 0.3\), you’d be happy
  • If your study has 80% power to detect \(d \geq 0.6\), back to the drawing board!

Effect sizes based on external criteria

• You are a domain-knowledge expert so you can justify effect sizes without any prior data
• What effect size is biologically/practically/economically relevant?
• You can consult with stakeholders to determine this
  • Example: a stakeholder is interested in a treatment that will increase yield by 2 bushels/acre, or decrease mortality by 25%
• The problem is often coming up with a way to convert the stakeholder’s wishes to a standardized metric

Effect size based on no prior information

• I often hear “Nothing like this has ever been done before, so I have no idea what the effect size should be”
• Usually this indicates a failure to think creatively
• You don’t need exactly the same study, anything with a similar response variable, or a different variable in a similar system, will do

Happy medium

• Common practice: target medium effect size
• Designing a study to detect a small or weak effect requires a huge sample size
• Designing a study to detect large effects is not worthwhile
• I do not recommend blindly using the medium effect size all the time
• The benchmarks are very general across fields and may not be appropriate for your study system
• No substitute for finding relevant data or consulting experts/stakeholders

Plug-and-chug power analysis

Sample size given power and effect size

• We are comparing four different groups in our study
• We want to have 80% power to detect a moderate effect size f = 0.25
• How many samples do we need?
• Formula says n = 45 per group

Effect size given power and sample size

• We can only afford 60 total samples (15 per group)
• What effect size can we detect at 80% power?
  • minimum detectable effect size (MDES)
• Formula says we can detect f = 0.44

Power given effect size and sample size

• Our preliminary analysis tells us we can expect effects around f = 0.4
• We can afford n = 60 samples
• What is the power of the study?
• Formula says power is 71%

Why do we need to do power analysis by simulation?

• Plug-and-chug power analysis is the exception, not the rule
• Most experimental and observational study designs in ag science require mixed models to analyze
• There is not a simple equation for power
• Simulation to the rescue!

Basic power simulation workflow

1. Generate a dataset using a given set of assumptions (effect size, sample size, variance components, etc.)
2. Fit a statistical model to the dataset and calculate a p-value for your hypothesis of interest
3. Repeat n times
4. Your power is the proportion of times, out of n, that you got a significant p-value

A simple simulation example: t-test

• Assume the following:
  • Treatment mean = 12
  • Control mean = 9
  • Standard deviation of both groups = 3
  • Sample size of both groups = 30
  • Individuals are normally distributed around group mean
• Simulate 10000 datasets and do a t-test on each one
• Proportion of the time that \(p < 0.05\) is the power!

Simulation gives same result as formula

set.seed(1)
p_vals <- rep(0, 10000)

for (i in seq_along(p_vals)) {
  x_t <- rnorm(n = 30, mean = 12, sd = 3)
  x_c <- rnorm(n = 30, mean = 9, sd = 3)
  p_vals[i] <- t.test(x_t, x_c)$p.value
}

mean(p_vals < 0.05)

[1] 0.9682

pwr.t.test(n = 30, d = (12 - 9)/3, sig.level = 0.05)$power

[1] 0.9677083

Power by simulation for mixed models

Mixed models have more unknowns

Power simulation

• How many replications should we include in a study design?
• Here we will simulate random effects for new datasets with different numbers of blocks (reps)
• Repeat many times for each number of blocks
• Do a statistical test of choice on each simulated dataset
• Calculate power as % of times the test rejects the null hypothesis

Power curve from simulation

With power analysis, more effort = better results

Power analysis is iterative

Type S and Type M error: another way to be wrong

Post hoc power analysis is pointless

• Reviewers in applied science journals often ask scientists to do a power analysis on a study after the fact
• But if you do a power analysis assuming the observed difference is the true difference, you will always find low power if you didn’t detect, and high power if you did detect
• If the sample size is low, we don’t know whether the observed difference is the true difference
• You already knew if the study was underpowered before starting the study
• The power of a study after the fact is a mathematical function of the p-value

Power isn’t everything

• It is not wrong to do an underpowered study, if it is the only way we can learn about a system
• It may be prohibitively expensive to add more experimental units
• It is especially important to be careful and precise with measurements and to ensure your small sample is representative
• Draw conclusions with caution, and be explicit about the limitations of your inference

What did we learn today about power analysis?

• Power analysis tells you how much you can trust the answers you get to the questions you care about
• Determining effect size(s) is the most difficult but most important part of power analysis
• Power analysis often requires you to simulate data based on assumptions
• The more effort you put into a power analysis, the better informed your assumptions are by real data, the better your power analysis

Final recommendations

No power = no new knowledge = no science