Experimental design matters!
Food & Feed Safety RU Science Coffee Hour, May 28, 2025
An experiment is a controlled procedure to compare hypotheses
- Gather evidence to support or refute a hypothesis, or compare between competing ones
- Give us information about causality: when x changes, what happens to y?
- (Oversimplified) scientific method: make observations of nature, use observations + prior knowledge to make a hypothesis, test the hypothesis with an experiment
An experiment is a conceptual model of reality
- “All models are wrong, but some are useful” - George Box
- Experiments sacrifice realism to achieve greater control — Tradeoffs!
The first known experiments
1000s: ibn al-Haytham experiments with lenses and mirrors to test the hypothesis that light goes into your eyes, not out of your eyes
1600s: Galileo experiments with steel balls to test hypotheses about how forces act on physical objects
The first randomized clinical trial
- 1747: James Lind split 12 sailors into 6 groups of 2 men
- Each group got a different treatment for scurvy: cider, sulfuric acid, seawater, etc.
- The ones who ate oranges and lemons (Vitamin C) got better!
Experiments become more systematic
- Late 1800s: the principles of experimental design we still use today were formalized
- 1880s: Charles S. Peirce’s randomized controlled trial in psychology, testing sensitivity to pressure with a repeated measures design
- 1920s: R. A. Fisher’s work on experimental design in agricultural science
- That’s why people across all fields use words like “plots” and “blocks”
Experiments enter the modern era
- 1948: First modern randomized controlled trial in medicine, testing effectiveness of an antibiotic at treating tuberculosis
- Before this, anecdotal studies (which are still useful and important today) were all there was
- 1950s: Japanese industry pioneers experimentation for statistical quality control of products
- 2025+: What is the role of small-scale controlled experiments in the world of big data and artificial intelligence?
What is a control?
- Control: Observations that show the effect of no treatment or a mock/placebo treatment; used to establish a baseline
- What constitutes a control depends on what exactly you are controlling for!
- Ideally, all variables we manipulate in an experiment have a non-manipulated counterpart in the control group(s)
- Positive and negative controls
- Negative controls provide a baseline to compare the treatment effect to
- Positive controls ensure the procedure is giving expected results using standard/usual methods
- In some designs, pre-treatment measurements serve as controls
What is an experimental unit?
- The thing or object to which treatments are independently randomly assigned
- Randomly assign a genotype to be planted on a plot of land
- Randomly assign a vaccination treatment to be given to individual plant or animal
- Randomly assign a management treatment to an entire experimental watershed
- Not necessarily the same as the observational unit we make measurements on
Experimental unit: population of E. coli cells, 100 mL
Experimental unit: drainage basin, ~1 km2
Treatment structure
- Examples: one-way, factorial, regression, response surface
- Factorial design: multiple factors with different levels, experiment includes combinations of levels across factors
- Common garden experiment to estimate G × E interaction
- More interactions = more realistic, but you need more samples and it’s harder to interpret — Tradeoffs!
Randomized complete block design (RCBD)
Each block is a complete replication: it contains every treatment combination once
Split-plot design
- Two (or more) levels of randomization
- One treatment is randomized at main plot level and another at subplot level
- Still a complete block design: all treatment combinations appear in each block
- May be used if it is logistically difficult to completely randomize
- Compared to RCBD, increased power to detect subplot treatment effect and interaction, but less power to detect main plot treatment effect
Trouble in RCBD paradise
- ARS scientists love randomized complete block designs, but that’s not all there is
- Other designs may be more efficient and give you more power for the same number of experimental units
Augmented design
- Controls or checks replicated in a standard experimental design (often RCBD)
- Most treatments unreplicated or have fewer replicates
- Error from checks used to evaluate the unreplicated treatments
- Useful for breeding trials where many genotypes are being evaluated
Other designs
May be chosen because they are optimal to detect certain effects, or because of logistical constraints
- Strip-plot
- Two treatments applied crosswise to each other
- Row-column designs
- Blocking in row and column direction
- Special case: Latin square (an underrated and efficient design)
- \(\alpha\)-lattice (type of incomplete block design)
- Blocks are grouped so that each group has one complete set of treatments
- As balanced as possible
Blocking in space versus time
- All the above designs are not just for spatial arrangement of plots
- Experimental “runs” or “batches” may also be considered blocks
- Growth chamber studies may be blocked in time
Blocks should be spatially compact
Units within the same block should be closer to each other than they are to units in other blocks
Pseudoreplication
- Pseudoreplication: Incorrectly considering experimental units to be independent, when they are correlated
- Confusion between experimental unit and observational unit
- May lead to confounding where we can’t distinguish different influences on the outcome
- This may be OK, but you must consider what population you can make inference about
Faulty randomization
What’s wrong with this experimental design? (Set up as RCBD)
- This design would be OK if the whole thing were replicated multiple times
- Or try a spatial random effect, but variation in the x direction is always going to be confounded with treatment
Statistical power
- Power: The probability that you will be able to statistically detect the phenomenon you care about, if it is real
- Power depends on the sample size, the design, and what effect you are trying to estimate
- Usually we design an experiment to get a desired level of statistical power
- We have to balance between false positives and false negatives — again, tradeoffs!
Fixed or random? (continued)
- Whether an effect is treated as fixed or random depends on your question (for example, time in an experiment repeated over multiple years)
- We can never reproduce the exact combination of environmental variables from a past year in a future year
- The question of “fixed or random” is less important than making sure all relevant influences on your response variable are accounted for in your model
- If unsure, consult a statistician!
Caution: experiments aren’t everything
- R. A. Fisher (again) took the idea that correlation does not imply causation too far
- Exploited pro-experimental bias to discredit anyone trying to link tobacco use to disease
- Just because randomized trials are the best way to conclusively establish causality, does not mean they are the only way or the best way to obtain knowledge
- We now have statistical methods to robustly infer causality from observational data
- Even if experiments establish that one thing causes another in controlled conditions, those are artificial and don’t “prove” that something will work in the real world
