Experimental design matters!

Food & Feed Safety RU Science Coffee Hour, May 28, 2025

Quentin D. Read, SEA Statistician

Key points

Experiments are imperfect models of the real world that help us understand cause and effect in the real world
The question you are trying to answer determines how you should set up your experiment — be aware of tradeoffs!
Designing an experiment right helps you maximize the ratio of signal to noise and get the best answer to the questions you care about

Gather evidence to support or refute a hypothesis, or compare between competing ones
Give us information about causality: when x changes, what happens to y?
(Oversimplified) scientific method: make observations of nature, use observations + prior knowledge to make a hypothesis, test the hypothesis with an experiment

1000s: ibn al-Haytham experiments with lenses and mirrors to test the hypothesis that light goes into your eyes, not out of your eyes

1600s: Galileo experiments with steel balls to test hypotheses about how forces act on physical objects

1747: James Lind split 12 sailors into 6 groups of 2 men
Each group got a different treatment for scurvy: cider, sulfuric acid, seawater, etc.
The ones who ate oranges and lemons (Vitamin C) got better!

Late 1800s: the principles of experimental design we still use today were formalized
1880s: Charles S. Peirce’s randomized controlled trial in psychology, testing sensitivity to pressure with a repeated measures design
1920s: R. A. Fisher’s work on experimental design in agricultural science
- That’s why people across all fields use words like “plots” and “blocks”

1948: First modern randomized controlled trial in medicine, testing effectiveness of an antibiotic at treating tuberculosis
- Before this, anecdotal studies (which are still useful and important today) were all there was
1950s: Japanese industry pioneers experimentation for statistical quality control of products
2025+: What is the role of small-scale controlled experiments in the world of big data and artificial intelligence?

Treatment: Something that we manipulate or impose on subjects in an experiment
Factor: A variable whose levels are set by the experimenter; different treatments are different levels of a factor
- Example: different doses of a medication, different management methods for an agricultural field

Control: Observations that show the effect of no treatment or a mock/placebo treatment; used to establish a baseline
What constitutes a control depends on what exactly you are controlling for!
Ideally, all variables we manipulate in an experiment have a non-manipulated counterpart in the control group(s)
Positive and negative controls
- Negative controls provide a baseline to compare the treatment effect to
- Positive controls ensure the procedure is giving expected results using standard/usual methods
In some designs, pre-treatment measurements serve as controls

The thing or object to which treatments are independently randomly assigned
- Randomly assign a genotype to be planted on a plot of land
- Randomly assign a vaccination treatment to be given to individual plant or animal
- Randomly assign a management treatment to an entire experimental watershed
Not necessarily the same as the observational unit we make measurements on

Experimental unit: population of E. coli cells, 100 mL