Experimental design matters!

Food & Feed Safety RU Science Coffee Hour, May 28, 2025

Quentin D. Read, SEA Statistician

Key points

  • Experiments are imperfect models of the real world that help us understand cause and effect in the real world
  • The question you are trying to answer determines how you should set up your experiment — be aware of tradeoffs!
  • Designing an experiment right helps you maximize the ratio of signal to noise and get the best answer to the questions you care about

What is an experiment?

An experiment is a controlled procedure to compare hypotheses

  • Gather evidence to support or refute a hypothesis, or compare between competing ones
  • Give us information about causality: when x changes, what happens to y?
  • (Oversimplified) scientific method: make observations of nature, use observations + prior knowledge to make a hypothesis, test the hypothesis with an experiment

An experiment is a conceptual model of reality

  • “All models are wrong, but some are useful” - George Box
  • Experiments sacrifice realism to achieve greater control — Tradeoffs!

A super brief history of experiments

The first known experiments

Image (c) History of Islam

1000s: ibn al-Haytham experiments with lenses and mirrors to test the hypothesis that light goes into your eyes, not out of your eyes

Image (c) Franca Principe/IMSS Florence

1600s: Galileo experiments with steel balls to test hypotheses about how forces act on physical objects

The first randomized clinical trial

  • 1747: James Lind split 12 sailors into 6 groups of 2 men
  • Each group got a different treatment for scurvy: cider, sulfuric acid, seawater, etc.
  • The ones who ate oranges and lemons (Vitamin C) got better!

Experiments become more systematic

  • Late 1800s: the principles of experimental design we still use today were formalized
  • 1880s: Charles S. Peirce’s randomized controlled trial in psychology, testing sensitivity to pressure with a repeated measures design
  • 1920s: R. A. Fisher’s work on experimental design in agricultural science
    • That’s why people across all fields use words like “plots” and “blocks”

Peirce & Jastrow 1885

Experiments enter the modern era

  • 1948: First modern randomized controlled trial in medicine, testing effectiveness of an antibiotic at treating tuberculosis
    • Before this, anecdotal studies (which are still useful and important today) were all there was
  • 1950s: Japanese industry pioneers experimentation for statistical quality control of products
  • 2025+: What is the role of small-scale controlled experiments in the world of big data and artificial intelligence?

Image (c) Mazda

Basic concepts in experimental design

Terms to know

  • Treatment
  • Factor
  • Control
  • Experimental unit
  • Randomization
  • Replication
  • Local control
  • Optimal design

What is a treatment?

  • Treatment: Something that we manipulate or impose on subjects in an experiment
  • Factor: A variable whose levels are set by the experimenter; different treatments are different levels of a factor
    • Example: different doses of a medication, different management methods for an agricultural field

What is a control?

  • Control: Observations that show the effect of no treatment or a mock/placebo treatment; used to establish a baseline
  • What constitutes a control depends on what exactly you are controlling for!
  • Ideally, all variables we manipulate in an experiment have a non-manipulated counterpart in the control group(s)
  • Positive and negative controls
    • Negative controls provide a baseline to compare the treatment effect to
    • Positive controls ensure the procedure is giving expected results using standard/usual methods
  • In some designs, pre-treatment measurements serve as controls

What is an experimental unit?

  • The thing or object to which treatments are independently randomly assigned
    • Randomly assign a genotype to be planted on a plot of land
    • Randomly assign a vaccination treatment to be given to individual plant or animal
    • Randomly assign a management treatment to an entire experimental watershed
  • Not necessarily the same as the observational unit we make measurements on

Experimental unit: population of E. coli cells, 100 mL