sample(c(1:4),100,replace=T) [1] 2 3 3 1 1 4 1 4 3 3 1 3 3 4 2 4 2 3 4 2 4 2 2 1 4 3 4 3 2 3 3 1 4 3 2 4 3
[38] 1 3 3 2 1 3 2 4 3 3 4 3 2 1 3 1 3 2 3 2 4 2 3 3 4 4 4 1 3 2 2 2 3 1 4 2 1
[75] 1 1 1 4 2 4 1 3 1 4 1 3 1 1 4 2 2 3 4 4 1 3 3 3 2 3Experimental Design
Introduction
By now, you should be very comfortable with sample and population distributions, central moments, and (Generalized) Linear Models. In this chapter, we take a break from the math and code of statistical models to look at experimental design. This is not meant to be a comprehensive review, but rather a short reference guide that you can use to design better experiments or by applying your statistical inference skills.
Understanding the structure of data and the theory of linear models can help you develop a deeper understanding of experimental design covered in previous chapters. We’ll revisit assumptions about the statistical sample (i.e. random, unbiased) and its mathematical relationship to the statistical population. Thinking about these assumptions in practical terms is key to designing better experiments that produce more reliable inferences.
First, we will look at some problems that are commonly observed in real biological datasets. Then, we will discuss some remedies, and how these relate to different aspects of the (Generalized) Linear Model.
Before continuing, take a minute to imagine that you are responsible for designing an experiment. Pick an interesting and important biological question and imagine that money and resources are not limited. What kind of experiment would you design to definitively answer your question? What elements should you include? What factors do you need to consider? What will your experiment actually look like?
Now imagine that you do have limitations on sample size, measurements, and treatments. How does this influence your design? These are the realities that all biologists face, whether in their own experiments or in analyzing data collected by others. With these considerations in mind, let’s review some key considerations and design elements.
Bias
The word ‘bias’ has a specific definition in statistics, it is used to describe a systematic problem with the data collection or analysis that consistently leads to an incorrect conclusion. A statistical bias is anything that causes our sample estimates to consistently differ from the ‘true’ population value. One popular analogy is to consider a dartboard of accuracy and precision where the bull’s eye is the ‘true’ parameter of a population. Alternatively, we can frame the analogy as bias (the inverse of accuracy) and variance (the inverse of precision).
The population parameter (\(\theta\)) could be the population mean, or the variance, or an intercept or slope defining the relationship between two characteristics of the population. Each point is a sample estimate (\(x\)) of the population parameter, and the expectation (\(E(x)\)) is the weighted average across all of the individual sample estimates. We can define the bias mathematically as a deviation between the expected value (\(E(x)\)) and the ‘true’ value (\(\theta\)).
\[bias(\theta) = E(X) - \theta\]
As an example, consider a bias in the estimate of a population mean. In the Distributions Chapter, we defined the statistical population as a large or infinite number of individuals and the sample as an unbiased, random subset of the population. If we repeatedly sample in a random and unbiased way from the population, then the population mean will fall within within the range of \(1.96 \times SE\), and the expected average deviation will be zero:
\[E(\bar x) - \mu = 0 \]
In other words, each mean of any one particular sample (\(\bar x\)) will likely differ from the population mean \(\mu\) due to the effects of random sampling, but the average difference of many samples will converge on zero.
Imagine if we use a net to sample fish and then weigh them, but some fish slip through the holes so that they aren’t sampled. This will bias the average of our sample to be larger than the true population average. This will happen consistently, every time we collect a new sample and calculate the mean. This difference is the bias.
\[E(\bar x) - \mu = bias(\theta)\]
Bias can arise anywhere in the experiment, from preparing the experiments and collecting data to the choice and application of statistical models. It is unlikely that we can exclude all sources of bias in any study, but we can take steps to minimize it, identify potential sources of bias, and avoid incorrect conclusions.
It’s important to emphasize that the bias parameter doesn’t show up in the output of our statistical models. Quite the opposite, we must assume near-zero bias in the parameters. In most of the statistical models we’ve covered so far, we test a null hypothesis based on a \(p\)-value, which itself is based on a test statistic derived from an estimated parameter. For example, the \(p\)-value of a \(t\)-test is based on a sample mean, or the difference between two sample means. The \(p\)-value of a linear model is based on the estimate of a slope or group deviation. Therefore, a bias (i.e., \(\theta > 0\)) can result in both Type I errors (i.e., failure to reject a false null) and Type II errors (i.e., rejecting a true null).
Question: What are some specific examples of bias that could occur in the experiment you thought of in the previous section?
Sampling bias
Sampling biases are very common in biology and likely unavoidable. Even if the holes in your fish net are small enough to prevent a size selection bias, you are less likely to catch the individuals that are faster, or more camouflaged, or hiding in denser weeds. In many lab or field experiments we exclude individuals that die too early, which is not likely to be a random sample. Human studies usually require consent, which may be biased towards particular personalities, ages, and economic status. Sampling bias is unavoidable, and offers a good reason to be skeptical of most scientific research. The best we can do is focus on the specific population parameters we want to estimate, think carefully about the unmeasured population for which we are trying to draw inferences, and try to minimize sampling biases that could cause a bias in the sample estimate of those parameters.
Artifacts
An experimental artifact is a feature of the specific experimental conditions that produces a bias in the effect of a treatment. For example, you might make a mistake when mixing together your growth media or drug concentration. Perhaps a key instrument you were using to take measurements was mis-calibrated or not functioning properly. Maybe some of your samples were mislabeled as the wrong group. Biases from experimental artifacts are often unknown and may be impossible to identify, but it can be reduced through replication under different laboratory conditions and by different researchers.
Confounding Variables
A confounding variable is a source of bias that prevents proper causal inference in a statistical model. You may have heard the popular phrase “Correlation does not imply causation”. This means that you shouldn’t infer that \(A\) affects \(B\) just because \(B\) is correlated with \(A\). Instead, both \(A\) and \(B\) may be affected by another variable \(C\), or several other variables for that matter. In this case, \(C\) is the confounding variable. The confounding variable can cause bias in the parameter estimates of our statistical models. We saw some specific examples of this with collinear variables in the Advanced Linear Models chapter.
Space and Time are two variables to be particularly skeptical about, because so many biologically relevant factors change geographically and through time.
Confirmation Bias
Confirmation bias is the tendency to accept things when they conform to a prior belief and reject or modify them when they don’t. Confirmation bias is difficult to root out, because it can often be subconscious, yet influence all of the many small decisions you make when designing and executing your experiment, as well as the post-experiment quality checks, model choices, and interpretation. It happens more often in science then we would like to admit, and it is common among students who are new to science but want to make a big discovery, armed with flashy new statistical tools and hungry for a p-value less than 0.05. It’s not realistic to eliminate confirmation bias, but it is possible to train yourself to be more systematic in your approaches and less invested in the results.
Solutions
It is probably impossible to eliminate all potential sources of bias. The best we can do is to think very carefully about all of our potential sources of bias, consider which parameters we are interested in, and try to design better experiments with more robust analyses that minimize potential for bias in our parameter estimates. In field experiments and observational studies where we have less control over the experiments, we can try to identify data sources to try to account for confounding variables in our analysis. If there are too many confounding variables, then we might not have enough power to test them all, and some may be simply unknowable.
At the very least, the persistence of bias in scientific studies should make us very skeptical of individual studies that aren’t replicated by other researchers. We should also be wary of the social and political context of research and the blind spots that these can create in identifying and rooting out statistical bias.
Design Elements
Treatments
Treatments are usually imposed by the experimenter. For example, you may test different concentrations of a chemical or stimulus in a controlled environment. But treatments can also be ‘imposed’ in observational experiments. For example, we might compare plants that have naturally colonized wet vs dry environments or we might compare genotypes of an invasive species sampled from its native vs introduced range.
Control treatments are among the most important. Sometimes they are obvious (e.g. drug placebo) but other times they may not be. For example, if you want to test the effectiveness of a pesticide, what is your control group? It could range from doing nothing to spraying with water or water + the surfactant used to dissolve the herbicide. What if you just want to exclude herbivores to see how they affect plant growth? Which control would you use?
Treatments are typically encoded as categorical variables in linear models. However, if treatments fall along a gradient (e.g. drug or pesticide concentration) then they can sometimes be analyzed as a continuous variable to fit some known or hypothesized relationship. This is common in enzyme kinetics, for example.
Question: How could you model nonlinear relationships like these?
Answer: Review the Advanced Linear Models chapter if you don’t know.
Randomization
When treatments are imposed, then we can randomly assign individuals to different treatments. This is called randomization.
R can be a great tool for randomizing individuals.
Challenge: Write R code to help you randomly assign 100 individuals into 1 of 4 groups.
There are many ways you could do this One way is to use the sample function:
Haphazard
Haphazard is the more appropriate term to use when a human ‘randomly’ assigns individuals to treatments. A different term is used because humans are very bad at generating random numbers. Here’s a quick test: randomly choose 100 numbers from 1 to 100 and write them into a vector in R. Then plot a histogram and see if they follow a uniform distribution. Now plot a histogram with runif(100) and compare the distribution to the one you made.
Balance
One big problem with true randomization is that we can get uneven sample sizes in our treatments, which can be a big problem. In an extreme case, we could end up with a treatment that has no individuals in it. But even less extreme cases can be problematic.
Recall from our distributions tutorial that the the population mean falls within \(\pm 1.96 \times\) the standard error. The standard error is the standard deviation divided by the square root of the sample size. Therefore, smaller samples have higher standard errors, indicating more uncertainty. In a balanced design, all treatments have equal sample size, which gives the most power to detect an effect.
Replication
One way to keep samples balanced while still having an element of randomization is to use replicates. A replicate is just an experimental unit on which a treatment is repeated, and independent of other replicates.
For example, we might be testing the effect of a drug on mouse behaviour. We would of course want multiple mice for each treatment, so each mouse could be a replicate. But what if we have multiple mice interacting with each other in the same cage? Are these mice really independent?
Pseudoreplication
Pseudoreplication occurs when something looks replicated at first but upon further inspection the replicates are not independent. This is often due to a confounding variable. For example, if we have 100 cells growing in two petri dishes (1 treatment + 1 control), we have only 1 replicate because the 100 cells are pseudoreplicated within a petri dish.
Statistically, we can account for pseudoreplication using mixed models and other advanced statistics, as covered in the next chapter.
Blocking
Blocking is a good way to control confounding variables in large experiments by replicating treatments over space (typically called blocks) and/or over time (typically called cohorts). Importantly, treatments are typically replicated across blocks or cohorts, but randomly assigned across space and time within blocks.
In particular, many large experiments may involve spatial or temporal heterogeneity. Examples might include having different people collecting data, or different cohorts or batches of study subjects measured at different times, or replicates in different incubators or different areas of a field experiment where environments are slightly different. For example, if we are growing plants in a greenhouse, we might have experimental blocks that correspond to greenhouse benches. If we have 10 benches, then we have 10 experimental blocks. But within a block, we have the same number of treatments (e.g. 100 control + 100 treatment), randomly assigned to each individual.
In large experiments like these we can divide up our observations by observer, or growth chamber, or location as blocking effect in our statistical models. Statistically, we can include block, batch or cohort as factors in our (generalized) linear (mixed) models. Mathematically, this reallocates some of our residual error variance to among-block variance, which can increase the statistical power of our model to detect biologically meaningful effects.
Block effects, cohort effects, and batch effects are terms often used in the biological literature.
Blinding
Blinding is common in clinical trial studies but (surprisingly) rare in many other experimental studies. Single-blind studies are those in which the subjects do not know which treatment they have been assigned to (e.g. drug vs placebo). This is important in humans because simply thinking we are taking a drug can affect our physiological response. Yes, our body can actually change its physiology in measurable ways – this is known as the placebo effect. This is less relevant for studies of plants and animals, which presumably don’t know (or don’t care) which treatment they are part of.
Alternatively, double-blind studies are those in which the experimenters who are administering the treatment and collecting the data are also blind to the treatments. This is relevant to plant and animal studies because we as experimenters may slightly adjust our observations, even subconsciously, to match our expectations.
Double-blind studies are not common outside of clinical trials, but they should be!
Factorial Design
A factorial design is just like a factorial ANOVA, except that it must include every combination of factor. For example, if we test the effect of two treatments on mice in three environments, we may want to include an equal number of male and female mice in each treatment and environment. This would be a \(2\times 2\times 3\) factorial design with 12 treatements representing each combination of treatment, sex and environment.
Statistically, a balanced factorial design is a powerful way to look for non-additive effects. For example, if a treatment affects male and female mice differently, then there would be a significant treatment-by-sex interaction term in the model. Interactions were covered in the Advanced Linear Models chapter.
Nested Design
A nested design is like a factorial design except that the factors are hierarchical.
Genetic studies often include a nested design. For example, imagine a corn breeding study where we have pedigree information. We may have many seeds from different corn varieties. Within each variety we may have different parental families (i.e. individuals who share the same mother and father). In this example, we could analyze the effect of family, nested within population.
Nested designs are a good way to account for pseudoreplication. In this example, two seeds from the same plant share the same mother and father, so they are genetically more similar than the expected similarity of two randomly chosen seeds.
Statistically, nested designs are often analyzed with mixed models, as discussed later in the Mixed Models Chapter. Mixed models account for non-independence of samples, which can be a good way to deal with pseudoreplication.
Pilot Study
We have often discussed the importance of sample size in our statistical models. Sample size determines the ability of our model to detect a biological effect. One key question in experimental design is therefore: what should our sample size be?
Obviously, bigger is better. But every experiment has logistic constraints, so a better question might be: Will my sample size be big enough to detect an effect?
The answer depends on the desired precision and power of the experiment.
Precision
Precision is a measure of dispersion. A statistical parameter (e.g. slope) is more precise if it has a small confidence interval (CI) and less precise if it has a larger CI.
In a parametric model, the CI is \(\pm 1.96 \times SE\), which is \(\pm 1.96 \times (\frac{s}{\sqrt N})\)
Therefore, a higher N yields a more precise measurement. But this also depends on the standard deviation of the sample (\(s\)), which in turn depends on both the sample size and the standard deviation of the population (\(\sigma\))
Power
Power is related to precision, but instead of a confidence interval, we are interested in detecting an effect. An effect is a a statistical estimate, like a slope or the difference in the mean of two populations. Specifically, power is the probability to detect an effect at a specific level of alpha, given a particular sample size. Alpha is similar to the p-value of a model except that it applies to the population comparison – similar to the distinction between \(s\) and \(\theta\).
Sample Size
Because power and precision depend on the confidence interval (CI), we can reorganize the equation to figure out what sample size we would need for a given power or precision.
We can set the CI as a number representing the range (e.g. 10 cm or 30 mg) and rearrange our equation to determine a sample size. In the case of a normally distributed parameter, the width (\(R\)) of the CI is:
\[ R = 2\times1.96\frac{\sigma}{\sqrt{N}}\]
If we round 1.96 to 2 and solve for sample size, we get:
\[N = 16(\frac{\sigma}{R})^2 \]
To find a good sample size for the experiment, we can estimate \(\sigma\) by running a pilot experiment and measuring the standard deviation (\(s\)). Substituting this into the equation gives an estimate of the minimum number of samples we should aim for in our full experiment. Note that \(R\) could be the range of a confidence interval or the difference between groups when calculating sample size.
Data Loss
In most experiments there will be some observations that are lost. Plants die, patients drop out of studies, and data collectors make mistakes that may result in missing data.
It’s important to keep this in mind when designing an experiment. When desiging your experiment, it’s important to think about what might cause your sample size to decline, and what you can do to meet a target sample size by the end of the experiment – and ideally keep treatments balanced.
Summary
There’s a lot that goes into experimental design. Think of it more like designing a building or a car. The more thought you put into it, and the more feedback you get, the better your design will be.
Remember:
Garbage in = Garbage out– Experimental design has the biggest effect on the reliability of your analysis, no matter how advanced your statistical tools are.