To draw general conclusions about a population, it is necessary to sample its variation.
Introduction
Variability in experiments is inevitable due to biological and technical effects. Controlling technical variability enhances internal validity, while maintaining biological variability allows generalization to the population. Experimental control, randomization, blocking, and replication help.
This paper delves into the impact of variability on the ability to replicate experimental results (internal validity) and generalize them to the population (external validity).
Altman, N., Krzywinski, M. Sources of variation. Nat Methods 12, 5–6 (2015).
The Importance of Understanding Variation
A well-designed experiment balances internal and external validity, aiming to observe reproducible effects due to treatment and measure variability for replication with similar but not identical samples.
The numerous sources of biological variability in mice make it impossible to adequately control the treatment when it is administered in vivo so that different animals have the same measurements. However, by carefully regulating some of these variables, we can lessen the effect of this variability on our measurements.
Examples of sources of variability that are completely controlled by experiment include gender and genotype. By choosing one level, we can completely remove the source, or we can choose multiple levels to ascertain the effects. We can treat gender as a fixed factor in our experiment since we can observe all of the levels that are possible. A genotype may have a random (noise) effect (many wild-type strains reflecting the wild-type population) or a fixed effect (specific genotypes of interest, such as a mutant and its background wild type). We can only draw broad conclusions about the effect of a treatment by purposefully creating variability, and even then, only across the factors that were changed.
Statistical Tools to Handle Variation Adequately
There is some experimental control over other sources of variability, including housing effects, temperature, and diet. Random assignment1 (to prevent bias), replication2 (to improve precision), and blocking3 (to isolate noise) can be used to manage uncontrollable or unknown noise factors.
Conclusion
In order to measure mean biological effects, we must separate sources of variation that are nuisance factors from those necessary to determine the extent of population variation in effects. The latter must be sampled and quantified in order to both robustly determine the uncertainty in our estimates and generalize our conclusions, while the former should be minimized in order to maximize the experiment’s power.
References
Altman, N., Krzywinski, M. Sources of variation. Nat Methods 12, 5–6 (2015). https://doi.org/10.1038/nmeth.3224
Krzywinski, M. & Altman, Nat. Methods 11, 597–598 (2014).