“Statistically based experimental designs have been available for over a century. However, many preclinical researchers are completely unaware of these methods, and the success of experiments is usually equated only with ‘p < 0.05’. By contrast, a well-thought-out experimental design strategy provides data with evidentiary and scientific value. A value-based strategy requires implementation of statistical design principles coupled with basic project management techniques. This article outlines the three phases of a value-based design strategy: proper framing of the research question, statistically based operationalisation through careful selection and structuring of appropriate inputs, and incorporation of methods that minimise bias and process variation. Appropriate study design increases study validity and the evidentiary strength of the results, reduces animal numbers, and reduces waste from noninformative experiments. Statistically based experimental design is thus a key component of the ‘Reduction’ pillar of the 3R (Replacement, Reduction, Refinement) principles for ethical animal research.” Reynolds, 2024.
Introduction
A successful research study maximizes reliable information for minimal animals, but relying solely on p-values can lead to questionable practices and malpractice.
Evidentiary strength is determined by upstream study validity, which must be built into the experiment before data collection. Scientific value is created through a comprehensive and planned experimental strategy, which involves three phases:
- Quantitative framing of the research question
- Operationalizing the research question
- Implementing an appropriate experimental design
Framing the Research Question
A good research question is simple, straightforward, and answerable, making the hypothesis focused, specific, and measurable.
A practical research question is defined by input and output variables, refined using the PICOT acronym. Output variables are prioritized to provide critical information, and the study is powered and interpreted based on the primary outcome. The design includes sampling times and a clear time frame, especially if the outcome is survival to a predetermined time or endpoint.
Operationalising the Research Question
The experimental design operationalizes the research question, enabling reliable discrimination of true effect from noise. It directs data collection, determines appropriate statistical methods, and ensures correct error terms for testing statistical significance. Three steps include choosing inputs, identifying replication units, and selecting design structures.
Choice of Input Variable
Factors are independent variables chosen by investigators to study their effect on response. Factor levels consist of prespecified values for each factor, such as nominal or ordinal. For continuous factors, reasonable values are chosen to bracket biologically likely responses. For example, dose concentrations can be set at four levels: minimum (or zero) concentration (e.g. saline or vehicle control with no active drug), the maximum tolerable concentration, and two intermediate concentrations.
Identification of the Unit of Replication or Experimental Units
Statisticians use a treatment to combine input factors and factor levels, randomly assigned to experimental units (EUs) in an experiment. However, confusion between sample size and observational units (OUs) can lead to pseudo-replication, falsely inflated sample sizes, spurious precision, and increased false positive rates. Therefore, EUs and OUs must be clearly identified a priori.
Selection the Design Structure
The design is the formal structuring of the factors and factor levels. Basic designs are briefly described below (Figure 1).
Completely Randomised Design – CRD
Completely randomised design, CRD, is a method commonly used in animal studies, but it lacks precision and is not practical for multiple factors. It also leads to imbalances in sample sizes, increasing variance and reducing precision. Restricted randomisation is recommended to balance sample sizes and potential confounders or nuisance variables, which may influence response but are not directly relevant for testing the central hypothesis.
Randomised Complete Block Design – RCBD
Randomised complete block designs (RCBDs) are a type of restricted randomised design that conducts independent randomisation for each block, ensuring balanced sample sizes. They improve power and precision by removing variation between EUs from experimental error, and stratified randomisation reduces imbalance across potential confounders.
Nested or Hierarchical Designs
Nested (hierarchical) designs are useful for randomised interventions within naturally occurring organizational units or clusters, requiring identification of the unit of randomisation and intervention level at which interventions are to be applied.
Factorial Designs
Factorial designs are a more efficient and animal-sparing approach to studying multiple factors and interactions, allowing simultaneous examination with fewer experimental runs. They are particularly useful for exploratory studies due to their potential for evaluating interactions and unexpected results.
More Complex Designs
Modern computer-assisted screening designs are increasingly useful for large-scale studies, particularly in drug discovery. With the increasing availability of easy-to-use commercial software, these designs are expected to find wider application in exploratory animal-based research.
Implementing the Experimental Design
The researcher must consider statistically based methods for implementing a study, including treatment allocation, data collection, assessment, and reliability. Bias minimisation and variance minimisation ensure data reliability and consistency. Sequential experimentation is a strategic approach where previous results inform the design of subsequent experiments, increasing the certainty of success.
Mimimising Bias
Bias is the systematic deviation of estimates from true values, primarily minimized during planning and design phases. Major methods include randomisation and allocation concealment. Large sample sizes do not reduce bias.
- Randomisation
- Blinding
Mimimising Variance
Statistical methods such as blocking, stratification, and clustering can control variation in experimental designs by grouping organisms into homogenous subsets based on classification variables. Non-statistical methods like refined husbandry and handling can minimize between- and within-animal variation. To minimize process variation, standardization of protocols, training personnel, and regular quality assurance checks can be used. Non-technical management tools like process maps, checklists, and performance-tracking graphics can also help check the stability of experimental processes.
Sequential Experimentation
Sequential experimentation is a data-driven approach to large, sprawling experiments, allowing for intelligent modifications to sample size and design. This method increases power, efficiency, and reduces costs. A pilot phase is recommended to identify problems and standardize procedures before animals are used. Optimal screening designs are appropriate for intermediate studies, with graphing results assessing relative importance and effect sizes.
What Comes Next? The Statistical Analysis of the Data
The experimental design structure and response variable determine the most suitable analysis methods. Conventional ANOVA models may be suitable for most analysis purposes if the design structure is preserved and normality, homogeneity, and independence of observations are met. However, non-normally distributed response data and serially correlated observations are common in biological research. More sophisticated analysis methods are necessary to accommodate these data types and study designs.
A Statistical Analysis Plan, SAP, should be put in place in the study design phase.
Summary
- Study design in animal-based research is often overlooked due to inadequate training in statistically based design principles.
- Researchers often focus on technical aspects of experiments, interpreting results based on small p-values without considering internal validity.
- Scientific statistics require a complete understanding of the statistical process, starting with actionable questions and bespoke statistical methods.
- A good design ensures appropriate animal numbers and valid, reliable results.
- Statistically based experimental design is a key component of the Reduction pillar of the 3R principles for ethical animal research.
Reference
Reynolds PS. Study design: think ‘scientific value’ not ‘p-values’. Lab Anim. 2024 Oct;58(5):404-410. doi: 10.1177/00236772241276806. PMID: 39365003.