Experimental Design

Designing an experiment may be viewed as involving the design of three distinct components: the response (what is to be measured), the treatments (combinations of factors related to the research goals and hypotheses), and the experiment (the manner in which the treatments will be applied to the subjects or units being studied). Exponent statisticians possess the training and experience to assist clients with projects ranging from the simplest, single-factor comparative experiments to more advanced studies using such tools as fractional factorial and response surface designs, to understand and optimize complex processes in the physical world.

Designed experiments, such as those conducted in manufacturing industries, involve the systematic variation of input quantities (e.g., raw materials, process characteristics) and observation of the resulting effect on output quantities (e.g., strength, yield). The purpose may be to screen a large number of potentially influential factors to determine which are most important and worthy of further study. Alternatively, interest may be focused on adjusting the values of key input factors to maximize the quality or quantity of the output product. Clinical experiments in the health sciences may be comparative studies to judge the effectiveness of a new product relative to an existing one. Engineering studies of component or system lifetimes may conduct accelerated testing under more extreme conditions than expected in the field, so that failures can occur within the constrained time period of the experiment.

Regardless of the nature of the experiment, the most common question asked of statisticians in this context is, “What size sample do I need?” But appropriate sample size is only one of the experimental design issues for which consultation with a statistician is warranted. Statisticians are typically most valuable at the earliest stages of a research project, because even the most sophisticated analysis cannot salvage a poorly conceived or executed study design. Involving the statistician early in the experimental design process can maximize the likelihood that sufficient and appropriate data are collected, and that the statistical analyses are both proper for the experimental data and are designed to answer the right questions.

Several basic questions require thoughtful consideration in assembling experimental evidence to address a research problem:


  • What is the goal of the research? Is the research exploratory (i.e., intended to search for relationships and generate hypotheses), or is the research confirmatory (i.e., intended to test previously posed hypotheses)? All aspects of the experimental design are influenced by the answers to this question.
  • What is to be measured? Care should be taken to ensure that the problem is defined with sufficient precision that meaningful research hypotheses can be formulated and evaluated with data that are available or can be gathered reliably. 
  • What statistical hypotheses are to be tested? Are these hypotheses directly related to the research goals? What statistical tests are to be used? Specification of hypotheses and statistical methods in advance of data collection effectively prevents “data snooping” (i.e., looking for patterns in the data and then testing for the pattern that has already been observed). The same experiment cannot be validly used in an exploratory manner to generate hypotheses and then reused to test those same hypotheses. 
  • How large an effect is important? Statistical hypotheses are frequently couched in terms of a null hypothesis (e.g., treatment A is equal to treatment B) and an alternative hypothesis (e.g., treatment A is better than treatment B). If treatment B is the standard treatment, how much better must treatment A be to make it a worthwhile replacement? Information on the size of an important difference is essential for the appropriate choice of sample size. 
  • How many observations are needed? Studies with too few observations lack statistical power (chance of finding a statistically significant effect) and cannot yield convincing answers to the questions that originally motivated the work. Studies with too many observations may misallocate precious resources and risk confusing statistical and practical significance.
  • Is the experiment worth the cost? If the number of observations needed to provide an adequate chance of finding an important result is too large, then a different experimental approach is needed. Sometimes this question is approached in reverse: given the sample size that fits the budget, what is the size of the effect for which there is adequate power?

To maximize the value of the experiment, Exponent statisticians will guide the experimenter though these questions and other practical issues, including how to randomize experimental runs to avoid order effects, when to use blocking to control the potential influence of secondary factors, what to do when some data are missing, and how to ensure quality control in collecting and recording data.

Professionals