Choosing What to Learn: Experimental Design when Combining Experimental with Observational Evidence
Abstract
Experiments deliver credible but often localized effects, tied to specific sites, populations, or mechanisms. When such estimates are insufficient to extrapolate effects for broader policy questions, such as external validity and general-equilibrium (GE) effects, researchers combine trials with external evidence from reduced-form or structural observational estimates, or prior experiments. We develop a unified framework for designing experiments in this setting: the researcher selects which parameters to identify experimentally from a feasible set (which treatment arms and/or individuals to include in the experiment), allocates sample size, and specifies how to weight experimental and observational estimators. Because observational inputs may be biased in ways unknown ex ante, we develop a minimax proportional regret objective that evaluates any candidate design relative to an oracle that knows the bias and jointly chooses the design and estimator. This yields a transparent bias-variance trade-off that requires no prespecified bias bound and depends only on information about the precision of the estimators and the estimand's sensitivity to the underlying parameters. We illustrate the framework by (i) designing small-scale cash transfer experiments aimed at estimating GE effects and (ii) optimizing site selection for microfinance interventions.