The international development community has witnessed a surge in theory-based evaluations that use counterfactuals, also called impact evaluations, to establish the attributable impact of development interventions. Amid this growing interest in using impact evaluations, challenges exist in using these methods for real-world programmes and investments.

Real-world impact evaluations are studies that evaluate and measure the causal change of government, non-government or multilateral assistance programmes that are planned and rolled out in real-world settings with (all the) associated political, data, implementation and resource constraints. This paper examines key questions that are typically associated with conducting impact evaluations in real-world settings, including costs, complicated requirements for implementation and the ambition of policy influence.

To illustrate constraints and challenges, we draw on impact evaluations funded through the Open Window funding modality of the International Initiative for Impact Evaluation (3ie). This is one of few funding modalities that awarded study teams for undertaking policy-relevant, real-world impact evaluations, unrestricted by funding size request or by subject area. All real-world impact evaluations examined in this study use counterfactuals where programmes were either allocated randomly or where quasi-experimental designs were used to mimic this random allocation.

This study finds that:

• Real-world impact evaluations experience many obstacles. The costs of these evaluations vary according to the question being asked. There is no one formula.
• Close monitoring of projects and the implementation of impact evaluations can help the projects and impact evaluations.
• Good formative research can help to reduce risks in exposing both critical bottlenecks and unexpected challenges in hypothesized theories of change.
• Building flexibility into grant windows can help funders manage their own expectations as well as ensure that teams are protected from reputational concerns.
• Building in a strong financial due-diligence process can help manage cost overruns.
• Early communication and planning between programme teams and impact evaluation teams is asine qua non.