Conservation, restoration and improved land management activities could potentially provide more than one-third of the cost-effective climate change mitigation needed to keep global warming below 2°C by 2030, with forests in the driver’s seat. But do we know how much mitigation different forest-based actions have so far achieved? Which ones made a difference? We argue that rigorous impact evaluations of forest interventions are needed to understand which forest-based actions work best – including where, when, why, and at what cost – to promote learning and inform future program design and implementation.

The journal PLOS ONE recently collected new quasi-experimental case studies quantifying the conservation impact of various forest policies and programs, including protected areas, law enforcement, payments for environmental services (PES), certification, and community-based natural resource management (Börner et al. 2016). A quick glance at results (see Figure 1) underlines that impact estimates are often small, frequently imprecise (with ‘long whiskers’, i.e. large variance around a central point), and that location bias is common.

Arguably experiments (e.g. randomized control trials) are important for establishing internal validity i.e. understanding and measuring if interventions are causing an effect and measuring the magnitude. However, what the PLOS group of papers does not do, is help understand whether these results can be used for other contexts. In other words, their ‘external validity’ is limited. This is because these are single site studies.

CIFOR’s Global Comparative Study on REDD+ aims to deal with this limitation. By focusing on multiple sites and evaluating the impacts of different REDD+ interventions on forest conservation (forest cover, land use change) and local livelihoods (income, assets, tenure security and perceived wellbeing) we are building a ‘multi-site’ study. Our work focuses on 22 sub-national REDD+ sites in six countries: Brazil, Peru, Cameroon, Tanzania, Indonesia and Vietnam (see Figure 2). The study combines socioeconomic surveys in 150 villages and nearly 4,000 households (including control groups) with Global Forest Change data (2000-15) and measures the effect of REDD+ interventions on a set of common outcome indicators.

So far, two rounds of panel data have been collected. The first round of data was collected in 2010-12 (pre-intervention) and the second round was collected in 2013-14 (early post-intervention). A third round is planned for 2018 to consolidate evaluated post-intervention impacts.

Findings to date highlight some important effects: First, REDD+ interventions have caused reductions in village-level tree cover loss in more than half of the REDD+ sites studied However, effect sizes are small (Bos et al. 2017; see Figure 3). Second, so far, we have observed no systematic negative impacts of REDD+ on local wellbeing at these sites. But also little evidence of significant co-benefits. Third, we find that despite substantial attention by REDD+ implementers to address local tenure issues, there is no clear effect of interventions on tenure security (Sunderlin et al. 2018). Lastly, although heavy-handed restrictions seem to be most effective in curbing forest clearing, on their own they negatively affected tenure security and local perceptions of well-being. Yet when alternative livelihood enhancements and incentives were added to the mix, negative well-being effects were reduced (Duchelle et al. 2017).

Emerging lessons from this study also have important learning for methods that evaluators may employ for understanding these interventions. For instance, we find that the approach we are using (Before-After Control-Intervention, or BACI) is sensitive to the choice of controls. Scales are important too: impacts of pilot actions tend to dissipate at higher scales, i.e. at the larger site-level interventions had smaller impacts (Bos et al. 2017; see Figure 3). Overall, slow REDD+ implementation so far has meant low ‘treatment intensities’, which together with a complex REDD+ policy mix (regulatory disincentives, incentives, enabling measures) makes it hard to tease out attributable impacts from specific interventions in the REDD+ bundle, especially in forest frontier settings with multiple, rapid forces of change. Yet, we are attempting to do just that, through linking the share of households treated by specific interventions to observed environmental and social outcomes.

Note: Scores are grouped into good, neutral and poor, where a good score means a relative reduction in tree cover loss over time (BA, BACI) and/or compared to the control area (BACI), and where n is the number of REDD+ initiatives in each category. Adapted from Bos et al. 2017.

This research contributes to the call for “Conservation Evaluation 2.0” in jointly comparing measured conservation and livelihoods outcomes of different policy tools applied across sites, through carefully matched treatment and control localities. Given the high stakes for forest policies and programs to provide both climate and development benefits, and early indications that conservation impacts can be highly variable across contexts and intervention types, knowledge is key: building the best possible evidence base for understanding the performance of forest-based actions on the ground is vital, prior to taking them to scale. Indeed, program planners and evaluators would benefit from a few conclusions if they aim to build robust evidence on impact. First, early planning and up-front investments to identify control groups that represent a counterfactual scenario is key. Second, collecting baseline data is critical. This is often in an narrow time window but is really important to determine whether and which outcomes can be attributed to different forest interventions (Sills et al. 2017).

Disclaimer: The views expressed in this guest blog are the author's own and do not necessarily reflect the views of the Independent Evaluation Unit of the Green Climate Fund.