LORTA Workshop 2019 – Mastering the art of measuring impact

February 2019 marked the first month of my internship with the Independent Evaluation Unit (IEU) at the Green Climate Fund (GCF). The period since that time has been challenging, but also served as a great opportunity for growth. Over the past few months, I have been actively engaged with the Learning-Oriented Real-Time Impact Assessment (LORTA) team at the IEU. This experience has given me a greater appreciation of evidence-based analysis and evaluation in the climate change field.

LORTA is a program run by the IEU to inform the GCF on the magnitude, scalability and significance of the impact of GCF projects with the help of high-quality data and theory-based experimental and quasi-experimental methods.

A few weeks ago, I worked alongside IEU colleagues organizing the LORTA Design workshop conducted at the University of Mannheim in Germany April 15-17. The objective of the workshop was to discuss the role and opportunity for project teams to incorporate rigorous impact evaluations into their project designs. More specifically, the workshop provided an opportunity for discussion for the more than 90 participants, including representatives from 21 GCF and private sector funded adaptation and mitigation projects, the GCF secretariat, the Center for Evaluation and Development, the Consultative Group on International Agricultural Research, the Center for International Forestry Research, academia, and the private sector.

Before the event took place, there was much behind the scenes preparation. These tasks included arranging lodging for participants, organizing the agenda, finalizing communication materials, and reviewing many proposals and preparing program briefs. Through a coordinated team effort, we were able to organize everything smoothly.

The 3-day workshop focused on several keynote speakers from the impact evaluation and behavioral science fields.
 

The opening day emphasized the role and importance of impact evaluation. Dr. Joerg Faust, Head of Deval, presented an informative session on the power of impact evaluation, particularly on implications for policy and development cooperation. Dr. Jyotsna Puri (Jo), Head of IEU, continued this theme by delving into a discussion of impact evaluation methods and asking, ‘what are the techniques and strategies for measuring and evaluating impact?’

Day two began with an interesting dialog on how to engage the private sector in impact evaluation. This discussion was facilitated by University of Ottawa Honorary Professor, Dr. Edward Jackson. Mr. Jackson emphasized the importance of aligning social and environmental impacts along with financial returns in private sector development projects.We also heard from Dr. Markus Froelich, Head of the Center for Evaluation and Development (C4ED). In this session Mr. Froelich illustrated a real-world example of implementation tracking in a GCF project in Zambia.

On the final day, Dr. Elspeth Kirkman, Head of Behavioral Insights, explained the influence of behavior and human psychology, focusing on the effectiveness of development projects and the magnitude of their impact.

Between the keynote sessions, I had the opportunity to participate in five practical tasks related to impact evaluation. These tasks were fundamental to my learning experience.

The first task of the workshop highlighted program modalities and evaluation. Interestingly, I observed that some participants had already formulated the evaluation questions they would like to answer with the help of impact evaluations, whereas others took longer to identify questions.

For the second task, the focus turned to discussion of theories of change (TOCs), where project participants addressed common evaluation questions by identifying assumptions and critical bottlenecks.

On the third task, project representatives worked with session experts to better craft more effective evaluation designs. Special attention was given to using counterfactuals and finding realistic but creative ways to incorporate these into project implementation.

Task four dived into sample size and power calculations. Mock-exercises with hypothetical samples were given for participants to test their skills and ask questions. This activity was extremely helpful. It is notoriously difficult to predict the actual sample size needed for evaluations, but sample size calculations are critical because they influence costs.

The final task on implementation served as a space for project representatives to showcase their experience with LORTA’s 2018 projects. This session provided a great benchmark for workshop participants and demonstrated the real-world challenges and opportunities that occur in the implementation process. Engaging in these activities provided me with a greater understanding of how impact evaluation is carried out in the field.

For more information on challenges and learning from real-world impact evaluations check out our new working paper.
 

My major takeaways:

• Venue. Finding a suitable space for a workshop is essential to the overall outcome. When selecting a venue, consider such details as proper lighting, clear audio quality, and sufficient working space. 
• Brevity is key. It is essential speakers stick to the agreed time-limit for their presentations, which should be brief and to the point. This ensures the audience remains focused during the session. A good example of this was on the last day during a ‘rapid-fire’ session where project teams had three minutes to present their evaluation strategies (and what they learned over the past two days).
• Foster Interaction. Interactive exercises and quizzes are a great way to spice up lectures and spur creative brainstorming.
• Think collaboratively. It is important to ensure everyone feels they have a stake in the overall success of the workshop.   
• Encourage friendly competition. The poster contest was a great way to foster interest and participation among attendees. Who doesn’t love a bit of friendly competition?
• Highlight Q & A. An exciting Q & A session serves as the foundation for a successful workshop. It leaves an important space for learning and dialog of diverse viewpoints.

 

Closing Thoughts

Organizing and attending the LORTA workshop was a unique opportunity to further explore the insights and challenges of impact evaluation in the climate finance field. Also, through our hands-on involvement in the task sessions we gained a to better understanding of the projects run by the GCF and the IEU’s role.
 

My active involvement in the LORTA workshop has made me even more excited about IEU’s real-time impact evaluations and the potential for providing valuable feedback to all of GCF. LORTA is on an important pathway towards identifying areas of research for future exploration that will quantify and measure paradigm shift in global climate action.

For recordings from the workshop, stay tuned for our new #IEUTalks podcast featuring the latest speakers from our events.

Disclaimer: The views expressed in blogs are the author's own and do not necessarily reflect the views of the Independent Evaluation Unit of the Green Climate Fund.