Looking back on 2013, one of the major accomplishments for SFCG in the MENA region was completing our first program in Yemen. Because there is very little evaluation capacity inside of Yemen, we decided to take a two pronged approach to the evaluation. First, we hired a Yemeni research firm to conduct a large quantitative survey across several governorates to assess viewing and listening patterns among youth. Second, we hired an external evaluator from Jordan to come and conduct the qualitative focus group discussions, to do all of the data analysis, and to write the final evaluation report.
Here are a few of the learning takeaways from the final evaluation process:
1. Less experienced evaluators can still do a great job. For our final evaluation, we took a risk in hiring a newer evaluator because he had a strong writing sample and had worked closely with a very senior evaluator, who would supervise him. Throughout the process, he demonstrated a level of flexibility, eagerness to please, and effective listening that we valued. We also made sure to introduce him to DME For Peace as well as the OECD-DAC Peacebuilding Evaluation Guidelines.
2. Program Managers can learn a lot from supporting the evaluation process. Our Program Manager was very involved in traveling with the evaluator and organizing the final focus groups, some of which he attended where appropriate. This gave the program manager a chance to glean extra insights from the participants that didn’t make it into the final evaluation, and created a learning opportunity for the evaluation process. Now he seems extra invested in the design effort for the next stage of the program!
3. Invest time in developing good index questions. On my last support visit to the Yemen office, we brainstormed index questions for tolerance among youth across tribal and ethnic lines in Yemen. However, when we used these questions later in the survey, they yielded inconsistent results. In the future, we plan to spend more time and perhaps use focus groups to develop these questions so that we can use them reliably across programs in our work with youth.
Have you ever managed a mixed methods and mixed team evaluation in a country with low evaluation capacity? How have you managed to bring all of the necessary capacities to the evaluation process?
Thanks for you sharing the lessons about Yemen. I am an evaluation consultant based in Yemen. I am confirming that well qualified evaluators are limited in Yemen, and the evaluation culture as well. Please let me know when evaluating opportunities are raised in the future. Thanks again for sharing.