Innovative evaluations of complex programs: The experience of the Small Producer Program in Mexico

Leer en español

Authors: Erick Herrera-Galvan and Karla Pinel | Country: Mexico and Costa Rica

The context of small-scale farmers 

It is estimated that small-scale farmers account for more than 500 million households in the world, equivalent to more than two billion people. In general, these farmers share challenges such as poverty, low level of literacy, low land productivity due to the use of traditional farming practices and limited access to improved agricultural technologies, poor access to credit, lack of producer organizations, and limited access to modern markets (Lowder, S. K., Skoet, J., Raney, T., 2019). 

In Mexico, about 70% of the Rural Economic Units are households that are in a situation of poverty. These mainly do agriculture and cultivate small areas (less than 5 hectares); about 20% are dedicated to subsistence activities, while 50% sell their surplus production on the market through a network of intermediaries and wholesalers who capture a significant portion of the final price. This contributes to the concentration of poverty in the countryside. (SAGARPA, 2014; COFECE, 2015) 

These constraints result in the production of small quantities of crops that end up being sold locally at the beginning of the supply chain, limiting their access to the most lucrative markets (IFAD, 2016). Therefore, it is vital to enhance the productivity of small farmers to improve their conditions and thus ensure an inclusive development process. In response to this global challenge, multiple and varied interventions have been implemented worldwide with the aim of improving small farmers’ productivity, increasing their access to markets, and reducing rural poverty. 

Some of these interventions have been evaluated, in these it has been found that components such as the supply of agricultural inputs, the development of agronomic and financial capacities, facilitating access to financial services, infrastructure and new technologies, and the creation of direct business linkages between small farmers and end buyers, effectively address the causes that limit small producers’ access to markets. 

Case: Evaluation of the Small Producer Program in Mexico 

The Small Producer Program of the Walmart Mexico Foundation is an initiative that began operations in 2011 and to date has worked with small agricultural producers in 14 states of Mexico. Its three main components are to offer agronomic and business training, as well as to provide commercial support to small producers and facilitate their linkage with large buyers. 

After six years since this program began and seeking to enlarge it to increase its impact on agricultural production, taking into account the imminent complexity it represented, its management decided that it was time to undertake an evaluation of the intervention model. The evaluation process was a challenge in itself, since the very design of the intervention and the complex context (in terms of the number of activities, actors and intervening factors) in which it was implemented made it difficult to establish linear theories and assumptions about its functioning and the achievement of its results. 

Thus, the evaluation team, together with the directive, strategic and implementing parts of the program, carried out an evaluation process that consisted of four stages and whose evaluation questions were the following: 

• Diagnosis: Is the program based on an updated analysis of the problem of interest?
• Theory of Change: is the program supported by a clearly defined, evidence-based intervention model?
• Design: is the program design supported by the diagnosis and Theory of Change?
• Results: what are the results of the program in terms of its Theory of Change? 

The first three stages had a relatively conventional evaluation process. The causes of the problem the program addresses, the way it intends to tackle it, as well as the theoretical causal mechanisms of its achievements, were defined in a participatory manner. However, the biggest challenge was found in the results evaluation part. Because the program is so complex, no conventional method proved to be adequate. Therefore, the evaluation team set out to find methodologies that would be better suited to address this situation. They concluded that a suitable alternative for outcome evaluation would be to adapt the outcome harvesting method. 

This qualitative method, that had not yet been documented in programs in Mexico, is an innovative way to evaluate complex interventions. At its core is a participatory strategy in which the evaluators listen to the different voices of those involved in the project, in this case small-scale farmers and other participating actors. These voices allow them to reconstruct the map of the ecosystem that the program seeks to change and thus, to understand its complexity and the perceptions that all stakeholders have about it. That is, it allows us to identify the progress and opportunity areas of the program from the perspectives of its beneficiaries and participants, in order to facilitate their consideration in the decision making process about the future of the program. 

One of the great advantages of the harvest of scopes is that it allows identifying from the different perspectives the changes to which the program contributed, whether they were positive or negative, intentional or unintentional, direct or indirect. There were some surprises in the EPI evaluation, for example: producers regularly identified that the program helped them to become better organized. This allowed them to generate more trust among their community members, which revolved around the business of agricultural production. However, strengthening producer organizations was not a formal component of the intervention, and because it was so highly valued by producers, it was a lesson learned for improving the program. 

The two main lessons learned 

Lessons were learned from this mid-term evaluation process. First, working under a participatory scheme in which the voices of the different people involved in the program are heard allows us to understand the complexity of both the programs design and implementation process and the public problem they are trying to address. 

Second, the use of qualitative evaluation methods allows an in-depth analysis of the causal mechanisms between the implementation of a program and the results it seeks to obtain. This is complementary to quantitative methods that measure the average effect of the intervention. Qualitative analysis generates evidence that allows decisions to be made to improve interventions on the go, and thus achieve their objectives. 

Erick and Karla are part of CLEAR’s evaluation team for Latin America and the Caribbean. Gabriela Pérez Yarahuán, Oliver Peña Habib and our trainer Giovanna Montagner participated in this evaluation.