Evaluating during the pandemic: The experience of an online outcome assessment survey in Brazil

Lire en francais | Leer en español

Autor: Hardi Vieira and Rodrigo Dias/ Brazil Hub-IFAD

1. Background and context:

The restriction of physical contact between people in the current COVID-19 pandemic makes it difficult to carry out activities in the field, especially for the application of surveys that used to be performed face-to-face. In an emergency, the need for creating or inventing new ways to do the same arises. Thus, a new methodology was introduced to conduct the outcome assessment surveys in three projects in Brazil: Viva o Semiárido, Paulo Freire, and Pró-Semiárido, funded by IFAD. 

During the first half of 2020, three assessments were held entirely online to obtain updated and reliable information on the action outcomes carried out in the field by the projects. This with a focus on the effect indicators that compose the Logical Framework. Besides, it also provided the opportunity to collect relevant data regarding the pandemic effects on families’ livelihoods.

• Survey design, methodology and data collection:

The main actors involved in the preparatory activities of the outcome assessments were the IFAD’s M&E specialist, government and M&E professionals, local technicians, and Technical Assistance entities. This team undertook the following steps to conduct the surveys:

• Determining sample size: The sample was designed based on the universe of families who received investments from the projects to improve family farming production capacities, conditions, and technical assistance. The sample considered a minimum number of young and female beneficiaries. 
• The questionnaire: The questionnaire was carefully made to be easily understandable by the beneficiaries and to get simple replies. It has an estimated duration of 30 minutes and comprised 30 simple, direct, and objective questions grouped by different topics to cover all it was expected to assess, including COVID-19 impact. 
• The selection of interviewees: The project’ teams carried out the selection of families to participate in the surveys. The selected families must meet four criteria to receive the survey form:
• be a beneficiary of the community productive agreement; 
• have a smartphone or have access to one within their family; 
• have access to the Internet; and 
• maintain representativeness by gender, age, and location in the territory. 

• Data collection: The Project Management Unit (PMU) shared the survey form link with technical staff based in the field and then they forwarded it to the farmers via WhatsApp. Together with the survey link, the farmers received an information card about the research and an audio file explaining the objectives of this action.
• Field execution: All surveys were carried out in the first half of 2020. The three surveys received 5,587 responses overall in less than two months (considering planning and execution). This execution speed is a substantial gain for an online survey with an average of 100 response questionnaires per day. 

Nevertheless, it is important to note that this type of survey has a limitation: random distortion of statistics–selection biases–given that having a smartphone and internet connection was necessary to be part of the survey. It can be assumed that survey participants may have better financial possibilities and access to information than non-participants. In other words, survey participants do not necessarily represent the characteristic of all beneficiaries within the project’s community. 

• Why is performing online surveys in the rural area a good idea?

The rollout of the survey’s results allowed obtaining important lessons learned for other projects that need fieldwork and could not be performed due to the COVID-19 pandemic. In addition, it is a good practice to keep implementing since it could be hard to reproduce the success rate of response in person, in a rural environment, without investing more resources.

The participatory methodology allowed beneficiaries to play a leading role, with broad and direct participation. Young people, due to their familiarity with technological means, played an important role in conducting the research. Nowadays, Brazilian projects are using online tools together with other initiatives (remote Technical Assistance and Rural Extension (ATER), small trading, etc.), and this successful experience encourages the use of it by beneficiaries and it contributes to reducing isolation in the rural area. 

It is important to highlight the rapid planning, execution and extremely low cost of the studies that were indeed prepared and implemented in a short period. Performing a complete digital survey also facilitates the analysis of the results, considering that the applications used generate databases almost instantly. However, given their limitations, the surveys do not replace the impact assessments. Still, they offer an important source of information in the framework of outcome surveys and for decision making to address, guide/adjust and improve the projects’ actions in the last year of implementation–the case of Brazilian projects.

Here are some lessons learned and suggestions coming from this experience: 

• Reduce the number of open questions to facilitate the process of consolidating responses.
• Consider the application of the questionnaire to families that are not beneficiaries of the project. Perhaps, interview families residing within the community itself to get information from control and treatment groups.
• Increase strictness concerning the proportionality of the groups surveyed. 
• Integrate this practice throughout the implementation as an M&E tool in future projects.
• Use this principle and methodology for other purposes like assessing the quality of services provided and the access of productive projects.