Catholic Relief Services

Catholic Relief Services was founded in 1943 by the Catholic Bishops of the United States to serve World War II survivors in Europe. Since then, we have expanded in size to reach more than 130 million people in more than 100 countries on five continents.
Website:

Evaluation Protocol and Methodology

Closing date

Introduction and Context

Six years after a major outbreak of violence, humanitarian needs remain widespread among crisis-affected communities in the Tanganyika Province of eastern Democratic Republic of the Congo (DRC). Improving security conditions in parts of the province have allowed tens of thousands of households displaced by violence in 2016 and 2017 to return to the communities of origin; however, most returnees struggle to meet their basic needs and re-start productive livelihoods in communities that burned and looted during periods of conflict. For internally displaced persons (IDPs), persisting insecurity has prevented a return to their communities of origin, leaving them dependent on limited livelihood options, support from host communities, and humanitarian assistance as they endure long-term displacement. Periodic outbreaks of violence also continue to drive new displacements in Tanganyika Province. In total, OCHA estimated that 1.2 million individuals across the province needed humanitarian assistance at the outset of 2020, with returnees and IDPs classified as the most vulnerable groups within the province.[1]

Since 2017 the DRIVE (Displaced and Returnee Households Invite Recovery in DR Congo) project implemented by Catholic Relief Services (CRS), with the financial support of USAID’s Bureau for Humanitarian Assistance (BHA) and its forerunners the Office of Foreign Disaster Assistance (OFDA) and Office of Food for Peace (FFP), has provided life-saving multi-sectoral assistance to crisis-affected households in eastern DRC and Kinshasa. Through six initial phases of implementation, the DRIVE project reached more than 97,354 households (estimated 339,004 individuals) with timely needs based NFI, WASH, and food assistance via a rapid response mechanism approach. On March 27, 2021, CRS launched a seventh phase of the project, targeting a total of 96,000 IDPs and returnees (16,000 households) in Tanganyika Province for emergency assistance over 12 months of implementation. In addition to continuing to meet emergency Shelter & Settlement (S&S) NFI, WASH, and food needs, the project increased its livelihood support for returnee communities, targeting 6,000 total households (36,000 individuals), among returnees receiving assistance in other sectors. As of November 2021, the project successfully reached 43,206 individuals through 4 interventions carried out in partnership with three national organizations Caritas Kalemie-Kirungu, Caritas Kongolo, and Caritas Manono. Interventions reached communities in three of Tanganyika’s six territories (See Annex 1 for map of project interventions), with the project providing assistance through voucher fairs and direct cash distributions. At the conclusion of the DRIVE project, a team led by an external evaluator will conduct a final evaluation to determine project achievements and outcomes and generate learning to be applied during future phases of implementation.

Research Questions

The final evaluation will focus on the following research questions:

1) To what extent did the project achieve its objectives?

a) What were the project’s major outcomes? How did they compare to the project’s intermediate results, strategic objectives and goal?

b) To what extent was the project implemented as designed?

c) What effect did the project have on participants’ well-being? What effect did the project have on local markets?

d) What unintended outcomes or effects did the project have as concerns project participants or other project stakeholders (e.g., local markets, host communities)?

e) How did project outcomes vary by subset of the participant population?

2) What factors contributed to project outcomes?

a) To what extent was the project strategy, including targeting strategy, community engagement strategies, choice of modalities, and complementary activities, relevant?

b) To what extent was the project strategy, including targeting strategy, community engagement strategies, choice of modalities, and complementary interventions, appropriate?

c) To what extent was project strategy efficiently executed?

d) To what extent was the project accountable to participants?

e) How did the project communicate with participants?

f) To what extent did the project mainstream protection and gender into design, implementation, and monitoring of activities?

g) How did the project utilize data generated by monitoring, evaluation, accountability, and learning (MEAL) systems?

h) What role did project partners play in the project?

i) To what extent did the project coordinate with other actors?

3) What challenges did the project face during the execution of activities? What did the project do to adapt in the face of these challenges?

4) What lessons learned and best practices were drawn from the project that can be applied to future phases of implementation?

Additionally, the evaluation will include a Strengths, Weaknesses, Opportunities and Threats (SWOT) analysis to inform future programming.

Of note, as per the project’s Monitoring & Evaluation Plan, post-activity monitoring (PDM) data collected following each project intervention will be used to calculate endline values for project indicators. Final evaluation data collected vis-à-vis project indicators will serve primarily to enhance understanding of changes resulting from project activities and the permanence of changes.

Methodology

To generate nuanced understanding of project results and factors contributing to those results, the final evaluation of DRIVE will utilize a mixed-methods approach. The external consultant, working in DRC, will assure overall leadership of the evaluation, supported by one CRS DRC staff member, who will assist the consultant in the day-to-day execution of evaluation activities in Tanganyika Province. CRS can also provide the consultant with an independent Assistant Team Leader, recruited based on their experience conducting similar evaluations in Tanganyika Province with external consultants. CRS has the information for a number of enumerators through its Caritas partners that the consultant can interview, select, and recruit for primary data collection activities from February 11 - February 28. The external evaluator will collect additional data through qualitative interviews directly in the field or via online calls with internal and external stakeholders. In addition to primary data collection, the final evaluation will include a desk review of DRIVE project documents and datasets and of secondary sources pertinent to humanitarian activities in Tanganyika Province.

Quantitative methods

To generate data on outcomes and the effects of project activities, as well as participant perceptions of project relevance and appropriateness, the final evaluation team will conduct a household-level survey. The evaluation team will collect data using tablets and the CommCare electronic data collection application. The external evaluator will develop the evaluation’s quantitative data collection tool based on the evaluation’s research questions and the project’s performance monitoring plan, and CRS staff will review the tool for relevance and appropriateness. After digitization of the tool in CommCare, the evaluator and CRS team lead will test the electronic version of the tool to allow for final adjustments to be made prior to the training of team members on the tool.

Sampling: Quantitative methods

To allow for robust conclusions to be drawn regarding the outcomes of DRIVE’s assistance, the evaluation will employ a stratified two-stage cluster sampling strategy. Participants will be separated into two strata (or groups). Group 1 will be composed of households receiving core emergency assistance (S&S, WASH, and food), while Group 2 will be composed of those core emergency assistance and complementary assistance (S&S, WASH, food, and agriculture). A target sample size will be generated for each group, with a desired confidence level (CL) of 95%[2] and a margin of error (MOE) of 6%. Within each group, a random sample of villages will be selected, with the probability of a given village being selected proportionate to the number of project participants residing within the given village relative to all participants served within the group. In the second stage of sampling, households will be randomly selected for data collection from each village targeted from the project participant database. When reporting on overall project outcomes, appropriate weighting will be used to reflect the relative over-sampling of Group 2 participants in comparison to the proportion they constitute of the total participant population.

In light of project implementation dynamics, two accommodations will be made when drawing the sample of villages covered by the household survey. First, villages where project implementation is still on-going or where PDM is planned but has not yet been completed will not be targeted for the evaluation. This is to ensure that a sufficient period of time has passed between the provision of assistance and the final evaluation and to avoid burdening communities by conducting PDM and final evaluation in the same locations during a relatively short period of time. Secondly, to account for the possibility of villages being inaccessible due to road conditions within a territory, the external evaluator will generate a list of reserve villages (two per territory covered). To account for households not being present at the time of data collection, a list of 15 reserve households will also be generated for each village selected.

Analysis: Quantitative methods

Analysis of quantitative data collected will primarily consist of calculation of frequencies for categorical variables and means, medians, and standard deviations for numeric variables. Where appropriate, results will be disaggregated based on gender and the gender-composition of household leadership. For NFI and food security index measures of interest[3] confidence intervals will also be calculated, and independent t-tests will be used to assess if a statistically significant difference in outcomes exists between groups receiving the core emergency package and those receiving the core emergency package (S&S, WASH, and food) and complementary activities (S&S, WASH, food, and agriculture). Where relevant given the evaluation’s research questions, the external evaluator will analyse quantitative data disaggregating by participant status (returnee, IDP, host family), by intervention, and by other relevant variables identified during data collection (e.g., territory, timing of assistance). The external evaluator will conduct all cleaning of data and generation of statistics using Excel and Stata. The external evaluator will analyse the baseline data and compare the results between the baseline and final evaluation data collection.

Qualitative methods

Qualitative data will be collected through focus group discussions (FGDs) with project participants and vendors and key informant interviews (KIIs) with internal and external project stakeholders. Two data collection team members, supervised by a team leader, will lead focus group discussions. One enumerator will lead each focus group discussion while the other enumerator will serve as note taker. Focus group discussions will be conducted utilizing a discussion guide, with minor adaptations made based on the group composition (e.g., gender, type of assistance received). Focus group discussions will focus on participants’ perceptions of outcomes and their sustainability, and of the relevance and appropriateness of project activities. In addition to conducting FGDs at the community level, the evaluation team will conduct focus group discussions with vendors of core emergency assistance inputs in each territory covered by the project (S&S/WASH NFI vendors, food vendors, etc.), as well as FGDs with agriculture vendors, where agriculture activities were implemented. Focus group discussions with vendors will focus on the efficiency of voucher fair organization and effects of project activities on vendors and on local markets more broadly.

In addition to focus groups, the evaluation will include key informant interviews with key

external and internal project stakeholders, with a tentative target of 21 total KIIs. The external evaluator will conduct all internal stakeholder interviews, which will include interviews with CRS DRC senior management, key members of the CRS DRC Operations Department, and members of the DRIVE project team. External stakeholder interviews will target members of peer organizations, OCHA, relevant emergency clusters, and Caritas staff. All KIIs will be conducted utilizing interview guides developed based on the evaluation’s research questions and tailored to interviewees’ role vis-à-vis project implementation. KIIs with internal project stakeholders will focus on the degree to which the project was implemented in-line with project strategy, challenges faced during project implementation, and best practices developed, and lessons learned during project implementation. KIIs with peer organizations, OCHA and clusters will focus on the relevance and appropriateness of DRIVE project activities given humanitarian context in the project zone and the effectiveness of CRS’s participation in coordination of humanitarian activities. KIIs with Caritas staff will focus on the relevance, appropriateness, and efficiency of the project strategy, with particular emphasis on the nature of collaboration between CRS and Caritas.

Sampling: Qualitative methods

Focus group discussions will be conducted in the same villages selected for quantitative data collection described in the previous section, with two discussions planned per site. Convenience sampling will be utilized to form focus groups, with community leaders (chiefs of villages, nursing staffs or teachers’ other community focal points provided general criteria based on which to organize groups (eg. Relais communautaires,). The evaluation team will strive to organize separate male and female focus group discussions, with efforts made to include participants of varying ages, and to include individuals with disabilities as possible. In some cases, it may be necessary to conduct mixed-gender focus groups due to time constraints, limited availability of individuals of a given gender, or the preferences of participants or community leaders.

With regard to internal and external key informants with peer organizations, OCHA, clusters and working groups, and Caritas, the external evaluator will choose the individuals to be interviewed based on the nature of their collaboration with the DRIVE project.

Analysis: Qualitative methods

Data collected during FGDs will be consolidated by those conducting the discussions and entered into answer matrices by the team members responsible for conducting FGDs at the conclusion of data collection in the field. The external evaluator will then review data and identify and code themes. Qualitative data will be used primarily to add nuance and context to quantitative data collection, in particular as regards factors contributing to project outcome and effects, and to identify challenges faced, lessons learned, and best practices developed through project implementation.

In addition to quantitative methods to measure progress against indicators, CRS will use complementary qualitative methods, including direct observations, focus group discussions, interviews, and participatory methods to triangulate data and ensure that all segments of the beneficiary population can provide feedback and input.

Desk Review

To inform the design of the data collection tools and to assess the degree to which the project was relevant and implemented as designed, the external evaluator will conduct a desk review of project generated documents, including the project proposal, semi-annual and monthly donor reports, PDM reports, field mission reports, needs assessment reports, market assessment reports, accountability reports, and project standard operating procedure documents. The desk review will also include review of project data sets (e.g. participant database, PDM data) and review of assessments and other reports generated by OCHA, FEWS NET, the Integrated Food Security Phase Classification, and peer organizations describing the humanitarian context in the project target zone.

Data Collection Team Training

Prior to the beginning of data collection, enumerators will participate in a two-day training, led by the external evaluator, evaluation team leader, and a DRIVE project MEAL Officer. The training will cover the evaluation methodology, best practices for data collection, use of data collection tools, and the CRS Code of Conduct and Ethics.

Limitations

As designed, the evaluation will have three major limitations.

  • Firstly, due to practical constraints on time and resources, the evaluation will not utilize simple random sampling, whereby respondents are chosen at random and every individual in the sample frame has an equal chance of being selected to participate in the quantitative survey. Cluster sampling, however, is a commonly used option in contexts where logistics and time do not allow for simple random sampling of a geographically diverse population, as is the case with the DRIVE 6 project, which provided assistance to households living in more than 330 villages across six territories.[4]
  • The security situation in Tanganyika remains dynamic, with the possibility that a territory or part of a territory targeted for data collection will not be accessible. If the evaluation team is not able to access a particular territory due to insecurity, the team, depending on location will proceed to a pre-selected back-up zone, selected via the same procedure utilized to generate the initial list of target locations.
  • Due to restrictions on road travel, humanitarian actors primarily travel between territories within Tanganyika Province via UNHAS and ECHO flights. Efforts will be made to book flights as early as possible, but it is possible that a sufficient number of tickets may not be available or that flight services will not be available due to mechanical problems or a change in service schedule. If the evaluation team is not able to access a particular territory due to flight-related constraints, the team, depending on location will proceed to a pre-selected back-up zone, selected via the same procedure utilized to generate the initial list of target locations.

Ethical principles

The evaluation will be carried out in accordance with good practices for ensuring ethical data collection and accountability to participant communities.

All data collection team members will be trained on and required to sign the CRS Code of Conduct and Ethics, and evaluation team leaders will include an appropriate explanation of the Code of Conduct and Ethics when briefing participant communities on the evaluation’s objectives and activities. Additionally, the data team training will cover good practices for data collection, with an emphasis on respecting cultural norms and collecting data objectively with a non-judgmental approach.

Prior to the start of each interview and FGD, evaluation team members will explain the following, using a prompt included in data collection tools:

  • Objective of the evaluation and data to be collected;
  • Purely voluntary nature of participation, which will not have any bearing on an individual’s or household’s eligibility to receive humanitarian assistance of any sort in the future;
  • Length of the interview or discussion;
  • How the data collected will be used;
  • How long the data collected will be kept;
  • How participant confidentiality will be maintained.

Data team members will ask for oral consent before beginning all interviews and discussions, and they explain that any sharing of data or research findings from the evaluation will be done in a way that does not allow participating individuals to be identified. For household level interviews, consent will be recorded via a Yes/No question at the beginning of the digital questionnaire. For FGDs, after received oral consent from all participants, those conducting the FGD will note receipt of consent on the note-taking document for the respective discussion or interview.

To ensure that participants are able to ask questions and share any feedback regarding the evaluation, the data collection team will remind participants of the project’s existing feedback and response pathways, and explain that questions and feedbacks can also be addressed to data collection team leader.

All data collected during the final evaluation will be de-identified prior to analysis, and no presentation of data or research findings will be done in such a way that allows for any individual participant to be identified.

No children shall be included in data collection for the evaluation.

COVID-19 risk mitigation

To mitigate against the risk of the evaluation contributing to the transmission of COVID-19, the evaluation team will be required to adhere to accepted COVID-19 risk measures before, during and after data collection activities, and participants will be briefed on measures they will be required to respect during data collection.

To ensure proper, consistent application of COVID-19 risk mitigation measures, the evaluation team training will include a session devoted to basic information on the risk posed by COVID-19, transmission pathways, and risk-mitigation measures to be applied within the context of the evaluation. All information will be based on World Health Organization (WHO), Inter-Agency Standing Committee (IASC), CRS agency, and DRC Health Ministry recommendations.

Before, during and after data collection, enumerators will be required to maintain social distancing, wear cloth or other non-medical masks, practice good sneeze/cough etiquette, refrain from touching their eyes and nose, avoid touching common surfaces, and engage in frequent handwashing or disinfection. While traveling to the field, evaluation team members will be required to use approved seating arrangements in vehicles. During data collection, evaluation team members will conduct all interviews and FGDs outside or in well-ventilated areas, with appropriate social distancing between participants. As part of the community entry process, the evaluation team will brief community leaders on the COVID-19 risk-mitigation measures that will be practiced and the reasons for them, and request community leaders’ support in avoiding crowding. Evaluation team members will be assigned a device for the entirety of the evaluation period and will be required to regularly wipe down their devices and disinfect with a disinfectant wipe during data collection and prior to transferring devices to the individual responsible for their management at the end of each day. CRS will furnish an appropriate supply of masks, hand sanitizer, and disinfectant wipes to all evaluation team members for the duration of the evaluation.

The external evaluator will be responsible for ensuring that evaluation team members have been briefed on COVID-19 risk-mitigation measures, while the team leader will be responsible for monitoring adherence to recommended practices during data collection activities. Adherence to measures and any challenges faced in applying measures will be discussed during daily de-briefings.

CRS’ final evaluation consultants should be vaccinated against COVID-19 to travel to DRIVE intervention sites.

How to Apply

All bids should be sent to: Kinshasa_bids@crs.org