Impact Evaluation in
Post-conflict Environments
A Critical Appraisal of Randomised
Controlled Trial (RCT)
Bachelor Thesis
Author: Rania Walid Supervisor: Heiko Fritz Examiner: Christopher High Term:Spring2021
Abstract
Impact evaluations in development interventions has been growing in recent years. The increasing demand for evidence-based outcomes has led to a debate of what methodology is best to evaluate the impact of development interventions. Accordingly, Randomised Controlled Trial (RCT) has been labeled as a gold standard for impact evaluations. The RCT method functions in a unique way, as it removes the selection bias and ensure high validity of a study. The aim of this research study is to critically assess the RCT as an alternative approach for impact assessment in relation to post-conflict countries; whether this claim holds in a conflict-affected environment or that the context-specific factors of post-conflict countries challenge the implementation of an RCT. This study implements mixed method approach by using simple descriptive statistics and semi-structured interview to answer the research questions.
The findings of this study indicate that context-specific factors of post-conflict environments pose challenges on the implementation of an RCT. As a result, these challenges threaten the quality of the RCT method which lies in reliability, internal validity and external validity. The findings also indicate that feasibility of RCT which lies in ethics, logistics and security, cannot be addressed individually, as the feasibility has a direct impact on the quality of the RCT method.
Key words
Impact evaluation, Randomised Control Trial (RCT), post-conflict settings, challenges, rigorous implementation.
Acknowledgments
I would like to express my appreciation and deep gratitude to my supervisor Heiko Fritz for guiding me throughout the writing process. With his continued encouragement and patience, i have been able to gain learning experiences.
Table of contents
1. Introduction ... 1
1.1 Relevance ... 2
1.2 Research problem ... 3
1.3 Objective and research questions ... 3
1.4 Disposition ... 4
2. Literature Review ... 4
2.1. Credibility of RCT ... 4
2.2 Challenges confronting RCT’s method ... 5
2.3 Challenges in conflict-affected environments. ... 6
2.3.1. Feasibility ... 7 2.3.2. Contextual issues ... 8 2.4. Proof of concept ... 9 3. Analytical framework ... 10 3.1. Reliability ... 10 3.2. Internal validity ... 12 3.3. External validity ... 13
3.4. Feasibility; Ethics, logistics and security. ... 15
3.5. Conclusion of the analytical framework ... 16
4. Research methodology ... 18
4.1 Mixed method ... 19
4.2 Quantitative method ... 19
4.2.1. Sampling ... 19
4.2.2 the research instrument and variables ... 19
4.2.3. Data collection and data analysis ... 20
4.3. Qualitative method ... 21
4.3.1. Sampling and the development of interview guide ... 21
4.3.2. interview process ... 21
4.3.3. Data analysis ... 22
4.4 Ethical Consideration ... 22
4.5 Limitation and delimitation ... 23
5 Findings ... 23
5.2 Qualitative findings: results from semi-structured interviews ... 28
5.2.1 Quality of RCT ... 28
Survey-related individual’s/general responses that concerns internal validity. 29 5.2.2. Feasibility of RCT ... 30
Survey-related individuals’ responses that concerns ethics ... 31
Survey-related individuals’ responses that concerns logistics ... 32
5.2.3 Other challenges of RCTs ... 33
6. Analysis ... 34
6.1 Quality and feasibility of RCT; ... 34
6.1.1 Reliability ... 34
6.1.2 Internal validity ... 35
6.1.3 External validity ... 37
6.1.4 Feasibility; ethics, logistics and security ... 38
7. Conclusion ... 40
7.1. The main findings of the survey and semi-structured interview ... 40
7.2. Discussion ... 41
1. Introduction
According to White and Raitzer (2017) ‘[i]mpact evaluations are empirical studies that quantify the causal link between interventions and outcomes of interest.’ In long and probably inconclusive debates about the appropriate methodology of an impact evaluation (White, 2009), the method of a randomized controlled trial (RCT) has often been labeled the gold standard of impact evaluation. Accordingly RCTs have been used widely for impact evaluations of projects in international development cooperation. Abhijit Banerjee, Esther Duflo and Michael Kremer were awarded the Nobel prize in 2019 for using RCTs as field experimental to alleviating poverty.
Field experiments in social science gained popularity when they were associated with welfare reform programs in 1975 (Gueron, 2017 pp. 29). The heated debates about social reform led to a question of whether those programs would be effective or non-effective. Thus, obtaining reliable and evidence-based outcomes has become essential as it identifies what works and what does not (ibid pp. 29-31). As a result, Randomised controlled trial (RCT) was the alternative approach that has these qualities because it functions in a way that could remove systematic bias. Besides this, it was also vital that the practitioners of RCT illustrate the feasibility of RCT; that is, ethical consideration, legitimacy and its capacity to be implemented in different settings (ibid). These dimensions were not the only ones - RCT was associated with informing policy-makers and making governments effective. To this end, evidence-based outcomes has to be proven, in which allowed the implementers to make the RCT method identical and rigour in evaluating the programs’ impact in order to inform findings in such way that increases the potential to be used in different settings (ibid). Consequently, the use of RCT has increased extensively to include different and advanced forms of programs such as multidimensional system wide reforms. This expansion on the other hand, has allowed the feasibility to be tested and hence performed in different settings (ibid).
be straightforward; one is exposed to the program and the another group is not, but measuring individual outcomes will not be the same over time due to other factors. Here the factors that contribute to the outcomes at specific point of time, will change in a different point of time.Thus, measuring the average impact creates reliable and credible outcomes of programs (ibid).
Currently, RCT is used to evaluate impact of interventions in under-developed countries and fragile contexts. Evaluating the effectiveness of development interventions is requested by donors in order to assess what works and under which circumstances that this can be possible (Wolfe, forthcoming).
Thus far, 58 published studies of RCT were implemented in conflict/post-conflict countries. These studies include peace-building programs where RCTs has been proven its attribution to the programs’ objectives (Wolfe, forthcoming). The aim of peace-building projects is to bring people together from different backgrounds, which can be fulfilled through the RCT method; RCT removes selection bias and ensures that participants are not only those who show flexibility and willingness to engage in peace-building programs (ibid). On the other hand, implementing an RCT to evaluate the impact of development interventions in conflict-affected environment may have to confront major challenges. For instance, feasibility; ethical, logistical and security-related challenges. Also, there are other major challenges that are associated with the quality of RCT; internal, external validity and reliability. These challenges may cause interruption of the program such in the examples of studies conducted in Egypt (Megally, 2019), in Nepal (Wolfe, forthcoming), and Yemen (McKenzie et al,2017). These specific-related challenges will be highlighted and addressed throughout the thesis.
1.1 Relevance
The shift from focusing on observing outcomes to what impact an intervention can make, is a subject that has been extensively growing in recent years. Yet the former has not proved the contribution that the latter has made (White, 2010). Furthermore, Donor’s organisations requisite rigorous evaluation of what works and what does not work of a specific intervention.
post-conflict settings pose specific challenges on the implementation of RCT and thereby impede a rigorous evaluation.
Furthermore, this critical appraisal of RCT method in relation to conflict-affected has relevance to the overall domain and the willingness of policy-makers to scale-up development programs (White, 2013), as the rigorous implementation of an RCT is directly connected with the context-specific challenges of conflict-affected settings.
1.2 Research problem
Conducting an RCT to evaluate projects is challenging, and even more challenging in conflict-affected countries. The reason behind this issue is that contextual reasons such as logistics and security can be an obstacle ahead of the implementing’s procedure of an RCT. Nevertheless, a literature review does not confirm this fact as there are around 58 completed RCT studies which were implemented in conflict/post-conflict countries. On the other hand, there are examples that challenge the implementation of an RCT. These challenges are security issues which can lead to diminishing the experimental groups. Another example can be changes in a political situation in a country which can cause changing behaviour among participants. In both cases, the causal inference is not clearly identified: whether the outcomes of an intervention are caused by the program or are affected of external factors. Thus, the question of whether RCT method is suitable for impact assessment in conflict-affected settings, requires a systematic research that can identify and conclude these challenges in one framework.
1.3 Objective and research questions
This study will take a critical view on the method of Randomised Controlled Trial (RCT) and discuss specific challenges that are related to the implementation of RCT in post-conflict settings. The aim is to achieve a systematic conclusion of the RCT’s method in relation to the context-specific factors of conflict-affected environments in order to encourage interventions by means of RCT method. While also to reconsider the design by including a conflict dimension. This project will be carried out through mixed methods. This is a preparatory work for further studies.
-1 What are the challenges of implementing randomised controlled trial (RCT) in a post-conflict/conflict setting?
-2 How have these challenges affected the implementation of an RCT?
1.4 Disposition
This study is organised as follows. Chapter 2 is the literature review. Chapter 3 is the analytical framework and then the methodology in chapter 4. Chapter 5 is the findings of both the quantitative and qualitative methods while the following section is the analysis of the findings which is chapter 6. Chapter 7 is the conclusion.
2. Literature Review
In this section, a scrutinised literature review of RCT as alternative approach for impact assessment will be presented. The section will exemplify - based on cases from the 58 RCT studies - challenges that confront RCT in relation to conflict-affected settings. Moreover, the literature review will combine literatures of context-specific factors of post-conflict settings with literature of RCT method. This synthesis will introduce the research gap of this study which lies in the relation between these context-specific factors and a rigorous implementation of an RCT. In conjunction with this, reviewing whether the claim made by RCT’s proponents that the method is an alternative approach for impact assessment also holds in conflict-affected contexts, is essential for the gap of this study.
2.1. Credibility of RCT
Here, to estimate a real impact of a given program on individuals, it is important to understand the underlying factors that make a program work (ibid).
For example, considering a scenario where clean water distributed to households and the households were told to use clean water on a daily basis, and also to avoid using contaminated water (Gibson & Sautmann 2020). The fact that the households’ behaviour has changed since they started using clean water, is what RCT bases its evaluation on (ibid). The averages of the behaviour change in treatment groups, will therefore determine the positive effect of a given program (ibid) and also will identify the mechanism of how a program has worked. Here, an impact evaluation is the difference that a program can make (before verses after) (White 2013). This is what policy-makers wish to obtain in order to scale-up projects (Gueron, 2017, p. 55). While on the other hand, the importance of generalising outcomes and scaling up interventions play less role when results concern a specific interests area in a specific population (Peters et al, p. 35).
( 1 )
2.2 Challenges confronting RCT’s method
RCTs that are conducted for assessing policy, are closely interacting with different components such as environmental, political and economical elements in a given country (Gibson & Sautmann 2020). Thus, there is a direct relation between an outcome of the RCT and external factors, which can pose challenges on the project during the course of the implementation (Ibid). For instance, in developing countries, a non-governmental organisation is likely to run the same type of a program during the course of the RCT’s intervention, where the NGO provides a treatment to a control group (Ibid). This as a result, can lead to biased outcomes of the RCT’s study. These types of challenges should be taken into account when planning an impact evaluation, as these challenges threaten internal and external validity (ibid). Consequently, biased results undermine the power of a study which they are high reliability, internal and the external validity of a given program (ibid).
researchers and partner’s organisations. Whereas scaled programs are usually implemented by governments that may lack resources (ibid 34-35).
Many attempts were undertaken to highlight challenges impeding the external validity. However, one major issue has not been given attention and this is the individual’s consent (Athey,&,Imbens 2017. p 79 ). I.e, there is noting that can tell that the target population would give consent to take part in an experiment. So that the results from a trial population (where the consent was obtained) can be generalised to the target population (ibid). However, this issue can be found across non-experimental studies and not only in a Randomised evaluation (ibid).
Banjerjee et al, (2016) suggested different approach for external validity such as multi-sites and already existed impact evaluations. For instance, the Graduate approaches that improves the livelihood of people who are trapped in extreme poverty. This initiative was implemented by many countries in the world after it was scaled up in Bangladesh by world organisation BRAC. The program is replicated in different sites. Each site has a researcher’s team in which they were in contact with each other to make sure that the outcome is compatible to the original program (Banjerjee et al, 2016). The last approach is structured speculation, which allows researchers to anticipate from the first sites what could be like in other samples at different sites. Structured speculation approach has advantages where researchers know specifically which sites these program fits, and push towards planning interventions that have high probability of generalising the outcomes (ibid).
2.3 Challenges in conflict-affected environments.
Fundamentally, internal validity is a basis for external validity (Athey &,Imbens 2017. p 79). That is, if a result of a study is because of a treatment and not by confounding factors (confounding factors: alternative explanation of the result rather than the program’s input; e.g systematic differences between units), the results can be generalised to new settings (Xin, et al, 2015). To be precise, internal validity deals with causal inferences whereas external validity generalise these causal inferences. Thus, the value of causal studies lies in high internal validity which is also a precondition for the external validity (Athey & Imbens 2017. p 79).
external factors in conflict-affected contexts can pose even more challenges than a non-conflict environments. These challenges can be e.g a deteriorated political situation that affect the quality of RCT and subsequently undermine the outcomes of the program implemented, see the case of Yemen (McKenzie et al,2017). Likewise, the study conducted in Egypt has a quite high attrition rate due to a terrorist attack that has taken place in the same spot where the program has been placed, which has undermined the internal validity (Megally 2019). Similarly, spillover effect is another dimension that threaten the internal validity. Spillover effect is interaction between units who receives the treatment and the ones who is not exposed to the treatment, which affect the outcomes of a program (Athey,& Imbens, 2017, p. 131). The spillover effect takes different forms. One form of it will be exemplified here in relation to a conditional cash transfer (CCT) that has been implemented in the Philippine to reduce the conflict effect. The CCT program has resulted in reducing conflict in treatment villages but increased in the neighbouring villages. This spillover of conflict to neighbouring villages is because of the government’s troops that has been placed in the treated villages which led insurgents to move to neighbouring areas (Crost et al 2015). The security implication in conflict-affected setting is not only in terms of logistical challenges that affect the implementation of an RCT but also in terms of ethical consideration which can put people and staff at risk (Wolfe, forthcoming).
The context-specific factors threaten the quality of RCT. For example in Nepal, a peace agreement was signed between the warranting parties during the course of the implementation of a youth peace-building program (Wolfe, forthcoming). The peace agreement has changed the situation which has blurred the validity of the study; it was not clear anymore if the changes seen in people’s behaviour were due to the program or the peace agreement. I.e participants’s engagement in community services were noticeable after the peace agreement and this was not clear if this was because of the program or the peace agreement (ibid). For that reason, program was discontinued because of the fact that the counterfactual’s value has not been identified (ibid). 2.3.1. Feasibility
groups. The percentage’s number of attrited participants was 19,76% in treatment group and reached 80,23% in control group (Megally, 2019). This imbalance between the two groups affects how precise the average of the treatment effect, which relies on the balance between the treated and control groups. So one cannot claim the truth of a treatment effect without this balance (Deaton & Cartwright 2018).
Furthermore, feasibility challenges also involve issues related to the nature of post-conflict settings such as a non-functional market and infrastructure that can impede the implementation of a long-term project (Bozzoli et al 2013). For instance, an agriculture program for reintegration of ex-combatants is implemented in Liberia which shows a positive impact through an indicator of participant’s willingness to learn. However, the income returns was low because of market constraints (Blattman &Annan 2011).
Deaton and Cartwright (2018) indicated that there are other factors in the actual experiments that influence the precision of the estimated average treatment effect (ATE). ATE is claimed to be close to the truth and also possibly to hold in a new setting or in general. However, contextual factors in conflict-affected setting which is discussed below (see 2.2.5), can affect the precision of the ATE estimator and a rigorous implementation of an RCT.
2.3.2. Contextual issues
Contextual issues are the factors that distinguish conflict-affected settings and it is boiled down to security and displacement (Bozzoli et al 2013). These issues can pose challenges for the implementation of an RCT. The importance of considering these conflict dimensions when planning an RCT study is expressed by Bozzoli et al (2013).
Woolcock (2013) puts emphasis on the importance of distinguishing between different types and levels of complexity, as this will help to predict the outcome of an RCT. That is, the likely of generalising outcomes can be considered based on an identified framework. Nevertheless, there is a lack of clear standardised measure of where conflict-affected countries stand in terms of the level of violence’s intensity (Uexkull, 2017).
still in use which can be misleading and underestimating the affected-people in a conflict zone (Kaldor 2012, pp. 208-210). This as a result, can threaten the validity of an RCT study (Bozzoli et al, 2013).
Context-Specific issues are important to consider when designing an RCT (Bozzoli et al, 2013). Millan and Macours (2017) discussed the importance of tracking participants in which will increase the validity of a study. The characteristics of under-developed environment lies in a high mobility rate thus, the necessity of employing tools in order to track participants has always been of great importance. Whereas missing participants would result in low internal validity of a study. Here attrition may lead to sample imbalanced as it discussed elsewhere in the paper (see the discussion in 2.2.4). However, RCT works without prior knowledge which this per se - can be an obstacle to produce good results and contribute to build a scientific knowledge (Deaton & Cartwright 2018).
2.4. Proof of concept
Opportunity Cost. The former is a concept that refers to the support of people to a government by giving information about the insurgents as this only happens if civilians are economically better off. While the latter refers to the impact of aid on individuals that leads to less engagement in violence’s insurgents. As a result, it creates peaceful market investment where engaging in war becomes costly and less likely to happen (Beath et al, 2011). Through these two assumptions, one can estimate the average impact of the cash transfer program (ibid). Therefore, cash transfer program was scaled up in the Philippine. But on the other hand, the implementation of cash transfer in the Philippine resulted in conflict spillover effects in the other villages that were near treatment villages (Crost et al 2015), see section 2.2.3.
3. Analytical framework
This developed analytical framework shapes the academic rigour for an impact assessment in which the research questions will be investigated. The academic rigour lies in reliability, internal and external validity and feasibility. Further, the feasibility includes ethical, logistics and security. These criterion also are the basis for the research methodology.
3.1. Reliability
The reliability in research is about consistency, i.e a measurement that is used in one setting, would hold in an another setting (Bryman, 2016, p. 41).
The reliability in RCT lies in the ability of the method to produce accurate estimate of an average treatment effect (ATE) (Deaton and Cartwright 2018), as the unbiasedness in a study sample is the accuracy on the average between the estimate and the true value of a treatment effect.
The uniqueness in ATE is that it needs minimal assumptions which is not the case with other statistical methods and observational studies (Deaton and Cartwright, 2018). RCT designed in a way that can function without the need for prior knowledge, so that the average treatment effect can be concluded by comparing a treated group to a control group (ibid). The control group is a counterfactual value that through it, the impact of a program is evaluated (ibid). This evaluation can be made by looking at the difference in means ( the average of outcomes) between the treated and the counterfactual comparison group (control group) (ibid).
so this as a result will pose constraints on the credibility of the value obtained once it is transferred to a new environment (Duflo et al 2006). White (2013) emphasises that the randomisation within the eligible population will produce accurate estimate of treatment effect and hence will avoid biasness caused by systematic differences between units. But if units are a mix of rural households and town households, there will be a systematic differences between participants. Thus, the sample size should be expanded in order to guarantee an accurate estimate of a project impact (White 2013).
On the other hand, precision of ATE lies in the balance of other factors between the treated group and control group of a trial sample (Deaton & Cartwright, 2018). These two groups should be identical and similar on average to the average of the true treatment effect in the treated group. If this precision is fulfilled, the truth of the trial can be claimed (Deaton & Cartwright, 2018). The argument that contradicts the previous statement is that the balance of the other factors between the two groups cannot be anticipated, and it is also not possible to know how much these factors will affect the impact of the treatment as unobservables can influence the treatment effect (ibid). These unobservables are not measurable such as motivation to take part in the program, which can differ across individuals (Bozzoli et al 2013). Deaton and Cartwright (2016) argued that there is not a conclusive way to ensure this balance between groups, as randomisation does not have the ability to find these difference beyond the observable variables, including the baseline data.
Furthermore, obtaining precision and unbiasedness of the ATE in conflict-affected environments can be a challenge, e.g displacement can cause high attrition that may lead to imbalance between the two groups. Attrited participants can be similar to baseline characteristics in both treatment and control groups in which was the case for the program implemented in Yemen (McKenzie et al,2017). The attrition for this intervention was high but attrited participants were similar to baseline characteristics between the treatment and control groups in which this balance can maintain high reliability, internal and external validity of the study (ibid). However, there are heterogeneity in the sample of applicants, and also the sample of the second part of the program was canceled. All these reasons have minimised the statistical power of the program so that the treatment impact could not be tested at different levels (McKenzie et al,2017). Consequently, this imbalance in the sample of applicants (besides attrited participants) can threaten the reliability of the study and the ATE measure to hold in another setting (ibid).
3.2. Internal validity
Internal validity defined as ”a study to have internal validity if the observed covariance between a treatment and an outcome reflects a causal relationship in which the variable were manipulated,” (Shadish et al 2002. as cited in Athey & Imbens, 2017, p. 79). That is, treatment’s variable can be used in different ways in order to test prior knowledge or theories Banerjee et al, 2016). This as a result, creates facts about interventions that have not been implemented before. While on the other hand, Deaton and Cartwright (2018) argues about the absence of structure basis in RCT (e.g theory or parametric) as its weaknesses is more pronounced when the results are transferred to a different context. This free-theory nature in RCT is a strength for the internal validity and to estimate ATE (Deaton and Cartwright, 2018), in which allows the manipulation of a treatment variable in a variety of ways in relation to an outcome. I.e to test different mechanisms that will lead to the outcome (casual relation between a treatment and an outcome) (Banerjee et al, 2016).
that exist between the two groups (ibid). From that sense, the program is internally valid.
The ability of randomisation to solve the selection bias is mostly absent in other methods, where outcomes can be due to other factors rather than the program’s treatment (Megally, 2019). In contrast, some researchers deny the importance of internal validity, as they claim that results that have high internal validity, are not necessarily applicable in other contexts (Deaton & Cartwright, 2018). This can be due to that consent form may not be obtained in the other settings. Consent form issue also exist across non-experimental and not only in Randomised evaluation (Athey & Imbens, 2017, p. 79). Internal validity is important for causal studies. However, the pitfalls encounter the internal validity mentioned earlier, see the examples (2.2.3), affect the precision of the causal inferences (Glennerster, 2017, p. 194). The pitfalls can be for example attrition which can caused by displacement of subjects; participants are not reachable for a follow-up endline survey (post-programme survey). Also, it may be a challenge to track people due to security issues (Megally, 2019). Consequently, attrition biases the result of the program if this has caused imbalanced between treatment and control groups in respect to baseline characteristics (McKenzie et al,2017). Wolfe, (forthcoming) indicates to the importance of oversampling in the case that will be a high attrition rate (e.g IDP sample is likely to have attrited participants).
Another example is a non-compliance where subjects assign to a program but not interested to continue, which results in a selection bias (Wolfe, (forthcoming). In conflict-affected areas, it is important to ensure that people from different communities are engaged in peace-building programs and not only those who interested in the program. So that it yields unbiased outcomes which result in a high internal validity (ibid). Thus, selection-bias can be avoided through expanding a program’s aim to include other dimensions that get people interested ( e.g vocational training and sports leagues) (Wolfe, forthcoming). The identification of the causal effect in conflict-affected environments will contribute in establishing generalisable approaches that sustaining peace (ibid).
3.3. External validity
In order to obtain high external validity in a study, the study population should be randomly selected and also within this population the treated and control group are also randomly allocated. However, randomising the study population (finite-population) is rarely done because it will introduce constrains on the evaluation's procedure of a treatment impact. This as a result, will weaken the internal validity in a study (Duflo et al, 2006). Duflo et al (2006) gives solutions to external validity’s concern which concludes in testing this specific program in different settings to see if that would produce similar outcomes. The testing is combined with a theoretical framework. I.e it is not reasonable to test all parameters within a study to see if they would hold in all settings. But theories would set a boundary to the extent that results could be replicated and also under which condition do they work (Duflo et al, 2006). Conditional Cash Transfer (CCT) and a Deworming intervention are two examples of interventions which have been replicated several times. For the latter it involved different teams and organisations (Duflo et al, 2006). The results of these interventions were similar to the original intervention. Nevertheless, there was a debate that medical interventions are possibly to be replicated than other types of interventions that involve behavioural change (ibid).
On the other hand, Deaton and Cartwrigh (2018) argue that generalising results that holds in one population relies on previous knowledge which is a basis for constructing scientific knowledge. However, having previous knowledge is contradicting RCT; RCT functions with minimal assumptions. Thus, RCT cannot alone contribute to the accumulation of scientific knowledge and it can only do that once it is combined with a theoretical framework (ibid). Deaton and Cartwrigh,further argue that reasons why CCT programs worked in different places in the world, is because of ’’structural reasons’’. In other words, humans are inclined to be better off as they would not deny the well being of their children. All these reasons stand as explanatory power to why programs such CCT worked broadly. Here, the claim that the proper design of RCT contributed to the high external validity of the CCT program, is not valid (Deaton and Cartwrigh 2018). Nevertheless, Banjerjee et al (2016), argue that using RCT for impact assessment means that researchers have control over where and what study sample is going to be studied. Thus, this allows researchers to know how the treatment effect will differ across different contexts. Furthermore, there are different approaches to evaluate external validity, such as using already existed evaluations in order to make assumptions of the likely impact of a program.
generalising outcomes require a framework and key elements that would tell in an accurate way how results that generated in one place, would be as similar in another place. Such key facts are the implementation capability which may be low in fragile contexts due to external factors, e.g, logistics (Woolcock, 2013).
Wolfe (forthcoming) addressed the external validity concerns in relation to peace-building programs, where attrition and non-compliance can largely affect the generalisability of outcomes. While randomising within the interested group would minimise these issues but instead, it would undermine the external validity. Wolfe (ibid) prioritised generalisability when the intervention is evidence-based, despite the risk of attrition and non-compliance involved. Woolcock (2013) pointed out these factors that influence the desired outcomes of complex interventions, as he described such contexts with High Causal Density. High Causal Density is individuals’ actions and external factors that affect the outcomes. Thus, proof of concept in relation to these interventions in such contexts has an uncertain position without strategies that comprehend the influential factors. I.e strategies will assess the nature of these contexts and thereby to provide suggestions for a rigorous implementation (Woolcock, 2013).
3.4. Feasibility; Ethics, logistics and security.
Ethics in research is discussed as a challenge. However, ethical consideration is associated with research quality; a good research design would be more ethical than a low quality research design (Feeney et al, 2020).
That is, good research design increases the probability of participants who benefit from a treatment. Moreover, participants who have been approached with respectful manner and compensated for the time they spent answering the survey questions - are likely to give right answers. In contrast, participants who feel in opposite way, are likely to give false or no answers which will affect the quality of a research and lead to higher attrition rate and missing data (ibid).
group position is important for the identification program’s value (Rosen et al 2006).
Still, ethics in research might confront challenges in a conflict-affected environment. These environments require understanding of certain issues related to this type of context such as traumatic experiences and economic hardships caused by conflict and destruction. Here the people in these environments are deprived of basic needs and randomisation in these contexts may entail an ethical dilemma (Wolfe, forthcoming). According to Wolfe (Forthcoming), funding is often a problem; researchers are not able to reach all people that need a program. Thus, randomising within the category of groups who mostly need the program - is considered an effective approach of using limited funding in which participants are selected systematically. Simultaneously, this creates evidence-based outcomes of the program effectiveness.
The debate of whether randomisation can reach those who need the program - is a question mostly attached to humanitarian situations, thus, the method can be adjusted accordingly (Wolfe, Forthcoming). For instance, communities who get the treatment must be equal in their needs to those eligible for the program. Another way is to create waitlist or allocate quotas of treatment to eligible participants in different phases of the program. These extra participants may or may not be included in the analysis (Wolfe, forthcoming). Researchers have to include these aspects when designing an RCT to be implemented in conflict-affected settings, as the institution review board (IRB) does not cover all aspects of ethical challenges (ibid).
In addition to the feasibility challenges in relation to ethics, logistics is another challenge of feasibility. For example, security poses logistical challenges that impede the implementation of RCT either in reaching the target sample of the experiment or in keeping the regular access to a sample and delivering an intervention (Wolfe, forthcoming). Further, changing conditions such as the escalation of violence, can put further logistical constraints on the implementation of RCT (ibid).
3.5. Conclusion of the analytical framework
Reliability Internal validity
The concerns of reliability in RCT lies in the estimate of ATE. Conflict-affected settings pose challenges that affect the estimate of the treatment impact. Further, unobservables may not correlate with observable variables which consequently overestimates or underestimates the treatment impact of a given program.
Security issues in conflict-affected setting can lead to high attrition and imbalance between a treatment and a control group which affect the precision of causal inferences. Likewise peace-building programs that aim to integrate people and solve the root of the conflict, can result in non-compliance and high attrition
which subsequently will
External validity Feasibility
4. Research methodology
Given the nature of the research questions and the type of this study as it is carried out as a desk study, the abductive reasoning is employed. In the abductive approach, the context studied is understood on a theory basis which is defined before collecting the data (Bryman, 2016. p, 394). Subsequently, it Conflict-affected settings characterise by high
Causal Density and external influences. This dynamic nature of conflict settings affect the rigorous implementation of an RCT. For example, implementation capability such as logistics, can be low. Thus, generalising outcomes should be based on key elements and a framework. Also, there is an emphasis on referencing to the context when implementing an evidence-base interventions, as this will help to generalise the outcomes.
Ethics reflect the quality of research while security and logistics affect the
should follow a scientific explanation that goes hand in hand with participants’ point of view (ibid).
4.1 Mixed method
This research project implements a methods of inquiry using simple methods of descriptive statistics.This yields genuine research findings as well as the interview guide used to substantiate the findings by means of semi-structured qualitative interview. This chapter introduces each method of the mixed methods design separately.
4.2 Quantitative method
4.2.1. SamplingThe sampling process began by defining a period between 1995-2019 and then finding publications of Randomised controlled trial that was implemented in conflict/post-conflict settings within the aforementioned defined period. The Uppsala Data conflict (UCDP) was used to search for countries that affected by war where 79 countries were identified as conflict-affected countries. These countries were based on intra-state conflict’s definition. In sequence, each of these 79 countries was searched for using several databases such as Google Scholar, Abdul latif jameel poverty action lab (J-PAL), Alliance for Peacebuilding – Peacebuilding evaluation, World Bank library and International Initiative for Impact Evaluation (3ie). In addition, when searching for each country, keywords were used next to the name of the country, for example ( Liberia+Randomised controlled trial RCT). These publications were reviewed for the sampling process where all authors of each publication were included in the sampling - whose are the respondents of the survey.
It was important for the purpose of the quantitative survey to find enough publications of RCT that based in conflict-affected environments. Thus, this systematic searching has allowed to find many publications in which one or several RCT was implemented in each of those conflict-affected environments.
4.2.2 the research instrument and variables
survey. The survey was divided into 4 sections; section (1) started with questions concerning general information about the intervention and researchers’ experiences in the field of RCT. In section 2 and 3, the questionnaire got more specific in relation to the challenges that related to each variable, which centred around the quality and feasibility of RCT, i.e reliability, internal validity, external validity, ethics, logistics and security . Although several questions aimed to tackle each variable, but also all variables were concluded in the last question in section (4). ( see question 1.4.1 in the appendix 1). The survey consists of 2 short answers,1 checkboxes, 6 multiple choice and 3 multiple choice gride. All in total were 12 questions, including the short answer question that related to the followup semi-structured interview; researchers were asked voluntarily to put their email-addresses if they wished to participate in the followup semi-structured interview.
4.2.3. Data collection and data analysis
The survey has been sent to all samples authors who did RCT in conflict-affected settings. Several publications were co-authored; the publications were 57 studies while the authors were 100 researchers in total which all were included in the sampling. Whereas co-authorship generally entails the risk that two authors of the same study responded to the questionnaire, this can be ruled out by the analysis of some descriptive variables. More specifically, only Liberia, Afghanistan, Bosnia, Uganda, DR Congo, Rwanda countries were mentioned more than once by respondents as the site of the RCT. Those respondents who had done the RCT in the same country, however, gave different responses on the type of intervention assessed. The respondents of the survey were 28 out of 100 authors from 57 publications.
4.3. Qualitative method
4.3.1. Sampling and the development of interview guide
The sample were respondents that followed up on the survey. It was voluntary and obtained through a formulated question in the survey that request respondents to kindly put their email-address if they wished to participate in a short follow up semi-structured interview where 4 respondents put their email-addresses. The interview guide was created based on patterns found in the statistical analysis of all responses as well as the individual response of the specific interviewee. The interview guide started with five general questions approached the four interviewees. Subsequently, a few questions were created particularly for each interviewee that only concerned their individual response. The interview guide was revised a few times before it was finalised, but also some changes have been made after it has been conducted. One amendment was after the first interview, while the second time was after the second interview as new insights came to light and helped to reset ground for the remaining interviews. For example, the first interviewee pointed out tracking participants as a crucial element when designing an RCT to be conducted in a mobile community, i.e in under-developed and post-conflict settings. Thus, question 2.1.5. that addresses high tracking rate was added to the interview guide (See appendix 2 for interview guide),
4.3.2. interview process
The four interviewees were approached through their email addresses that they provided in the survey. However, three of four interviews were conducted as one respondent was not reachable. The three interviews were held using Zoom Program as the respondents were based in USA. This way also has enabled to record the interviews by a feature in the program.
However, the second and the third interviews did not follow the plan made for the first interview. The researchers are expert in the field, where leading the discussion without sending the interview guide in advance, has sparked more conversation that allowed to gain more data and increase the quality of the answers.
The questions were reordered during the interview. For example, question 2.1.5. was answered before getting to it as the interviewee's discussion on question 2.1.1 at the beginning of the interview, was closely interlinked to the theme of question number 2.1.5. Thus, the interviewer proceeded with question 2.1.5. at that point in time (see appendix 2).
The duration of each interview differed across the three interviews, depending on the interviewees response to questions. There is not a specific period of time set for the interview, however 7-10 questions were planned which could last 20-30 minutes in approximate time. This was a reasonable duration to keep the researchers in the interview. However, the actual duration for the first interview took 35 minutes, whereas the second lasted for about 54 minutes, and finally the third interview was 29 minutes.
4.3.3. Data analysis
The primary data was analysed through thematic analysis method (Bryman, 2016. p, 584). The themes were addressed by the interview questions, which developed alongside the process of analysing the data. The process started by coding common and relevant answers to each question. The coding process has enabled to sort out the data within categories that later on was structured under themes. Some of these themes were incorporated as respondents did not address some questions separately. In return, the thematic analysis method has allowed to develop more codes.
4.4 Ethical Consideration
This research study applies the ethical codes to ensure the integrity of the study. These ethical principles are summarised in four aspects (see Bryman, 2016 pp. 125-131) which would be discussed in relation to both methods (survey semi-structured interview). According to Bryman, (ibid)quantitative research is more simple in handling confidentiality than qualitative method - That responses are anonymous and the data are illustrated without revealing individual identity.
confidentiality and that data would be used for the purpose of this study only. In accordance, the responses of the survey were anonymous except for those who liberality put their email-addresses to participate in the followup semi-structured interview. To this end, these four particular respondents did not enjoy this anonymity any longer.
Following the ethical principles in relation to the qualitative method, it was through a verbal consent; the interviewer confirmed that the data obtained would be handled with good care and that the participants reserved the right to decline answering any question. A permission was also taken from the researchers who were interviewed, that the thesis would make references to their names in association with the responses to the interview related questions.
4.5 Limitation and delimitation
One main limitation that confronted this project, was that this study was based on online publications of RCT. This meant that these studies were completed and challenges have been overcome. Yet, the missing perspectives of studies that were not completed and published, could pose constrains on this project. That is, the challenges faced the researcher in the course of the implementation of an RCT in conflict-affected contexts may strengthen the validity of the data. Thus, there has been an attempt to resolve this issue by brining up the discussion of this limitation in the interview guide; researchers were asked if they were aware of cases where RCT was not completed due to specific challenges related to conflict-affected environments ( see the questions in the appendix 2). Yet, this limitation is still valid.
This project has delimited the type of interventions that was included in the sampling; medical interventions were excluded as this project targets development and social interventions only.
5 Findings
responses to the survey questionnaire. Moreover, the qualitative findings also present separate aspects of challenges that are not part of the quantitative findings.
5.1 Quantitative Findings:
This section presents the findings of the survey among authors of studies that implemented an RCT design in a post-conflict environment. The section starts with a summary of the findings and moves then to more specific findings in selected categories.
Figure (1): Summary findings of challenges for an RCT in post conflict environments according to different categories
In order to compile the summary findings for each category, the answers to question number (1.4.1.) (see appendix 1 for the questionnaire) were numerically coded with the following scores: 1 – ‘not at all’; 2 – ‘a bit’; 3 – ‘much’; 4 – ‘very much’; and 0 – ‘I don’t know’. The scores were summed up over all respondents and the total score divided by the number of respondents. Hence, each column represents the average score. Results are shown for all interventions as well as separated for interventions in the areas of economic reconstruction and peacebuilding respectively.
Comparing different types of interventions, differences are most pronounced in those categories that relate to research quality. Researchers who implemented RCTs related to post-conflict economic reconstruction found maintaining both high internal validity and high reliability substantially more challenging than researchers in peacebuilding. Though prevalent, differences are less pronounced in categories relating to the feasibility of the research.
.
Moving to more specific findings on individual categories, figures 2 and 3 refer to internal validity.
As figure 2 demonstrates, the majority of respondents did not experience specific challenges that the post-conflict setting pose on internal validity. Moreover, there are no difference pronounced in responses between different types of interventions.
In figure (3) 42.2% of respondents have experienced changing conditions in post-conflict environment that pose challenges on internal validity. These respondents are from various interventions, i.e peace-building and economic reconstruction (see survey questionnaire in appendix).
Moving on to logistics, which is generally considered the most severe challenge, figure 4 details the type of logistical challenges that respondents faced.
More than half of the respondents answered that the institutional environment or the lack of transport challenged the implementation of the RCT much or very much.
The overwhelming majority of respondents did not consider displacement and destruction of infrastructure much of a challenge for the implementation of the RCT. This is surprising as both are often considered essential characteristics of post-conflict environments.
Using a similar way of representing the findings, figure 5 provides insights into respondents’ answers with respect to the type of ethical challenges faced.
Trauma, poverty and security were prime concerns of the respondents in relation to ethical challenges of implementing an RCT in a post/conflict environment. Note, though, that all except one respondent answered that precautions were taken to handle the ethical concerns.
5.2 Qualitative findings: results from semi-structured interviews
5.2.1 Quality of RCTTwo interviewees emphasised that the feasibility and the quality of RCT cannot be separated. I.e the instability would affect the research quality in a way that can lead to the interruption of a project:
“Years of instability and so that complicate the implementation of the research and that has consequences for internal validity “ (Interviewee 1).
According to interviewee 1, the internal validity is mostly about how you handle the logistical issues. That is, if the design is not based on population distribution, maps and specific information, then the reality may look different; villages that were supposed to be part of the sample, were not found in reality. Consequently, these logistical challenges affect the precision of inferences. Likewise interviewee 3 has linked the proper implementation of RCT to logistics :
¨In Uganda we had to build logistics from the scratch from the ground. If you want things to go well correctly, you have to be present on the ground ¨.
In conclusion the three interviewees argued that the reason why the survey’s respondents agreed that the quality of RCT is of less challenge than feasibility, is because the challenges that confronting the quality of RCT also exist in a non-conflict setting.
• Survey-related individual’s/general responses that concerns internal validity.
Interviewee 2 indicated in the survey that the changing condition has effects on the internal validity. He further explained that the changing condition was a curfew imposed by the government due to localised violence. The security issue and the logistical challenges have impeded reaching the training centre, which resulted in closing down one training centre of total 6 training centres. As a result, the sample size was reduced which could affect internal validity; ¨At that we got around the internal validity problem by just completely drop entire that neighbourhood. But still a problem potentially for external validity because we cannot estimate the effect of that neighbourhood anymore because it is not part of our sampling ¨
In general, internal validity can be undermined due to ethical concerns. Interviewee (1) has indicated this issue in relation to a program implemented in Afghanistan where the design has considered to give incentives to control group. The control group was given expensive incentives in which the real effect of the program could not be identified. I.e the internal validity of the program has been compromised with the ethical concerns. Similarly, interviewee (2) has indicated that participants may leave questions unanswered or not respond to the questions truthfully, which lead to attrition, non-random and non-responses. Consequently, these survey related issues affect the internal validity. Also, certain topics may be sensitive such as the engagement in violence that is motivated by religion. Thus, the researcher’s team had to create questions that were self-administered where a tablet and a headphone were given to participants to listen to the questions and answer them accordingly. Through this way, inconvenient situations with enumerators would be avoided and the internal validity would therefore, be increased. What was helpful according the interviewee, is that these questions were tested during a pilot study to see participants’ reaction to specific questions and hence to find a way to deal with issues that would likely have been arisen during the implementation of a study.
In addition, low internal validity can be caused by low tracking rate, which interviewee 3 has referred to, however he denies the link between this issue and post-conflict settings. The interviewee further elaborates on the other challenges related to internal validity which are linked to logistics; remote areas are difficult to reach. These not the only factors, other external factors such as partner's organisations, can affect the internal validity. That is, Partner’s organisations that are not interested in learning but willing to help people, are more likely to start implementing the research design and then cancelling it.
5.2.2. Feasibility of RCT
Question :-¨Both destruction and displacement are discussed in the literature as post-conflict phenomena that potentially impede the implementation of an RCT. The responses to my survey do not confirm this at all: neither destruction, nor displacement have been mentioned much as challenges for the RCT.
The findings of the semi-structured interview confirms that destruction and displacement have to do with a program placement. Interviewee 1 has further elaborated that war zone is a place which unexpected events can happen in a certain places but it does not last long;
¨the Liberia election study that was 2011, 8 years after the peace agreement and after a significant peered post/conflict reconstruction taking place, we were away from that the period of active conflict such that as major displacement issues have been resolved and whatever displacement issues that remained were fairly well contained ¨ (Interviewee 1).
¨In Nigeria, that is a place sporadic low scale violence that goes on there; every year or two there is some flare up where there is a religiously motivated riot¨ (interviewee 2 )
Whereas interviewee (3) argues that it depends on the type of sample rather than a setting. For example, farmers do not move even if violence erupted while youth moves and often integrate:
“I am working with farmers in Myanmar, in some parts in Myanmar are ongoing conflicts, but farmers don’t get up and move, so it is not that hard to find people when you are working with the population that is not very mobil” (interviewee 3)
Generally, tracking participants is very difficult whether in a conflict area or a non-conflict area, It can be more obvious in conflict settings but the problem exist in a non-conflict setting as it depends more on the nature of the sample (interviewee 3). On the other hand, Interviewee 2 argues that in order to maintain an RCT’s project, it is important to place projects in settings that would allow practitioners to operate and complete the project, e.g in fragile contexts.
In conclusion, the three interviewees have agreed that challenges of destruction and displacement depend on the area and the population. I.e most of RCTs’ projects are placed in areas that have some extent of peace, and destruction and displacement may be limited. This will enable researchers to work and complete the RCT projects.
• Survey-related individuals’ responses that concerns ethics
the matter that it is important to take into consideration that people in these setting is deprived from basic needs and their time spent in answering a survey or interview questions should therefore be compensated. The aim of the research is to contribute to development and wellbeing of people which is made through compensating people for their time. Compensating also is a way to deal with the economic sensitivity. However, conflict increases the economic hardships and thus researchers should be aware of these challenges, i.e staff should be trained to be sensitive to the situation. For example, when people being asked to talk about events that happened during the war, this may put people in a distress position. Therefore, the staff should comprehend the context before starting interviewing people.
In contrast, interviewee 3 has explained based on an experience by working with internally displaced in northern Uganda - that the economic hardships that people face, also exist in a non-conflict environment. That there are no differences between poor people in conflict/post-conflict setting and a non-conflict setting in terms of food insecurity. For example, internally displaced people who live in camps, have access to food through Food programs. However, subjects in a conflict-affected setting do differ from those in a non-conflict setting in terms of traumatic experiences. Similarly, interviewee 2 agrees with the fact that poor people in conflict-setting are not necessarily different to the people in a non-conflict environment; the curfews that the government imposed due to violence in the area, may have posed economic hardships. But in general, that is not the main reason of why people are poor. In Nigeria, people are still quite poor though people are not directly affected by the conflict.
• Survey-related individuals’ responses that concerns logistics
Two interviewees indicated to the survey questionnaire that there were logistical challenges that complicate the implementation of RCT. These logistical challenges were destruction of relevant infrastructure, lack of transport, lack of communication and other challenges which lies in security. Though security was a concern that has been related to the ethical challenges in the quantitative findings, but it has relevance to logistics in the qualitative findings.
developed since the pre-war, which have effects on the design and implementation of the study. Another relevant aspect to logistical challenges that was pointed out by the interviewee, was that to get stakeholder agreed on certain things;
‘’We worked with election commission and UN peace keeping operations. The UN peace keeping force commander was not interested in what we were doing and had the authority to say no. We had to persuade him that this worth doing’’.
On the other hand, interviewee 2 mentioned that security has challenged the implementation of RCT, i.e the safety of the enumerators to travel to certain areas as there are also a restriction in terms of time; a team has to be back at specific time. Furthermore, logistical challenges have also been related to gender sensitivity, for example leaders were not willing to talk to women, which posed some difficulties on the implementation of the RCT.
5.2.3 Other challenges of RCTs
There were aspects that the quantitative findings of this research have not included. I.e respondents were those who have completed RCT’s projects, which may lead to bias results of this research project. Thus, question 4 (see interview guide) is an attempt to find out what other challenges that may impede the implementation of the RCT’s projects and thereby not completed. The interviewees have pointed out aspects that are not related to the RCT method itself but to the actors involved in the research. These actors may either impede the implementation of an RCT or lead to interrupting a project, e.g a political view of some funding agencies. These agencies may fund the project for personal purposes and thus have control over how the design of RCT should be. This has a great relevance to conflict-affected settings where governments and organisations in these settings sensitive to politics and thus put constraints on researchers in how a project should be designed.
”Tracking rates can be much difficult when you have a mobil population, but you have to be ready for that. My colleague did data collection with internally displaced people and he was actually able to give high tracking rate” (interviewee 3).
6. Analysis
6.1 Quality and feasibility of RCT;
The findings of the quantitative survey show that challenges for the implementation of RCT can differ across different interventions.
Respondents who consider the quality of RCT more challenging for the implementation are those who implemented intervention related to economic reconstruction, while it is less of a challenge for peace-building interventions. The literatures have suggested that this can virtue of the type of interventions. I.e peace building addresses the conflict directly in which researchers are aware of the specific challenges that these setting may pose on the implementation of RCT and hence consider these at early stage when designing an RCT study. Whereas economic reconstruction related interventions such as education and health programs, do not tackle the conflict dimension where the setting can challenge a rigorous implementation of an RCT (Bozzoli et al, 2013).
Furthermore, respondents of the interview have indicated that the challenges encounter the quality of RCT in post-conflict settings, also exist in a non-conflict context. Thus, displacement for example is not a major challenge because a post-conflict setting can be similar to a non-conflict setting.
6.1.1 Reliability
reconstruction-related interventions and those who researched peace building-reconstruction-related interventions.
Furthermore, heterogeneity between units somehow affect the precision of the average treatment effect. The precision of ATE lies in the balance of other factors, in which they should be similar on averages between a treatment and control groups. If both groups are similar on averages, that means that the unobservable characteristics are correlated with the observable ones. But as it noted in the analytical framework - the sample size in reaching the precision of the average treatment effect in the case of heterogeneity, has been of great importance. However, conflict-specific factors - such as the example of Yemen - has obliged implementers to cancel the second sample due to the security issue. This as a result, has minimised the statistical power to test the impact of the program from different dimensions which weakened the reliability and subsequently to hold in another setting.
Furthermore, the analytical framework has presented that other factors could affect ATE, e.g the conditional cash transfer (CCT) that aimed to increase schooling in a conflict-affected setting. For the CCT case, it could be dangerous for children to attend school or they may have had other priorities at home than going to school. As a result, this context specific-factors could affect the desired outcomes of a CCT and would have undermined the program. Thus, unobservable variables could affect the observable variables even though they were correlated and similar on averages for the similar program in a different setting. Thus, differentiating between setting and the levels of complexity is important as it increases the consistency of ATE.
6.1.2 Internal validity
Respondents in figure (2) and (3) have experienced challenges such as changing conditions and other challenges that threaten the internal validity. Whereas most of respondents have answered that attrition was low and random (see figure 6 in appendix 1).
Firstly, the relation between the sample type and attrition is consistent with the analytical framework that suggests that the likelihood of attrition increases in relation to IDP sample. Thus oversampling in the case of IDP has a considerable importance. Secondly, the findings claim that high attrition can occur in a non-conflict setting and not only in a conflict-affected setting. This goes hand in hand with what has been indicated in the literature review that tracking participants has a great importance in under-developed countries that characterise of a high mobility rate, as high tracking rate will increase the validity of the study (Millan & Macours 2017).
On the other hand, the analytical framework suggests that attrition in relation to changing conditions, is a major challenge that affect the internal validity. For instance, in Egypt where an unexpected terrorist attack occurred, leading to a very high attrition that has implications on the internal validity.
The qualitative and quantitative findings confirm that changing conditions have effects on the internal validity and external validity. The survey indicated that 42,9% of respondents have experienced changing conditions. The analytical framework discussed the connection between context-specific related factors that characterise post-conflict settings, and changing conditions, which can affect the quality of RCT, e.g the changing conditions in Nepal. The changing conditions in Nepal has resulted in behavioural changes of participants which had consequences on internal validity, i.e the impact of the treatment could not be identified (Wolfe, forthcoming). Nevertheless, the program has shown positive effects on the people, but this positive effect can be attributed to the rule of invariant capacity presented by Cartwright (Cartwright, 2010). That is, the cause always has some fixed effect, i.e the cause alone contribute to a systematic outcome (Cartwright, 2010).Whereas an impact evaluation is the difference proved with and without the intervention, which has to be identified through a counterfactual value (White, 2013).
