I
N T E R N A T I O N E L L AH
A N D E L S H Ö G S K O L A NHÖGSKOLAN I JÖNKÖPING
Program evaluation and aid effectiveness
A case study of Sida as a learning organizationMaster Thesis in Political Science Author: Martin Salmonsson Tutor: Benny Hjern Jönköping January 1st 2009
Magisteruppsats inom statsvetenskap
Titel: Program utvärderingoch biståndets effektivitet Författare: Martin Salmonsson
Handledare: Benny Hjern Datum: 2008-06-01
Ämnesord: program utvärdering, den lärande organisationen, Sida, utvecklingsteori
Sammanfattning
Program utvärderingar utgör grunden till det formella lärandet inom Sida (Swedish International Development Cooperation Agency). Utvärderingars syfte är att bistå med kunskap och Utvärderingar ska garantera att insatser är baserade på god förståelse om verkligheten i mottagarländerna. Genom att gynna organisatoriskt lärande förmodas utvärderingar bidra till biståndets effektivitet (Stefan Molund, 2004).
Teorier om den lärande organisationen hävdar att organisationers fall beror på medlemmars tendens att förenkla och misstolka verkligheten. Medvetet eller omedvetet leder detta till att organisationens vision försvagas, medlemmars engagemang försvagas och den verklighet man sökt att förändra förblir den samma (Peter M. Senge, 1994).
Genom att bistå sektorer som hälsa, utbildning och demokrati mm. har det svenska biståndet präglats av en objektiv eller positivistisk syn på verkligheten i utvecklingsländer. Insatser inom hälsa leder onekligen till effektivitet i fattigdomsbekämpningen.
Min uppsats visar hur Sidas strävan efter objektivitet löper risken att försvaga organisationens vision. I de fattigaste och mest socialt komplexa utvecklingsländerna finns inte
förutsättningarna1 för objektiv utvärdering, och resultaten av en utvärdering blir ofta öppna
för olika tolkningar. Trots att utvärderingarna skildrar en sann bild av verkligheten så leder detta till frustration hos handläggare. Resultatet blir att utvärderingar görs av program som handläggarna redan har god kunskap om. Mitt resultat visar att denna trend successivt
försvagar Sidas vision då insatser ämnade åt att öka effektiviteten i biståndet allokeras från de ”fattigaste länderna” till länder som kommit längre i utvecklingsprocessen.
1 De “fattigaste” utvecklingsländerna saknar de institutioner för datainsamling som krävs för objektiv
utvärdering. Att skapa förutsättningarna för den kostnadseffektiva objektiva utvärderingen i utvecklingsländer är ett utvecklingsmål som vilket annat som ingår i den övergripande fattigdomsbekämpningen.
Master thesis in political science
Title: Program evaluation and aid effectiveness Author: Martin Salmonsson
Tutor: Benny Hjern Date: 2008-06-01
Subject terms: program evaluation, learning organization, Sida, development theory
Summary
Studies of how and why organizations learn have been going on for many decades. This paper is an attempt to study the role of evaluation for learning within the field foreign aid. The study is focused on Swedish aid and Sida (Swedish International Development Cooperation
Agency), and its program support to Angola. The first part is done as a text analysis of four programme evaluations in Angola between the years 1991 - 2006. The second part is an interview of seven senior program officers at Sida head quarters in Stockholm.
With no generally accepted development theory to rely on in a highly heterogenic developing world, foreign aid politics has been characterized by a “learning-by-doing” process (Erik Thorbecke, 2000). Despite the uncertainty surrounding development theory, Sida´s work is guided by a strong common vision to fight poverty through ODA (official Development Assistance). In the 21st century, after 50 years of cooperation, the contining high levels of
poverty Sub-Saharan Africa has started to tear on this vision (see Figure 2).
Organizational learning is dependent on a common and realistic vision. If vision and reality is too different the common vision of organizational members is often lowered in order to ease the tension. In such cases the organisational members do not learn anything new, passion for ones work is replaced by frustration and the organization slowly looses it’s purpose (Peter M. Senge, 1994).
Conditioned aid and the redirection of ODA from project- to sector support illustrate an objectivistic belief among donors that there is a “best way” for developing countries to escape extreme poverty. External evaluations are expected to supply the objective information about program effectiveness and efficiency.
In the poorest countries of the world preconditions for conducting objective evaluation are rarely present. Creating the institutions necessary for objective evaluation in these countries is a part of the larger development process. This thesis concludes that the strife toward
objectivity runs the risk of creating a passive- rather than a creative tension between vision and reality within Sida. Program evaluations today become more important bases for decision-making, and thus it is possible that countries unable to satisfy the conditions
necessary for objective evaluations will be denied ODA. The vision of fighting world poverty through ODA is changed and lowered. The new and less ambitious vision is: to fight poverty in poor countries developed enough to satisfy the necessary conditions for objective
evaluation. In the complex field of ODA, with a strong humanitarian purpose, a lowering of donor visions run great risk of leading to a legitimacy crisis (Figure 4).
Table of Content
1. Introduction...5
1.1 Problem and Purpose...6
1.2 Research questions...7 1.3 Limitations...7 2. Method...8 2.1 Text analysis...8 2.2 Qualitative interviews...8 3. Theory...9
3.1 Learning organization according to Peter M. Senge...9
3.1.1 Building shared vision...9
3.1.2 Personal Mastery...10 3.1.3 Mental models...10 3.1.4 Team learning...11 3.1.5 System Thinking...11 3.2 Evaluation theory...12 3.2.1 Pseudo evaluations...12
3.2.2 Objectivist/positivistic evaluation approaches...12
3.2.3 Subjectivistic/Relativistic evaluation approaches...13
4. Data gathering/Empery...14
4.1 Text analysis - Background...15
4.1.1 The 1991 Evaluation...15
4.1.2 The 1999 Evaluation...17
4.1.3 The 2003 Evaluation...21
4.1.4 The 2006 evaluation...23
4.1.5 Summary of text analysis...25
4.2 Interviews with executive officers – results...27
4.2.1 Defining evaluation...28
4.2.2 Prior knowledge...29
4.2.3 Relevance of evaluations...29
4.2.4 Objectivity vs. relativity...31
4.2.6 Relevance and accuracy...34
4.2.7 Summary interviews...35
5. Discussion...37
6. Conclusion...42
References:...44
Appendix A: Interviewguide...46
Apendix B: List of abbreviations...48
Appendix C: Relevant figures and graphs...49
Appendix D: Evaluation quality standards, principles and criteria...51
1. Introduction
This chapter will present the purpose and scope of the essay. It gives the reader an insight into the current debate surrounding Sida, evaluation and learning organization. My focal country is Angola. It also presents the research questions and limitations.
The history of ODA (Official Development Assistance) has been characterised by a learning-by-doing process since no generally accepted development theory has been fully applicable to the complex development circumstances in the world’s poorest countries. Estimating the real impact of ODA has been difficult. External factors outside the control of the program
sometimes have greater influence on overall poverty reduction and overshadow the impacts of ODA.
In many places around the world, especially in Sub-Saharan Africa, countries remain at the same low levels of GDP/capita as they did at the time of independence (World Bank Indicators, 2001. Figure 1). The low levels of poverty reduction in many African countries have increased public mistrust in ODA. Since the early 1990´s the donor community has chosen to condition aid in hope to increase effectiveness in supported programs. Public mistrust makes foreign aid politics vulnerable. Donors intending to achieve effectiveness may act on public pressure instead of on empirical knowledge about how to best fight poverty (SOU, 1994).
“True performance (effectiveness) is difficult to measure, but some evidence is worrying especially for Sub-Saharan Africa where development agencies have a record of over optimism.” (UCD economics, Global poverty and foreign aid, 2006, p.27)
In a Swedish governmental independent study on Swedish foreign aid politics in 1994, the concluding remark was that ODA is surrounded by uncertainty to which Sida have to adjust in order to achieve effectiveness (SOU, 1994) . The dominating theoretical framework to guide such an adjustment is based on development economics. If certain policy requirements are met by recipient countries, certain objectives will be achieved. The uncertainty that exists has given rise to a high level of polarization of ideas amongst those politicians and economists who have worked within this field. In newspapers we read how debates arise between those who believe in ODA´s ability to contribute to development (i.e Jeffrey Sachs) and those that argue that the only impact of ODA is an increased dependency (i.e William Easterly). A result of this debate has been that the pro side has begun arguing for the reallocation of aid to countries with “good economic policy” to reduce dependency and increase effectiveness (Burnside and Dollar, 2000). Peter Boon, a famous economist and theorist in development aid, argues that aid should be allocated to countries with capacity to evaluate programs. From the current shape of things it would be expected that such an approach would perhaps lead to aid allocation to countries which may not be the most needy in terms of poverty conditions (Effective intervention: making aid work, chapter 3).
In my bachelor thesis I concluded that this polarization is the result of political unease rather than of an in depth analyze of the field. Field workers argued that, despite the dependency on ODA, they saw positive effects. If this is true it means that the lens through which officials and politicians view poverty reduction efforts tends to miss certain aspects of the programs observed from a “field perspective”.
At the end of the Cold War in 1989, development thinking took over as the main driving force within politics on foreign aid. Recipient countries, who had been receiving ODA primarily in order to fight either communist or capitalist ideals, began receiving foreign support more specifically to fight poverty. The end of the geopolitical nature of ODA led donors to
condition their support rather than giving it for solidarity reasons. The effectiveness of ODA became a priority and conditioned aid gave donors hope that funds would be allocated more effectively. The 1990´s was characterized by a donor perspective on ODA that was focused on effectiveness and results. Evaluation, monitoring and other follow-up activities assumed that conditioned aid would ensure effectiveness (Sida evaluation manual, looking back, moving forward, 2004).
ODA is a complex venture, especially in Sub-Saharan Africa where questionable governance and the dimensions of poverty2 daily affect development progress. Seen from a learning
organizational perspective, donor agencies and recipient governments share a vision and objective of fighting poverty and increase the living standards of the poorest people in the world. The main actors are still divided into donor and recipient, which indicate that
alignment and harmonization within cooperation are not as present as they appear to be. The fact that donor agencies feel urgent to condition aid illustrates how donor and recipient may differ in their opinions on how to best use ODA.
The donor community strives toward objectivity. They approach reality despite the absence of a generally recognized development theory (Thorbecke, 1999). While the learning-by-doing approach has characterized the evolution of ODA, certain outcomes are assumed achievable as long as certain inputs are provided. It is from observations of reality founded on an objectivist epistemology that cooperation members are supposed to learn about their work. I question that objectively assessing donor supported programs is as achievable as assumed within current ODA programs. I believe that developing countries, at the beginning of a development process, are not ideal for conducting such objectivistic program evaluations. “How one chooses to evaluate a supported program obviously affect what types of
conclusions one can draw. The chosen method in turn is related to the way one observes the support… In Angola the objective was to make the whole health sector work better, but with the scarcity of reliable population data the program is not assessable in relation to this objective.” (Evaluation 1991, p4)
1.1 Problem and Purpose
Sida has a noble and important task: to fight poverty in the least developed countries around the world. The main problem of this thesis is that organizations that do not learn tend to fall apart as reality remains unchanged while their vision is weakened.
The purpose of my study is to analyse Sida´s program evaluation process against a theory of the learning organization presented by Peter M. Senge (1994) in order to see if evaluations contribute to a passive- or creative tension among the executive officers at Sida.
2 “The dimensions of poverty” is described by the World Bank as being the correlated factors which
simultaneously and negatively affect poverty conditions in a country such as corruption, femine, violence, crime etc.
1.2 Research questions
The main question guiding this study is: Is SIDA a learning Organization? What is a learning organization?
How does program evaluation contribute to learning?
Is Sida´s evaluation approach fit to deal with situations in the field? Is Sida experiencing a learning handicap?
1.3 Limitations
This study is limited to the bilateral cooperation between Sweden and Angola, one sector (health) and one way of retrieving knowledge (evaluation).
Program evaluation is integral part of the learning process within Sida. It is formal rather than informal learning that is under focus in this study. The formal procedures usually get to represent the learning characteristics of the organization (Timbro, 2007). Informal learning includes day-to-day dialogue and discussions around the coffee-table (Sida evaluations, 2004). What is of importance here is the formal means used by the organization in order to facilitate learning. Program evaluations are examples of tools for formal learning within Sida. Evaluation can be conducted on many levels within an organization. Program evaluation is one of many types of evaluations through which the stakeholders3 retrieve knowledge. This
study has chosen to focus on program evaluation. Other forms of evaluation (i.e. policy evaluation) have been conducted by SIDA but falls outside the scope of this study. These are the thematic evaluations conducted not by the embassies or the specific sub-divisions but by the special The Department for Evaluation (UTV).
The study recognizes that there are other methods of retrieving knowledge within Sida (i.e. monitoring and reporting). This study looks at program evaluation because it is aimed at a deeper assessment of program impact and also has an official purpose of contributing to learning.
The text analysis is limited to the Swedish-Angolan cooperation. This cooperation was phased out in 2006. The health division at Sida, under the division of social and democratic
development (DESA) is dealing with many programs around the world. My text analysis is trying to determine which evaluation approach (see chapter 3.2) is used by Sida, how it is applied to the field. I have chosen one specific sector and a single focal country in order to do a comparison over time.
3 Stakeholders in a development program or project are the owners and the beneficiaries of the program.
The selection of interview respondents is based on their experience and position within Sida. The interviews are also limited to the executive officers in control of the evaluated programs at Sida Stockholm.
There are many different studies made about organizational learning. Argyris and Schön wrote about the single- and double loop theory (Argyris, organizational learning 2nd edition,
Blackwell business 1999). Georgy Bateson and W. Edwards Deming are theorists known for developing concepts for organizational learning. This study will not directly consider their theories as their work is more orientated towards specific types of organisations. I choose to focus on Peter M. Senge in this study. He has developed a terminology and theory which is applicable to this study. His book, the fifth discipline, is the base for many studies on organizational learning. It gives the researcher a lot of freedom to interpret and apply his theory on different organizations.
2.
Method
2.1 Text analysis
The primary purpose of the text analysis is to determine the epistemological and ideological characteristics of Sida´s evaluation approach4. This part is done by comparing four program
evaluations, done in Angola between 1992 and 2006, with Sida´s evaluation manual. When I have determined the extent to which these evaluations follow the guidelines of Sida´s
evaluation manual they are compared with 3 evaluation approaches presented in chapter 3.2 in order to determine their epistemological characteristics.
In the second part I analyze four program evaluations in order to see whether their
epistemological characteristics influence their ability to assess the health support in Angola. The purpose is to see how the evaluation manual is utilized in the field. The purpose of this text analysis is to determine how suitable Sida´s evaluation approach is in reality and how the theoretical limitations and advantages of the specific approach are experienced in practice.
2.2 Qualitative interviews
In order to say anything about learning, I approached the primary learning subjects of the program evaluation: the executive officers at Sida, Stockholm. The purpose of my interviews is to show how recommendations from the evaluations are taken into consideration. Within each division there is an executive officer who is in charge of a specific country support. The executive officers have a direct power over the programs; they write terms of reference (the evaluation questions), order the evaluations and hire the external consultants. They are in charge of writing a management response letters as well as distributing the final evaluation report to various stakeholders. It is mainly through the executive officers that evaluations
4 History has created two schools of thought with different beliefs of how to best assess programs and/or projects
in reality. Depending on whether you believe that reality is objectively observable or whether you believe that everything is relative different methods are used and different expectations are placed on program evaluations (see chapter 3.2).
contribute to organizational learning. Even if they are not at the top of the hierarchy, they are in a central position when it comes to learning from evaluations.
The interviews use “open questions”. By letting the executive speak freely about their experiences with program evaluations, as a learning tool, I am letting the interview unfold itself naturally rather than by force.
Ten executive officers at Sida´s headquarters in Stockholm were asked to take part and seven finally agreed to participate. The interviews were carried out during one week in the fall of 2008. Each interview took about 40 minutes.
3. Theory
This chapter presents and defines the concept of “learning organisation” according to Peter M. Senge in the context of program evaluation.
3.1 Learning organization according to Peter M. Senge
For Peter M. Senge (1994) a learning organization manages to change their reality while keeping their vision constant because they learn. A “non-learning” organization5, according to
Peter M. Senge (1994), lowers its vision and related objectives rather than attempting to reach them by changing its approach. By keeping “an open mind” an organization avoids the risk of making false judgements about reality.
Peter M. Senge explains how a learning organization is characterized by a constant strive toward development and change because it is in its best interest. Innovations such as the airplane or the personal computer consist of components. When talking about innovations in organizations Peter M. Senge defines the components as disciplines. Dedicating oneself to a discipline means that that you can never settle down. Just as it is useless to say “I am an enlightened person” it is equally ineffective to say “we are a learning organization”. The more we learn the more aware we become of our own ignorance. (Peter M. Senge 1994)
The five “learning disciplines” presented by Senge are known as personal disciplines. They all have to do with how we think, what we really want and how we collaborate. They are: System thinking, personal mastery, mental models, building shared vision and team learning.
3.1.1 Building shared vision
Shared vision stimulates a shared commitment. It stimulates a creative tension between reality and vision. A shared vision contributes to solidarity and stimulates cooperation.
In most classical organizations personal- rather than shared visions are present. The vision exists among a small group of people whom are trying hard to impose it on the other employees or members of the organization.
Many visions are directed outwards and strive to create a reaction against an external party. However a vision that does not show any higher ambition that to beat an opponent is
5 Peter M. Senge has no antonym for “the learning organization”
insufficient. When it is achieved it easily becomes defensive and the organization only aims at preserve what it has.
What we want to create differs from what we want to avoid. Many organizations gather strength only when their existence is threatened. There are two basic “energy sources”, which can motivate an organization; fear and aspiration. Fear produces negative visions and a defensive tension (see personal mastery) and aspiration produces positive visions. Fear can lead to changes in the short run, while aspiration can lead to development in the long run. “What is characteristic for a learning organization is not their grand vision but rather serious strive to keep close contact to reality as a basis for its vision.“ (Senge, p 208)
3.1.2 Personal Mastery
“A truly creative person knows that one has to subject to resistance in order to create. Without pressure this is not possible.” (Senge, p. 148)
Organizations learn because individuals learn. Development on an individual level does not guarantee a positive change for the organization, but it is a precondition. Personal mastery is developed as the individual strives to realize a vision, while being aware of the reality. Many feel that it is hard to talk about their visions even though they are aware of them. Peter M. Senge argues that this is a result from a division between the visions and the current reality. All persons have visions, while stuck in reality. The difference between reality and vision creates a tension. There are two ways to reduce of this tension; ether your vision moves toward your reality or the other way around. In this sense, the tension is either defensive or creative. From a personal mastery point of view, this tension is something positive (creative). It constantly boosts our ambition as we strive to move reality closer to our vision. This leads us to constantly set new goals while keeping the vision unchanged.
In many cases our vision is based on things we want to avoid rather than things we want to achieve. Although these are visions as well, they are negative visions since they make us put energy into defending ourselves rather than creating something new. “When fear of failing dominates our reality a defensive tension causes our vision to move closer to reality. When this happens the tension is relieved, leaving us powerless and numb”. (Senge, p 143-145) A sense of powerlessness is often imbedded in our sub conscience. These feelings create a structural conflict where belief in our ability to create disappears. The way to deal with such structural conflicts is to keep to the truth.
3.1.3 Mental models
Our mental models affect our ability to observe the system as a whole. They are closely linked to personal mastery as they have a tendency to be hidden in our sub conscience. Personal mastery makes us aware of our mental models. Prejudice is the most common mental model. Mental models have great impact on our behaviour when the reality is hard to grasp. In such situations they work as a defence mechanism, allowing us to jump from observation to generalisation. What happens in most cases where mental models are at work is that what was
once treated as an assumption is now considered a fact. Without a good understanding of our mental models generalizations will often result in short-term solutions.
In the classical hierarchical organization there is a tendency among leaders to force their own mental models on their co-workers. Mental models are impossible to get rid of and therefore consensus is not something that the organization should strive for (Peter M. Senge, 1994). What is important is to enable different interests to be presented and listened to.
Two proficiencies are needed to enable organizational development; reflection and
examination. To reflect is to slow down the thinking process in order to understand the mental models at work and how they affect our actions. Examine helps us to handle interactions with other people, especially in complex situations with many differing interests.
3.1.4 Team learning
In modern organization we work in teams and not by ourselves. The discipline “team
learning” begins with “the dialog”. Dialog means employees ability to collaborate and “learn together” with an open mind. Dialog differs from the normal discussion in which opinions and arguments are thrown back and forth in a struggle to win or loose. In the discipline “team learning” one learns to recognize the patterns of defensive reactions which undermine the dialogue.
“Team learning is important because modern organizations are built in such a way that education occurs within groups and not individually.” (Senge, p 23)
3.1.5 System Thinking
According to Peter M. Senge system thinking is to see the world in a new way. Organizations fail to learn if they and their employees do not see the system as a whole.
The essence of a learning organisation is found in the discipline of system thinking. The human being has a natural ability to observe how different happenings affect each other and lead up to a specific result. By observing weather patterns such as darkening skies, cloud formations and increased wind we can conclude that it will soon start to rain and we hurry to seek cover.
In an organization the same forces are present. Being a part of “the network” and not external observers it is more difficulty to see the whole pattern. Our tendency to be astonished by occurring problems is proof of our inability to se the system as a whole. It is hard to recognize which solutions move the problem from one part of the system to another, since those who solved the problems in the past are seldom the ones that have to cope with problems in the future. Today’s problems come from yesterday’s solutions. Senge explains that short term solutions are what lead to long-term problems. We want to see fast results when problems arise. System thinking is what Peter M. Senge refers to as the fifth discipline. It logically connects the other four disciplines (personal mastery, mental models, building shared vision and team learning).
3.2 Evaluation theory
Evaluations are the formal tools used by organizations to learn about the true “merit and worth” of a specific program or intervention. According to classical evaluation theory the definition of evaluation is: “…a study designed and conducted to assist some audience to assess an object’s merit and worth” (Stufflebeam, 2000). In development cooperation the definition has been slightly modified. According to the Organization for Economic and Social Development (OECD) and the Development Assistance Committee (DAC) the definition of evaluation is: “… A systematic and objective assessment of an ongoing or completed project, program or policy, its design, implementation and results.” (SIDA, 2004)
Evaluation is dependent on other forms of follow-up activities such as monitoring. Monitoring records activities and results against plans and budgets, an evaluation investigates deeper. It is argued that monitoring, although signalling failures in a program, can not explain certain problems. To deal with such questions of cause and effect, evaluations are required. (Sida evaluation manual, 2004). Program evaluations done by Sida follow a set of criteria and quality standards. They are presented in detail in Appenix D.
Evaluations are not only conducted for the purpose of learning (formative). Another purpose is control and accountability (summative). Most evaluations try to satisfy both purposes. In theory, however, the final use of the evaluation differs depending on who reads it. Robert E. Stake explains this difference: “When the cook tastes the soup, that is formative; when the guests taste the soup, that is summative” (Looking Back, Moving forward, p 20)
3.2.1 Pseudo evaluations
Approaches categorized under this heading are recognized by Stufflebeam as being bad or questionable practices. They deliberately shade or falsify findings. Pseudo evaluation defies the essential purposes of evaluation, a systematic assessment, and is deliberately biased. These approaches are often motivated by political objectives. Persons seeking authority may be tempted to present unwarranted claims about his/her achievements. Self-interests guide such evaluations toward pre-established results or answers. Evaluations become a tool to maintain an unfair advantage. Pseudo evaluations fail primarily as legitimate evaluations in their distribution of unbiased information to involved stakeholders. Regardless of real worth of the program the evaluation will be used to state something either positive or negative about the program.
3.2.2 Objectivist/positivistic evaluation approaches
“…it is better to answer a few questions well than to attempt a broad assessment of something’s merit and worth.” (Stufflebeam, 2000, p. 29)
Also known as quasi-evaluation the approaches falling under this category are using given sets of methods or questions in order to ensure technical quality rather than full coverage. They are labelled quasi-evaluations since they sometimes provide information that fully assesses one object’s merit or worth, while failing completely in others. Characteristic for these approaches is that they are all pre-occupied with using specific methods or questions rather than primarily focusing on what is needed for the specific situation. (Stufflebeam, p30) The objectives-oriented- and outcome evaluation approach is a classic example of outcome evaluations. (Stufflebeam, p 101). All other factors outside the scope of the objectives related outcomes may be overlooked. This may result in too narrow conclusions for judging the program’s merit and worth.
One example of an objectivistic evaluation approach is the decision/ accountability-oriented approach. It includes an objective orientation to finding best answers to context-limited questions. Like most objective evaluations it aims at engaging all stakeholders in focusing the evaluation. In this approach the advance organizers include not only decision makers but stakeholders at all organizational program levels.
The objectivistic approach to understanding reality prefers quantitative information. Differing of interests is believed to be a human error. It looks for measurable data and not opinions in order to reach generalizability.
3.2.3 Subjectivistic/Relativistic evaluation approaches
“… Needed information may not be anticipated at the beginning of an evaluation. Thus, it is better to addressing “all” stakeholders questions although it may lead to less homogeneous results.” (Stufflebeam, 2000)
Commonly they seek to make a difference in society through program evaluation by focusing on empowering the disadvantaged stakeholders. They favour a constructivist orientation and the use of qualitative methods in determining a program’s merit and worth. As a result they avoid the possibility of finding best answers by stressing cultural pluralism and the presence of multiple realities.
The social mission is what is of importance in these approaches. Some consider these approaches to be too concerned with objectives that may not fully relate to the standards of evaluation. These approaches run the risk of producing biased evaluation results as focus is put on the disadvantaged stake holder’s participation, rather than supplying accurate information about a program’s merit and worth. Evaluations using this approach often
culminate in conflicting and subjective findings due to their pluralistic nature. In many cases, such as in the Client-centred approach, the problem of pluralism is avoided as no final
authoritarian conclusion but rather leaves interpretations to the final reader.
The main advantage of these approaches is that they strongly advocate democratic principles of equity and fairness. They employ practical procedures for involving the full range of stakeholders.
The clients must be willing to endorse a quite open, informal and flexible evaluation plan and being receptive to an equal and fair participation of a representative group of stakeholders. Thus an appreciation of ambiguity is a must among stakeholders.
In contrast to the Objective evaluation, the evaluator’s goal is to carry on a continuous search for key questions during the course of the evaluation. As no best answers are assumed the evaluation questions are believed to be rather loosely formulated in the beginning of the study process. It will become more precise as the evaluator interacts with stakeholders
(Stufflebeam, p 81). Special attention is given to side effects of the program rather than outcomes related to specific objectives. The evaluator uses all relevant techniques to portray “the complexity of the program” and communicates the complexity even if the result makes decisions more difficult. The main limitation of the approach is related to external credibility since local autonomy is favoured. This may also affect the independent perspective of the evaluators. Rather than bringing closure, the approach may bring confusion among stakeholders.
Constructivist approach is another example. In contrast to the client-centred study this
approach’s methodology goes from being divergent to convergent. While retrieving ultimately correct answers is considered impossible the evaluator seeks closure by continuously
engaging stakeholders in dialectical process aiming at achieving as much consensus as possible. The evaluator employs a relativist perspective, stressing locality and specificity over generalizability (Stufflebeam, p 72).
These approaches seem utopian because it demands the continuing participation of a wide range of stakeholders. This decreases the possibility for the evaluation to produce the timely reports that funding agencies and decision makers demand. Sometimes stakeholders,
especially those with little influence, are poorly informed and are thus poor data sources. Due to the evaluators tendencies to report competing answers this approach is not favourable as a base for decision making (Stufflebeam, p 74).
4. Data gathering/Empery
In this chapter I have done a text analysis of the four comprehensive program evaluations of the Swedish health sector support to Angola. Sweden has through Sida given aid to Angola during a 30 year period from 1977 to 2006.
14 Fact Sheet Angola
Population about 14 million Capital Luanda
Population Growth Rate 3% per year GDP growth 11,7% (2004)
GDP/capita 1305 USD (2004 estimate) Inflation 36% (2004)
Foreign Debt 9,1 billion USD (2004 estimate) Budget Deficit 5,3% of GDP (2004 estimate)
Current Accounts Balance 84 million USD (2004 estimate) Export 9.6 billion USD (2003 estimate)
Import 4.1 billion USD (2003 estimate)
Corruption Perception Index 133th place of 146 countries (2004) Human Development Rank 164 of 175 countries (2003)
Poverty level about 63% under the poverty line (2000) Under five mortality rate 260 per 1,000 (2001)
Maternal Mortality rate 1,700 per 100,000 live births (2004 estim.) (average in developing countries 440)
Adult Literacy Rate 42%
HIV/AIDS Prevalence rate 5,5% (estimated national average 2003) Access to safe drinking water 38% (2003)
4.1 Text analysis - Background
The main goals of the bilateral health support to Angola have, as in most cases, been to reduce poverty and contribute to long-term development (evaluation, 1991). By supporting primary health care in Angola, the Swedish support tried to accomplish these development goals. The first evaluation of the health support to Angola was conducted in 1991, twelve years after the initiation of the programme. Prior to 1991, assessments of programs took the form of reports on budget disbursements and basic presentations of results (evaluation, 1991). The Swedish support started as a long-term development support to a wide range of health programs. In the beginning it was guided by positive future expectations. Later on the program changed to a more short-term humanitarian disaster support, characterized by a concentration of support to projects in the capital province. Since Angola had one of the highest maternal mortality rates in the world, one of the main objectives of the Swedish-Angolan cooperation from 1990 to 2006 was to treat this situation.
4.1.1 The 1991 Evaluation
An evaluation of the health support had not yet been done, making the purpose of the 1991 evaluation to summarize and critically analyze experiences from the past twelve years. Special attention was given to how the support had emerged, and what circumstances had influenced the development. The evaluation was also aimed to serve as a basis for decision-making for future support.
The following key issues were identified. 1) The relevance of the support in relation to the health situation and the common situation in Angola. 2) To what extent had the goals been achieved in the various programs. 3) Had the programs been efficient? 4) The relationships between the various programs. 5) The characteristics of the programs planning- and leading functions. 6) The role of personnel support. 7) Program sustainability.
The evaluation was conducted by a team of external consultants hired by Sida. The evaluation describes how the support to the various programs has emerged during the cooperation period, and the factors influencing the development of the support. The report continues to describe each of the programs separately using the above evaluation questions. Finally the conclusions from each program evaluation are synthesised and the evaluation questions are answered on a general plane.
The historical description of the support is based on Sida`s own documentation. These documents are related to the Angolan proposals, agreements and reports about the general situation of Angola’s health care.
The team of consultants used a limited amount of systematic evaluations conducted by persons not engaged in the cooperation. This is called an independent perspective with the purpose of yielding objective data. Additionally interviews with the Angolan counterparts in the Sida supported programs were conducted and called an Angolan perspective.
An interesting observation is done very early in the evaluation. The evaluation team seems to question an objectives-based approach. They argue that evaluating program impact is difficult in the circumstances present in Angola. The evaluation team had to deviate from the
evaluation criteria as these were difficult to answer. They were only answerable if some
preconditions were met, such as reliable population data, which for understandable reasons were not present in Angola. This suggests that Sida at all times, even before the evaluation, assumed that objectivity would be attainable simply because the cooperation rested on a good and stable theoretical foundation.
“The Sida-support is a sector support. The goal has been to make the health sector as a whole work better by supporting an amount of key programs; a positive synergy effect was believed to emerge… If the aim is to study the effects then one should accordingly focus on how the whole health care has changed during the period of SIDA support. Data regarding health care and the health situation in Angola is however very limited. Therefore such an analysis is not very meaningful.” (Sida evaluation, 1991)
The evaluation team had to deviate from the original evaluation approach in order to draw any conclusions about the program´s merit and worth. The following quote is an example of this: “… It has been hard to get a grip of the health support. This has to do with the huge amount of programs supported. Despite the existence of a great deal of information, it is too often not relevant in answering the above evaluation questions… Much of the support that has been given for long-term development has received the characteristics of ad hoc emergency aid.” The consultants explain how the documents supplying information about the programs are scanty and subject to various interpretations. They see that central stakeholders are the only ones aware of the Sida-supported activities. Interpretations of the relevance and soundness off the support vary amongst them.
“We asked Angolan program workers what type of support has been the most meaningful. On a provincial level and in the health centres few people knew what SIDA had supported, so it is primarily co-workers on the central level which have given their opinions. The main
objectives of the support were formulated in general terms. This leads to differing interpretations of goals. Whether the support has achieved its goals depends on what perspective you take and whom you ask.”
The consultants also points out that SIDA must take into account that Angola is a country at war. Many of the evaluation questions are similar to those in “normal” development
circumstances and consequently are difficult to answer.
Conclusion
The main observation is that despite the need for the consultants to change their approach the only ones aware of the Sida activities are central stakeholders. The lack of objective data causes the consultants to turn to more subjective information. The real program impact should be to reduce maternal mortality rates and improve health. This impact can not be measured in the evaluation. In order to pursue an objective approach the consultants turn to interviews with stakeholders. A problem to the evaluation emerges when the consultants find that the only ones aware of the Sida support are stakeholders at the central level.
Many of the evaluation questions are a blueprint of standard evaluation models and not adjusted to the complex circumstances of a country in civil war. Problems occur when dealing with questions of effectiveness and efficiency because objectives were loosely formulated.
The cooperation had been initiated on optimistic assumptions and not taking into account the country’s complex realities.
A trend throughout the evaluation seems to be that the evaluator(s), based on the objectives-based approach taken, struggles to avoid judging the programs as failures They cannot see any general improvements in the Angolan primary health-care situation. They come to the
conclusion that the continued support is relevant not because specific objectives have been met but because the overall health situation in Angola is in equal need of support as it was when the cooperation began in 1977.
The evaluators understand that when Sweden started to condition the support, the Angolan party took an increasingly defensive position. They saw this change as a sign of increased control. It led to difficulties of interpretation of the often loosely formulated program goals. This increased subjectivity was not as present when support was given unconditionally (Evaluation 1991, p 8).
The fact that Sida initiated the evaluation and the evaluation approach initially seem to assume that objectivity will be retrievable, implies that Sida and the central-decision makers in Stockholm already felt that they had a good idea about the program’s merit and worth but needed an evaluation to confirm this. The evaluation approach, which claims that it is better to answer a few questions well than engage in an in-depth analysis of a programs merit and worth, guided the evaluation. As specific data is missing, the evaluation is forced to turn to a less objectivistic approach and interviewing stakeholders. The 1991 program evaluation come to the conclusion that stated objectives had been met to a low extent. The evaluators
understand that program goals are hard to estimate in the complex situation which continues to face Angola, but concludes that program objectives must be made more clear in order to make any judgements about program effectiveness (Brolin et al, 1991).
4.1.2 The 1999 Evaluation
In 1992, after the finalization of the 1991 program evaluation, Sida decided to concentrate its health support to the Capital of Luanda. The maternal health program, which had been initiated due to the high maternal mortality rates observed in the central hospitals in Luanda, would be the main focus of the support from 1995 and onwards. This concentration was caused by the increased administrative burden of the Technical assistance staff (TA) put on Sida. The 1999 evaluation therefore had to deal with a more concentrated and less diversified program support. The problems facing the objectives-based evaluation approach from 1991, would however reappear as the complex situation (the civil war, immigration from periphery to Luanda etc.) continued to make objectively observable impacts of the program difficult. According to the terms of reference there are two main purposes for the -99 evaluation. Primarily it would assess the outcome of the support in relation to program objectives. Such an assessment as important, particularly as the support was aimed at being strategic. Secondly the evaluation would be used as input in the decision process for the future support. (Sida evaluation, 1999, Appendix A)
The following key issues where identified: 1) To describe the development of the support since 1988 in terms of financial-, equipment and technical assistance. To describe to what extent the support had been distributed in accordance with the intentions expressed in triennial
plans. To assess the results and impact in relation to stated outputs and objectives, particularly the achievements with regard to reducing the maternal mortality. Results and impact in relation to input (costs) where also to be analysed. 2) In the various documents that have been prepared since 1991, the formulation of objectives for the programme where not identical. The consultants where to describe whether this was a sign that significant
reformulation of the objectives had taken place and, if so, discuss the reasons for it. 3) To describe Sida´s system for dialogue, review and monitoring of the support in relation to the Ministry of Health. 4) To describe the technical assistance component, how it had been defined, designed and utilised over the years and to assess its relevance and effectiveness. Furthermore the consultants where asked to discuss the use of long- and short-term advisers. 5) To describe the system of salary supplementation to midwives, assess its relevance and explore the possibilities of a phasing out of this externally funded supplementation. 6) To explore the issue of sustainability, and assess the possibilities of a future support with regard to the need for an increased ownership and sustainability.
A team of external consultants where hired by Sida to conduct the evaluation in accordance with the quality standards and Sida´s aspiration to conduct a neutral and objective assessment. The 1999 evaluation is based on document analysis and semi-structured interviews with Angolan and Swedish TA staff. Chapter 3 of the evaluation describes Sida´s strategic inputs; salary supplements, communication means, training to motivate staff etc. the issue of
sustainability is discussed in relation to the input analysis. Chapter 4 analyses the impact of the support. Chapter 5 describes the form of support and of Sida´s system for dialogue. The evaluation aims at covering the full range of the support in accordance with the terms of reference. The evaluation relates sustainability to inputs and impact on stated objectives and outcomes. In this sense the evaluation can be said to follow an objectives-based outcome evaluation. Focus is, as in the 1991 study, on impacts and outcomes related to some stated objectives rather than looking at all relevant outcomes.
The 1999 evaluation, despite the concentration of the support, faces the difficulty of trying to estimate the programs merit and worth in relation to stated objectives. The consultants again feel that it is assumptions, rather than an empirical analysis of beneficiary’s needs, that guide the support. They see this as a result of a top-down planning approach. When the consultants are forced to adjust to this reality, subjectivity is allowed entrance into evaluation results. According to theory, such relativism causes the evaluation to diverge rather then converge (Daniel L. Stufflebeam, 2000). Such evaluations run the risk of becoming quasi-evaluations simply because the timeframe and overall planning of the evaluation did not anticipate such a change in approach. This is true primarily if the evaluator is not prepared for the change of circumstances that he/she encounters.
For similar reasons as in the 1991 evaluation, it was difficult to analyse all evaluation questions in the terms of reference. The consultants point out how the following constraints affect the conclusions.
The Lack of information from beneficiaries is once again taken up as a major problem. The evaluation was not planned to include any interviews with the beneficiaries. However, it turned out that very little information of this kind was available, which was a serious limitation when assessing the program. The consultants argue that the lack of such information could be explained by the dominant "expert perspective", and the top-down planning of the programme. Such planning rests on a number of assumptions.
The Lack of relevant information about disbursements. While a lot of the studied documents contain budgets, only one single report included information on the actual disbursements. It was therefore not possible to assess the program and the support from a financial point of view.
Incomplete documents where taken up as a third problem facing the evaluation. Many documents missed dates when they were produced as well as references to other documents. “To refer to other documents was long considered as too academic, and not applicable to Sida-documents. However, without such references it is difficult to conclude how various documents have influenced each other, i.e. to what degree they are integral parts of a dialogue.”
Biased interpretation. The problem of interpreting information based on interviews and documents hampered by language and cultural barriers was not unique for this study. Most information obtained in interviews reflected the position, interest and experiences of the person interviewed. As efforts have been made to get a broad spectrum of opinions this sometimes resulted in contradictory views, which ultimately had to be interpreted and assessed by the consultants. Even if bias was unavoidable, the evaluation team attempted to reduce systematic biases as far as possible.
These constraints caused many of the predetermined evaluation questions to be left
unanswered. The complex circumstances surrounding the support did not allow it. The team of evaluators had to deviate from the original evaluation approach as reliable data was scarce. The achievement of program objectives had to be evaluated through interviews with program stakeholders rather than relying on documentation. Again only views at the central levels where obtained. There is a lack of a reliable health information system in Angola and
consequently the existing statistics are incomplete. For example, when 80% of deliveries are home deliveries, an improved access to delivery clinics can result in an increase in
institutional maternal mortality rates (MMR-i)6 because more complex deliveries come to the
hospitals. Therefore the goal of reducing maternal mortality is not appropriate because it can not be objectively measured.
“The impact of the Sida maternal health programme in Luanda can not be measured by the intuitional Maternal Mortality (MMR-i) statistics… The evaluation of the programme impact on maternal health is therefore dependent of other indicators.”
The Terms of reference of the 1991 evaluation express a concern about a continuous change in program objectives. The consultants find that there is neither reference to the former plan of operation, nor to other preparation documents. Consequently, they found it difficult to trace the rationale behind the modifications of the objectives. Consequently their recommendations had to be based on a number of assumptions.
During the preparation of the 1991 evaluation the Swedish government had stated that a development co-operation aiming at institutional building, partnership and sustainability was
6 Maternal death or maternal mortality, also called “obstetrical death”, is the death of a woman during or shortly
after a pregnancy. When assessing the progress of the maternal health program the primary indicator is the maternal mortality rate, the number of maternal deaths per 100.000 live births.
no longer possible. At the same time the dialogue between Sida and the Angolan counterpart grew weaker.
The consultants argue that as support becomes more humanitarian the dialogue between donor and recipient grows weaker. In such circumstances the evaluation also suffers as participation grows weaker. The objectives-based evaluation is jeopardized as the assumptions made at Sida Stockholm, reflected in the formulation of the evaluation questions (TOR) do not reflect the reality in the field. An interesting observation in the evaluation reflects this: The same objective evaluation questions as in the 1991 evaluation are used, despite the fact that the support have moved from long-term development aid toward more humanitarian aid. This suggests that Sida was unaware of the need for changing evaluation approach.
When answering the question of program sustainability, the evaluators concluded that the program of maternal health is fully dependent on support. The maternal health care program in Luanda was not sustainable.
“The dialogue has grown weaker over the years, and the support has already several characteristics of a project. The Swedish support implies that important investments have been made, both in form of human and physical resources. One justification for continuing the support to the maternal health care programme is that these investments otherwise certainly will get lost.”
The salary supplementations, which are taken up as a separate evaluation question, should gradually be reduced according to the consultants. The evaluation argues that there is no evidence of them having contributed to the program objectives during the period. Salary supplementation continued to be fully dependent on external founding and was not sustainable.
Conclusion
The concluding remarks made in the evaluation, particularly about the effectiveness of salary supplementation, show proof of Sida´s inability to separate assumptions from reality. There is a frustration expressed by the consultants about the assumptions made by Sida that the salary supplements would have made any significant impact during the period from 1995-1999. The Sweden –Angola cooperation went through major structural changes between 1991 and 1999. The support was concentrated to the capital of Luanda, specifically to the health of mothers and children. The overall program objectives did not change since 1991, except the move away of support from the periphery to the capital centre. This change affected the focus of the evaluation, but not much changed in terms of evaluation questions and methods. This change of focus was to a large extent due to the consultant’s ability to adjust to the complex situation.
The 1999 evaluation of the health support to Angola differs from that of 1991 primarily in that only one specific program is the focus of the study. The evaluation become more specified and had to abandon attempts at generalization.
Objectivity is still characterizing the evaluation questions, suggesting that “best answers” are retrievable and subjectivity avoidable. Some new, additional questions are present concerning
the dialogue between Sida and the Angolan counterpart as well as the formulation of objectives.
The independent status of the evaluators adds to the objectivity of the evaluation. They also contribute to adapting the evaluation to the circumstances they encounter in the filed. There are problems when attempting to analyze the quantitative data in order to judge the impact of the program. The lack of accurate demographic- and health statistics lead to more questions than answers. Impact in relation to objectives was difficult to measure.
Despite the fact that few comprehensive evaluations had been done and that the support changed from long-term- to more short-term disaster aid the evaluation is formative rather then summative. The circumstances facing the consultants in the field demand deviation from the original evaluation approach. It is interesting that such deviation is observable again in the 1999 evaluation as it was in the 1991 evaluation. It is obvious that if the consultants would have kept strictly to the evaluation manual a less optimistic picture of the support would have resulted. The external evaluators manage transform the originally summative evaluation into a more effective tool for learning.
The evaluation is external and those who did it stood outside the partner country organization and the supporting donor organization. While initiated by Sida, when inconsistencies in the cooperation are observed both Sida as well as the Angolan counterpart are held responsible.
4.1.3 The 2003 Evaluation
The main purpose of the 2003 evaluation of maternal-health program in Angola was to assess the present program in terms of long-term sustainability. Also the evaluation should bring suggestions to be used in the preparation of the next phase of the support to the health program.
In 2002 a pace accord brought peace to Angola after almost 30 years of civil war. One again possibilities for long-term and an expansion of the programs emerged.
“The evaluation shall specifically analyze possibilities to change the current health programs based on humanitarian aspects into a long-term sustainable cooperation.”
Despite the improved situation in the country, with peace and cease fire, the Swedish government decided to phase out all development aid to Angola in 2003. This decision was taken after the preparation of the terms of reference of the 2003 evaluation and perceived as a paradox amongst the external evaluators hired by Sida.
Focus was put on maternal health, the midwifery school, child health in relation to objectives, expected results and activities. Differing from the 1999 evaluation the 2003 evaluation is aimed at covering the full sector support. The key issues below are related to the evaluation criteria presented in the Sida manual for evaluation.
The following key issues where to be addressed: 1) The achievement of objectives and results as described in project documents, the realism of objectives and the use of indicators. 2)
Improvements/changes in terms of the quality of the provided health care. 3) The impact of the support to the national maternity hospital, Maternidade Lucrécia Paím.4) The impact and efficiency of the national immunisation program, particularly the division of resources between routine and campaign vaccinations. 5) The effectiveness of salary supplements7. 6)
The impact of long-term and short terms advisers in capacity building.
A team of external consultants was hired to conduct the evaluation. The sources of
information for the evaluation where the program co-ordinators and other personnel, project documents, operational plans, quarterly and annual reports and other reports and studies. After preparation and collection of relevant documents from Sida, the main part of the evaluation was conducted in Angola. This included interviews with present actors in the project. The total number of consultancy weeks was expected to be between 8-10. The consultants emphasised that this evaluation of effectiveness and impact of various interventions does not have the profundity and breadths of a scientific report; it was difficult for an evaluation carried out in a few weeks.
The main limitation expressed by the external consultants was that the beneficiaries of the program, the mothers and the children, did not seem to have any voice in the program. Very little information was available about the views and demands of the main target groups. There was much quantitative information in the reports and other documents. The evaluators remind the reader of the fact that this data must be regarded as quite insecure. Similar to previous evaluations the generally stated program objectives are taken up as a problem facing the consultants:
“…plans are elaborated within a reasonably good structure. However, there is often lack of coherence between the objectives at different levels and between expected results and objectives, i.e. it is often difficult to understand how the results will contribute to the
achievement the specific objective… the budget is not complete; government inputs are listed but not calculated… The present annual plans are very extensive… are plans only made to satisfy the requirement of the donor?” (evaluation 03, p 10)
In the 2003 evaluation the consultants again emphasize that, in order to evaluate in accordance with the TOR, objectives should remain the same for many years. Change in objectives should only be done if there is a remarkable change of circumstances and conditions of the project. The consultants found that in the four plans during the studies period the development objectives were changed with every new plan. In addition the
consultants found that in many cases there was insufficient follow-up of stated objectives. For example, in the initial project plan of 1991-1995 the long term objective was to reduce the institutional maternity mortality rate (MMRi) to 400 per 100.000 live births. Whether the program has contributed to the achievement of this development objective is not commented in any report. The evaluation also found that there was a discrepancy between those writing program plans and those who implements the plans: “The lack of realism in calculating expected result is obvious. It seems as if the project writers are not aware of that the project should guarantee that the planned results should be realized; probably this is one reason for the confusion.”
The salary supplements continue to be a controversial issue. The evaluation of the maternal health program carried out in 1999 suggested continuous support, while also suggesting a
7 This question had been treated already in the 1999 evaluation
phasing out of the salary supplements. As recommended, Sida continued the support of the health program, but not all recommendations where implemented. Contrary to suggestions, salary supplements have remained. The 2003 evaluation conclude that the importance of the salary supplementation had been reduced due to inflation and the fact that the same amount was shared by more health workers. The evaluators came to the conclusion that this type of toping-up of salaries should be completely phased out (Sida evaluation, 2003).
Conclusion
The evaluation is objectives-oriented and programme effectiveness is judged in relation to the achievement of stated objectives. Similar to previous evaluations, the purpose of the
evaluation is to give Sida in-depth information for decision about future support. Despite this approach the decision to phase-out all development aid to Angola had already been taken. Just as in 1991, the beneficiaries are left without any voice in the evaluation. The people that are giving their perspectives of the situation are primarily co-workers at the central level. The absence of voices form the beneficiary groups continue to pose great constraints for the full estimation of the program’s merit and worth. The consultants realise that the evaluation in more client-oriented, but in the recommendation they emphasize that such an approach is not sufficient and beneficiaries need to be included.
The external characteristic of the evaluation, despite the evaluator’s independent position, seem to be highly influenced by the objectives orientation of the study. As difficulties emerge in attempts at measuring the real impact of the support, the result seem to be explanations of why this is rather than finding alternatives to assess the impact of the support. The evaluation becomes summative in character and the formative, improvement, decision-orientation of the study gets lost.
The evaluation, as previous ones, conclude that for the maternal health program most women do not attend delivery rooms and that most children are not reached through the present preventive health care. The reasons why are not treated in the evaluation, which can be explained by its objectives-orientated design: the evaluators need primarily to address the questions in TOR and not those emerging during the evaluation process.
4.1.4 The 2006 evaluation
The purpose of the 2006 evaluation was to document the history of 30 years of health-support to Angola, in relation to the Swedish decision to phase-out its bilateral assistance. The
difference between this evaluation and the other three analyzed above is that it covers the whole cooperation period and all related programs, regardless of weather they had ended or where still active. The main purpose of the evaluation was to shine light on advances and setbacks of the cooperation by combining the Angolan and Swedish perspectives. The 2006 evaluation is more a summary of past evaluations and a more descriptive study less concerned with future recommendations.
The following key issues were addressed: 1) In terms of effectiveness, to what extent and in what way had the Swedish development cooperation contributed to improve health in Angola? 2) The analysis should, to the extent possible, be based on existing evaluations,
reviews and studies and what could be learned from such a long cooperation.3) The study should take into account data from both the Swedish and the Angolan side and attempt to produce conclusions relevant to them both. 4) The evaluation should produce a description of results obtained in the health sector in Angola. 5) To Assess the degree to which Swedish support to the health sector in Angola had been effective and contributed to the results achieved.
A team of external consultants was hired to conduct the evaluation. The 2006 evaluation was divided into four parts. The first part was based on Swedish experiences. In the shorter second part the history of the support was looked upon from a seemingly different angle. A brief summary of the Angolan analysis of the different programs, as it was presented in a special evaluation carried out by an Angolan team in 1991, is presented.
The third part of the report is an overview of the supported programs, their development and their present situation after the Swedish support has been phased-out.
Part four presents the consultants conclusions as a basis for a discussion about what Sida as well as the Angolan counterpart institutions can learn from this long-standing cooperation under difficult circumstances.
The 2006 evaluation explain why the Swedish support would, in 1989, be concentrated to improve maternal health. This decision was taken in response to the reported high
(institutional) maternal mortality rate in the country. For this reason Angola expressed interest to include also maternal health in the Programme. This observation, taken from documents from the 1988 agreement, shows how the need of assistance is the primary motive for support rather than the existence of clear program goals and expectations of results.
The 2006 evaluation, similar to the previous ones, express a concern about Sida´s tendency to take unilateral decisions and the lack of dialogue. In the Angolan-Swedish discussions in March 1991 Sida suggested a closer cooperation with the United Nations (UN) for the Primary Health Care (PHC) and water programmes However, according to informants, Sida had already at this time decided to outsource all the health support, either to multilateral organizations or to a Swedish company. The Angolan counterpart was opposing this initiative and wanted to continue bilateral cooperation with Sweden. This shows how Sida decided about how the future support would look like before having conducted any thorough assessment on this question and without dialogue with the cooperation partner.
In the 2006 report there was an ambition to include all stakeholders, including the Angolans. However, it was not easy to collect specific information from this big group of interviewed people. For many of the Angolans, the Sida experience was a long way back in time and their memories were not always exact since they had moved between different posts and
programmes in the Health Sector. The same was true for the Swedes.
Conclusion
