Lost in comparison: a program theory analysis of performance measurement in social work

http://www.diva-portal.org

This is the published version of a paper published in Nordic Social Work Research.

Citation for the original published paper (version of record):

Denvall, V., Agevall Gross, L., Kjellgren, C., Skillmark, M. (2021)

Lost in comparison: a program theory analysis of performance measurement in social work

Nordic Social Work Research, 11(3): 249-263 https://doi.org/10.1080/2156857X.2019.1690559

Access to the published version may require subscription.

N.B. When citing this work, cite the original published paper.

Permanent link to this version:

http://urn.kb.se/resolve?urn=urn:nbn:se:lnu:diva-94711

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=rnsw20

ISSN: 2156-857X (Print) 2156-8588 (Online) Journal homepage: https://www.tandfonline.com/loi/rnsw20

Lost in comparison: a program theory analysis of performance measurement in social work

Verner Denvall, Lotta Agevall Gross, Cecilia Kjellgren & Mikael Skillmark

To cite this article: Verner Denvall, Lotta Agevall Gross, Cecilia Kjellgren & Mikael Skillmark (2019): Lost in comparison: a program theory analysis of performance measurement in social work, Nordic Social Work Research, DOI: 10.1080/2156857X.2019.1690559

To link to this article: https://doi.org/10.1080/2156857X.2019.1690559

© 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

Published online: 14 Nov 2019.

Submit your article to this journal

Article views: 46

View related articles

View Crossmark data

Lost in comparison: a program theory analysis of performance measurement in social work

Verner Denvall

, Lotta Agevall Gross

, Cecilia Kjellgren

and Mikael Skillmark

a

School of social work, Lund university, Lund, Sweden;

Department of social work, Linnaeus university, Växjö, Sweden

ABSTRACT

Since the 1990s, social work has been subject to requirements for mon- itoring and quality assurance. Here we examine one of the monitoring systems: Open Comparisons (OC). Gradually expanded and published online by a national agency, the OC now has around 350 indicators that cover major areas within social work in Sweden. We use program theory to clarify the operational idea in which OC is based. To do this, we analyse domestic violence data gathered from two social service organizations, from the regional level and from governmental agencies. The results show a strong normative support for OC within national level. However, OC is time consuming, its data are questionable, and its reliance on name-and- shame seems dubious. OC represents a trend in social work that may in fluence changes in routines and provide new kinds of performance measurements that a ffect how social work is organized.

KEYWORDS

Audit; domestic violence;

performance measurement;

comparison; social work

The use of instruments to measure and compare actions is a dominant trend in welfare states. To support e fficiency, transparency, and equity, government organizations monitor and measure e ffects and work processes. Hence, these instruments need indicators that provide information on the quality of actions being performed (Pollitt 2013; Van Dooren, De Caluwe, and Lonti 2012).

Such indicators are based on data from sources such as o fficial statistics in annual reports or from targeted surveys. These reports and surveys may be produced by researchers, skilled administrators, and more recently social workers. Indicators (yardstick comparisons) were introduced around 1830 by Jeremy Bentham as a corrective mechanism to optimize production (Clarkson 2010). In the 1950s, indicators started to be used to measure welfare. During the last 20 years, indicators have gained renewed signi ficance as part of New Public Management (NPM) and as part of the audit society in general, in fluencing how public administration operates. In modern strategies of govern- ance, the use of indicators falls under concepts such as performance measurement, outcomes-based management, and results-based management.

The concept Open Comparisons (OC) is one such monitoring activity used in Sweden. OC consists of a collection of indicators developed in collaboration with state agencies and the association of local authorities (SALAR; SKL in Swedish). In 2006, health care began to use OC, soon followed by social work. To support national coordination and e fficiency, the Swedish government has decided to strengthen the quality of the work performed through extensive monitoring (SOU 2015). Gradually expanded, OC now includes around 350 indicators that cover several areas within social work such as addiction, elderly care, homelessness, and domestic violence. Swedish municipal social services annually respond to surveys about their activities sent to them by the National Board of Health and Welfare (NBHW; Socialstyrelsen),

CONTACT Verner Denvall Verner.Denvall@soch.lu.se https://doi.org/10.1080/2156857X.2019.1690559

© 2019 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.

This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://

creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the

original work is properly cited, and is not altered, transformed, or built upon in any way.

a national agency under the Ministry of Health and Social A ffairs (Socialdepartementet). The data are then compiled and published online. Using OC, a municipality can compare its performance with other municipalities. In addition, OC is intended to improve governmental transparency, monitoring, and support development, although international research indicates that there is limited knowledge about the e ffectiveness of OC (Pollitt 2013; Power 1999). Several similar indicator systems exist in Europe and the USA (Balle Hansen 2016; Pollitt and Bouckaert 2011).

This article aims to explore what happens when this monitoring is applied in social work, speci fically in the field of domestic violence. Domestic violence is a complex social problem with many professional actors having partly interconnected and partly contradictory understandings of its problems and solutions. The measurement of dynamic issues is a signi ficant challenge. If data are not compiled properly, organizations might act on inadequate information. In recent years, research on performance measurement has grown and there is a good general idea of its opportunities and challenges. In particular, research has shown that gaming (bending the rules) and cheating (breaking the rules) are inevitable features of monitoring and evaluation strategies, so further analyses of the mechanisms and contexts of these strategies is needed (Pollitt 2013, 359).

Using program theory, this article analyses how OC is applied in social work and we examine and problematize underlying assumptions in OC. We assume that the norms codi fied in writing as goals and agreements reveal only part of the picture. The ideal rational legal form of bureaucracy does not fit with the way organizations really act. Organizations depend on prominent ideas within their environments and catch up with organizational trends that propose how an organization is supposed to be. This approach is openly presented as a prescriptive, formal theory of change that guides organizations ’ actions – i.e. what ought to be done and why. Beyond these apparent activities, we might find invisible structures, ambiguous goals, and relations of dependence. These unac- knowledged aspects may be deeply embedded in everyday activities and represent an organization ’s factual, informal theory of change, re flecting the way things are done rather than the way things should be done (Chen 1990; DiMaggio and Powell 1983; Donaldson 2007; Perrow 1986). This con flict between what is done and what should be done may exist as multiple program theories among stakeholders. These ideas might be explicit as well as implicit, requiring evaluators and researchers to tease out meaningful understandings (Balle Hansen and Vedung 2010). Three research questions will be addressed:

Q1. What is the formal theory of change in OC?

Q2. What is the informal theory of change in OC?

Q3. How does practice in social work handle OC?

The article is derived from a study about the design of interventions that address domestic violence. The research examines how di fferent forms of governance – i.e. non-profit organizations, government agencies, and local social work – affect interventions using evidence and NPM (Agevall Gross, Denvall, Kjellgren and Skillmark. 2015; Skillmark, Agevall Gross, Kjellgren and Denvall 2019). Here we will o ffer a critical view of performance measurement and question the relevance of OC in social work. Based on policy documents and interviews, we present the overall visions and program theory of OC. We report how actors at the national level (the National Board of Health and Welfare and SALAR), regional level (County Administrative Board), and local level (two munici- palities in southern Sweden) use OC in an attempt to improve the way crime victims are treated.

Because no clients have been contacted, there was no need to apply for ethical approval.

The indicator movement and performance measurement

At least three intertwined factors are encouraging the use of OC: the quest for social indicators, the NPM, and the audit society. In the postwar period, international organizations such as the UN and the OECD designed speci fic indicators to measure welfare as a way to complement the more economic dimensions of GDP. Components such as health, education, security, and the environ- ment were supposed to support socially-oriented urban planning. Indicators, especially in the EU, now serve as measurements of sustainability, poverty, and well-being. Indicators are considered necessary as they make it possible to overview and concisely summarize complex multiple welfare problems (Socialstyrelsen 2010, 131). Indicators may provide essential information about the social conditions in a community and the impact of social policy. OC represents a major investment in the use of indicators although they no longer measure the living conditions. That is, OC measures the quality of actions or what outcomes these actions produce. Today, they resemble the early Bentham idea of a dipstick to check employees more than the later UN idea of welfare measurement.

NPM and Total Quality Management (TQM) have received broad political and administrative support and encourage results-based public service (Arnaboldi, Lapsley, and Steccolini 2015; Balle Hansen 2016; De Bruijn 2007; Evetts 2009; Pollitt 2006). Like other organizations, social work is subject to extensive quality assurance systems and various evaluation instruments. Originally, quality assurance e fforts were started by the private sector and have been described as a reaction to primitive, rationally-based goal performance. Instead, soft values were highlighted and the emphasis was on receiving feedback from employees and documenting processes (Wandersman et al. 2000). Today, quality assurance is connected with NPM and the new control system that changes the way work is managed (Hasselbladh, Bejerot, and Gustafsson 2008).

There is a wide-spread belief that it is bene ficial to monitor results and processes. In the audit society, organizations invest considerable resources to measure their activities and to obtain data to visualize processes (Power 1999). This commitment to auditing has signi ficant consequences as audit systems require organizations to design their internal structures for auditing, a requirement that inevitably depends on experts. Audit systems require continuous analysis of new data if the goal is to improve and optimize outcomes. Therefore, such information should be of good quality and give managers tools to improve governance. Critics claim too much focus is put on e fficiency, cost reduction, quantitative performance, and governance models as this limited focus de-emphasizes the complexity of social problems (Arnaboldi, Lapsley, and Steccolini 2015), yet this ‘governing by numbers ’ has been suggested as a new way of governance where audit technology and rating systems are core mechanisms of change (Shore and Wright 2015). Acting as a ‘performance – enhancing drug ’ management by numbers might support as well as weaken performance (Hood 2012). Regardless, such ideas might serve as mechanisms of hope when searching for the rational organization (Brunsson 2006).

Indicators and social welfare

The quest for quanti fication and results supports the growth of metrics such as indicators, statistics, organizing ratings, and ranking systems. They support national goals of performance that may be based on data from national agencies, often statistics, presented as key performance indicators (kpi) (Balle Hansen 2016). However, it is challenging to build skills that implement and support performance measurement as extensive organizational investments are required (Hunter and Nielsen 2013; Pollitt 2013). Di fferent logics for executing activities in organizations compete for space, which a ffects the quality of the data used (Høybye-Mortensen 2016). Poor input or lack of input may even create a false basis for prioritization (De Bruijn 2007; Pollitt and Bouckaert 2011;

Groth Andersen and Denvall 2017). Other problems arise: synecdoche (one indicator stands for the

whole performance); threshold e ffects (motivation below target, de-motivation, performing above

target); ratchet e ffects (next year’s targets are based on last year’s performance); definitional drift

(bending the categories to favour the organization); escalation (in time measured activities will be favoured and grow); performance paradox (the longer a set of indicators is continued, the more skilful gaming and cheating occurs); symbolic use (information is gathered for show); perverse e ffects (undermines morale and behaviour among staff); and discipline effects on employees (steering and correction) (Arnaboldi, Lapsley, and Steccolini 2015; Clarkson 2010, 176; Hood 2012; Pollitt 2013, 352 –354; Sauder and Espeland 2009).

The more complex the information system, the more di fficult it is to produce reliable data (Høybye-Mortensen 2015; Shaw et al. 2009). Transparency and information are expected to help users and patients make decisions about the services they wish to receive. However, aspects that are easily quanti fied receive attention, whereas aspects that are not easily quantified (e.g. care work, prevention, and long-term outcome) may be ignored (Lindgren 2014). Researchers as well as professionals themselves, especially medical professionals, have criticized the over-emphasis on quantifying performance as this type of analysis tends to ignore the role of professional competence (Hasselbladh, Bejerot, and Gustafsson 2008). The constitutive consequences seem to be signi ficant;

organizations prepare to be assessed and measured as auditees which may a ffect them as providers of crucial services (Dahler-Larsen 2011; Power 1999; Shore and Wright 2015).

Open comparisons

The indicators in OC are divided into outcome indicators (what the actions have resulted in), process indicators (what is actually done), and structural indicators (the conditions of the processes and results). These dimensions were inspired by Donabedian’s influential article on quality assurance in medical care (Berwick and Fox 2016; Donabedian 1966) and were suggested to be used in social work by a commission in 2007 (Socialstyrelsen 2007). Since then, indicators have been developed and in 2015 a national strategy was established. OC is defined (Socialdepartementet 2015) as follows:

[A] set of indicators that are reported annually and that re flect the activities of quality, performance and e fficiency. OC is a part of the national knowledge management. (Socialdepartementet 2015, 5)

Moreover, the indicators in OC ‘should be supported by research or, in its absence, proven experience. An indicator will also give direction to a desired result’ (Socialstyrelsen 2014, 12). Six quality areas are common in the indicators presented within social services and health services:

knowledge-based, secure, personalized, efficient, equitable, and accessible. Since 2011, social ser- vices units within the 290 Swedish municipalities have been asked to respond to annual surveys concerning domestic violence. In February 2016, they responded to 98 questions (about 110 pages) plus follow-up questions (11 about violence), in total a response rate of 91%. The results from the survey are presented as 350 indicators on a website where municipalities are ranked and their results are reported in tables coloured green (25% good performance), yellow (50% average performance), and red (25% bad performance). In the area of domestic violence, OC has been published on five occasions, between 2012 and 2016, and 62 indicators are expected to constitute the tools to monitor, analyse, and develop social service activities with the support to victims of violence at the local, regional, and national level. Many of the questions focus on whether the municipality has written management-established procedures (routines) for monitoring or action. ‘Routine’

refers to a specific approach for the activities to be carried out (Socialstyrelsen 2016).

Indicators are supposed to support the establishment of evidence-based practice in social work

and to stimulate the development of good quality. Several of the indicators included in the OC are

based on methods presumed to benefit clients/patients and where government agencies want to

know the extent to which they are used. Government funding to develop the quality of social

services and non-profit organizations working with domestic violence is now allocated according to

whether these organizations work with methods based on scientific evidence and their responses

from OC.

Indicators of OC have been researched as a monitoring technology in a limited way both nationally and internationally. This research mainly highlights comparisons within the health care system (Blomgren and Waks 2011) or performance indicators (Bevan and Hood 2006;

Clarkson 2010). According to Bevan and Hood, OC focuses on what can be measured rather than on what ought to be measured. This problem stems from the fact that indicators understate the complexity of organizations with multiple stakeholders and goals (Arnaboldi, Lapsley, and Steccolini 2015). Swedish research in elderly care has shown that the use of OC is time- consuming and that the ‘name-and-shame mechanism’ might increase compliance and thereby support management control (Lindgren, Ottosson, and Salas 2012, 47; Hanberger and Lindgren 2018). No unit wants to be ashamed of its performance (i.e. be shamed into improving). The unwillingness to be associated with poor performance seems to be a central mechanism in its potential as an instrument of change, so it contributes to a ‘new mindset’ in health care. Research with a focus on social services for the elderly shows that the instrument tends to be most useful for managers and those responsible for quality issues, whereas users and even the profession ’s view of quality fades into the background (Johansson and Lindgren 2013). To be e ffective, using OC to determine program e ffectiveness requires clear goals, improved data quality, and enhanced coop- eration between actors; if not, what is measured gets done and what is not measured gets ignored (Vårdanalys 2012).

Material and analyses

This study derives its data from a multiple case study of two municipalities working with domestic violence with specialized units of social workers responsible for the support of abused women and their children. These municipalities had distinguished themselves for being driving forces in the development of social work for abused women, yet they were organized in different ways and used different instruments. Like most other municipalities, they participated in the annual data collection about OC. We have reconstructed the program theory of OC and related it to practice by interviewing key stakeholders and studying relevant documents.

We conducted 16 interviews with 19 participants (Table 1).

The interviewees were selected based on their different roles and perspectives in relation to the handling of OC. The interviewees are both top officials at various levels in the municipalities and regional support agencies – i.e. managers, social welfare committee chairman, and various frontline staff, including specialized functions such as women’s coordinator and development manager. At the national level, we interviewed representatives from SALAR and the National Board. The

Table 1. Interviewees.

Municipality 1 Top o fficial

Manager – administrator Manager – social worker Top politician

Three social workers (group)

Municipality 2 Top o fficial

Manager – administrator Social worker

Top politician

Regional level Manager

Manager – administrator 1 Manager – administrator 2

SALAR Manager

National Board of Health and Welfare Leading commissioner Manager – researcher Manager

Manager

N 19

interviewees addressed their professional or political role in their work with domestic violence and OC. Interviews were recorded and transcribed by the authors. Documents were collected and analysed from the national and regional agencies as well as the municipalities.

Our analytical tool, program theory, focuses on ‘ . . . what must be done to achieve the desired goals, what other important impacts may also be anticipated and how these goals and impacts would be generated ’ (Chen 1990, 43). Program theory tells us to look for the structures of the idea and its logical starting points. Then the researcher should trace its actual use and implementation.

By reconstructing and analysing the theory of change within the program, we have targeted underlying assumptions and core logics of OC (Rogers and Weiss 2007; Vedung 2011; Weiss 1997). We have worked with three distinct analytical concepts: (A) the situation and the way problems are de fined (the theory of problems); (B) the normative elements (ideas about how to conduct OC in terms of investment in personal, methods, and desired outcome); and (C) causality (why things are supposed to happen the way they do) (Chen 1990; Donaldson 2007). These concepts will be used when we present our findings. In our interviews, we asked the respondents to explain OC, how they responded to the national surveys, their views of the advantages as well as challenges of OC, and how they regarded the outcomes of OC.

Themes relevant in the analysis included the following areas: Who are the key players? What do they want to achieve? Why and how do they achieve these goals? How can the emergence and spread of indicators be explained? Data from interviews and o fficial documents illustrate how the idea has been understood, received, and used by stakeholders (politicians, managers, and social workers) who are a part of treating domestic violence. The analysis is built on a multi-vocal and pluralistic program theory in which di fferent stakeholders’ perceptions are considered (see Balle Hansen and Vedung 2010).

Results: the program theories of OC

Following is the analytical concepts A-C where we move from the o fficial accounts and the factual theory to the informal theories of change underlying OC. A summary where the three research questions are answered concludes the results part.

The theory of problems

When establishing indicators, professional representatives and researchers took part in working groups initiated by NBHW. Our respondents found it difficult to develop indicators for domestic violence. Many questions and indicators were considered; ideally there should not be too many and they should assess quality. Stakeholders had different motives. Below is a quotation from a representative from SALAR taking part in the development of OC.

I think we captured many good and important areas but it is hard to ask questions so they really show the actual quality. I think that one could use it very pragmatically. That you look at the results you have and get, and use the results for development purposes. [. . .] not to name and shame any municipalities, that ’s not the point, but maybe you look at the di fferent results in the county and compare with the nation and that sort of thing.

Which problems should these indicators deal with then? They are not easily discovered in docu-

ments from the national level. Instead, it is mentioned in general terms that better knowledge is

needed about activities in social welfare and their outcomes. OC in health care was presented as an

ideal by the investigating commission (Socialstyrelsen 2007, 6). Early on, this sector was subject to

the new management ideals of NPM that were imported and translated from the business sector to

create greater efficiency (Erlingsdottir 1995). When planning the period of 2014–2015, the problems

in health care were described as the growing population of elderly, new diseases, and concentration

on health centres instead of treatment at hospitals (Regeringskansliet 2014). No such description

exists within social services, but later OC was described as an important tool for the implementation of improvements (Socialdepartementet 2015). Unlike health care, social services lacked su fficient registries and structures for documentation. These tools needed to be processed and national agencies have played an active role in creating a national information structure in social work (Socialstyrelsen 2007).

How then is this regarded at the local level? Opinions di ffer among the respondents. Several declare that they believe in OC and that it can serve as a ‘wake-up call’ and give the opportunity to compare their actions with others. One director points out that it is an instrument that makes it possible to measure their practice and outcome: ‘You get confirmation that there needs to be a change ’.

But it can also be a means to draw attention to shortcomings, perhaps as an aid in the search for resources:

So I hope that there are questions about initiating investigations and things like that. It is the requirement to open an investigation even in case of an adult. (social worker)

This social worker wants support so her management can make it possible to spend more time on needs assessment for all adults who need support. Now they need to prioritize those victims who need protected housing. Other motives may be follow-ups of regulations and support to create organizational structures. They might increase the security of the individual and support the information collected. Indicators that are assessed centrally are important and favour documentation:

A high level of assessment is important for cooperation and for the work to be professional and raise quality.

(manager in a municipality)

OC also faces criticism:

I think they are too focused on what documents they [NBHW] have and if they have followed up last year and so on [. . .] you put more e ffort into answering the survey than you then do analysing the results [. . .] we’ll find a scapegoat all the time. (manager in a municipality)

Other voices mention that the OC can strengthen the profession, that it will be a welcome aid in the evaluation of its actions, and that measurement is necessary. The problems the OC might solve emerge indirectly in the respondents’ stories: weak professional social workers with limited knowl- edge of the actions need better evaluation, documentation, and measurements to demonstrate the needs of their clients and where OC might provide support.

To sum up, not much is said in documents or interviews about the problems OC are supposed to solve. Instead it seems as if OC can be a promising audit instrument that still needs to be used and developed. OC offers national agencies more effective governance, transparency, and over- view. Locally hopes are raised that indicators could strengthen documentation and argumentation about resources. Thus OC appears as a political instrument that provides a national agency the opportunity to influence social work and social workers the opportunity to argue for expanded resources.

The normative elements

Although the requirements in recent years have been formalized, normative government of social services is increasing. Together with the design of OC, the National Board has also produced several documents to support the interpretation of what constitutes quality in victim support work.

Manuals, general advice, and various forms of evidence and supporting documents have been

distributed. The indicators demonstrate the desirable quality of activities to support evaluation and

improvement. It is also supposed to ‘measure and monitor quality in order to clarify the quality of

the various stakeholders’ (Socialstyrelsen 2007, 8).

A lot of information needs to be gathered that can be measured and monitored and thus used for comparisons, both in relation to earlier documentation and in relation to other activities, locally, regionally, or nationally. The results of the quality assessment should then be visible to stakeholders, including managers, politicians, users, and citizens who are supposed to be able to have access to

‘good information about the activities and the results’ (Socialstyrelsen 2007). The National Board further emphasizes the importance of transparency and national comparisons and highlights the opportunities for follow-up and development. It seems that the social service user indirectly bene fits.

How then should the OC be implemented and what are the results so far? The interviewed politicians have vague statements about OC.

I: OC, is it something you know about? Is it in use?

P: I should know, but I can ’t answer that. But I can find out. (political chairman)

One of the managers in charge said that unmerited focus on documentation not only provides the board information that politicians exploit somewhat arbitrarily and the media report often sensationally but also stresses the municipalities ’ resources.

The respondents find it challenging to gather the required materials. Moreover, they believe the information gathered is interpreted personally. Everyone responded based on his or her specific work, so the responses were not originally planned:

So, no, it ’s delicate with OC, it’s really hard to get the issues so they can’t be misunderstood or can’t be interpreted in more ways than one. [. . .] There are ten or eleven of us [responding to the survey], but views and opinions are tossed around in the air and somebody sits and writing at the computer. (social worker who participated in the national team)

Municipalities need to organize the answers, many people need to be involved, and it is not always the right person contributing data according to their field of expertise. One of the managers is quite critical about how they had to answer ‘No’ (marked in red on the webpage) to a question about routines even though they had an established system implemented along with a researcher they were satisfied with: ‘The response was rated red since it was not included in what the National Board considered OK’. This quote shows how the national agency tries to encourage the use of approved methods through OC.

Neither of the two municipalities uses the material regularly: ‘We take a brief look but it is of no importance’. There are many other tools and comparisons that are more specific and useful locally.

Although some managers have hopes for better control and support resources, some professionals are equally frustrated that OC is not being used consistently:

We want it to be green, we don ’t want it to be red [laughs]. [. . .] I would like to see an analysis on the management level of this, open comparison is valid not just here in domestic violence without regard to the other areas in social services, we should take a comprehensive approach when looking through it in the analysis, but it has not been done. (social worker)

When the indicators hit the local level, the top-down control seems obvious:

There are very many indicators, and that they are to a large extent of a control character [. . .] this stigma of shame should be so strong, you should avoid bad results, and that is the e ffect of this control. (manager at a regional council who supports municipalities in analysing OC)

According to a representative from the national agency, both control and development were goals:

But it is clear that if you look at what is the purpose of public performance of the government, it was both control and development. I think that the municipalities also felt that it was an inspection. Sharp result indicators as in health care were something to long for. Unequal social services look di fferent in different municipalities. And you want everyone to be above a certain minimum.

Summing up: Rankings in OC trigger municipalities to organize new task assignments, and to

compare their performance with other municipalities.

C. The causality

Causality is not expressed. OC was assumed to be an innovative mechanism for development but without scienti fic evidence of proof. Instead, OC rewards fast data for publication and comparisons to create a sense of ‘name and shame’. The latter reflection comes from one of the prime researchers at the National Board of Health and Welfare a couple of years after the introduction of OC.

In an instructive film on OC, the National Board of Health and Welfare explains how to use the tool: ‘It’s very positive and indicates the location to get together, to work together. It will be a win- win for both users, patients, and for the principals ’ (author’s translation). Stakeholders are seen as consumers and are assumed to gain from increased quality of operations, so they are cast as passive rather than active actors.

What cause-and-e ffect phenomena do respondents identify in the municipalities? OC is not used locally in the monitoring, there are internal doubts as to how it should be interpreted, and its validity is perceived as low. To be useful, OC needs to be included in the discussions between professionals and managers. These are the experiences of a manager at the regional level who tries to support opportunities for meetings through the exchange of experiences in networking, research, and dialogue in the form of thematic conferences both when completing questionnaires and when analysing results.

Two respondents from a regional administrative board who implement the government ’s action plan against domestic violence describe the OC as ‘unpredictable’ and ‘totally insane’, so they do not use OC:

The numbers they come up with say nothing, a disaster, because that depends on who they have sent it to. The municipalities that work the least were those who had the highest best results [. . .] so you cross a bit because you do not know and you think you do not know what it means either. (manager 1)

In particular, the criteria the questions build on are di fficult to interpret, and they require proper preparatory work:

It is not quite easy to compare within our area because there are always di fferent criteria compared to [. . .] very crazy criteria too, di fficult, you can interpret them in many different ways. (manager 2)

The o fficial ranking of municipalities using the colours green, yellow, and red could be sensatio- nalized by the media, resulting in a lack of focus on the most important aspects of analysis and discussion:

So at the national level they have enormous expectations that the supposed stigma of shame should be so strong, you don ’t want to have bad results, this is steering. (manager at the regional level)

Other stakeholders see the use of OC as a ‘flagship’ for the national agency:

[The national agency] seems to be very proud of OC; it looks great. And it should be interesting to really know if it is that good. [. . .] I felt quite critical [. . .] there should be other ways; it costs an awful lot of money this whole thing. (manager at the national level)

Concluding: No casual chains show the enhancement of combating domestic violence. This study corroborates other research on governing by numbers that demonstrate threshold e ffects, defini- tional drift and data distortion.

Summary

Q1: The formal theory of change.

We perceive the overall chain of governance presented at the national level as follows:

(1) Problems with domestic violence exist in society; (2) The collection of information about the

problem through OC is possible; (3) OC provides an improved basis for decisions and improvement

of quality; (4) Changes in routines and in methods are implemented; (5) Enhanced support for women will be the outcome. This prescriptive program theory resulted in the following flow chart (Figure1).

According to the formal program-theory of OC, indicators address the lack of overview, weak structures, and absence of impacts in social work. Local organizations will compare themselves with other organizations in the hope that this competition will enhance continuous improvement. The use of open ranking should improve the quality of the support o ffered to abused women.

Q2. The factual theory of change

Our analysis of the data gathered about the program theory for OC in domestic violence problematizes the o fficial picture. The audit society and NPM provide new opportunities for governance. In complex societies, monitoring results is a way to identify where traditional bureau- cracies fail in their governance and how they can improve their performance. Speci fically, govern- ments can improve their performance by using indicators to ensure the following: (1) Obtaining a richer picture of the results and processes in social services; (2) Providing better information about the work to combat domestic violence; (3) Establishing a ranking system so poorly perform- ing municipalities can identify where they need to improve; (4) Gathering information that policy- makers can use to evaluate what procedures are in place; (5) Implementing strategies that are data driven to improve social services; and (6) Ensuring government programs are designed using standardized features (Figure2).

Open comparisons ability to improve interventions aimed for abused victims is weak. OC supports overall monitoring of social work, it provides government agencies with information and its rankings are supposed to motivate organizations to improve their performance.

Q3. How does practice in social work handle OC?

The picture of how respondents within social services handle OC varies according to position.

Top-level managers and politicians support the overall idea of performance measurement, but at the same time see no immediate bene fit with OC within their own organization. Middle-level managers criticize OC based on its governing e ffects: creating additional work, risking red markings for routines they themselves consider OK, also risking being the subject of media attention.

Frontline workers perceive OC as an opportunity to get resources for the area of domestic violence and support for their professional work. The interviewed social workers are well aware that data validity in OC is unreliable. The latter critical attitude is shared by our respondents from national,

Figure 1. The formal program theory of OC at the national level.

Figure 2. The factual program theory of OC at the national level.

regional and local level. No data supports that OC improves the quality of interventions and thus conditions for abused women.

Discussion

Since the 1990s, the ideas of new public management have caused extensive changes in the public sector. These changes have also affected how social work is performed as it has been been subject to various forms of monitoring, evaluation, and quality assurance. This research has examined one of the monitoring systems applied, Open Comparisons (OC); however, OC has been implemented without support from research. Nonetheless, strong normative support for OC exists within governmental agencies. OC provides national agencies with local data and is a tool for implement- ing EBP. Unlike present understandings, our analysis of the data indicates that local performance has not been affected to any significant extent by using OC even though a comprehensive admin- istration has been created where employees of municipalities are supposed to collect data, register information, and analyse the results generated by the OC.

This study shows that the appearance of OC could be understood in terms of governance manifested in modes of control and the implementation of tools. Another driving force is the belief that indicators can be used to improve and develop social programs. The material does not discuss the supposed relation between the problems of victimized women and how domestic violence should be solved using indicators. However, the quality of performance in social work is in question. National agencies use these indicators to compare the effectiveness of social programs provided by municipalities, a strategy that is presumed to specifically improve the overall quality of the services provided to victims of domestic violence. However, no distinct casual chains are presented. Instead mechanisms are recognizable from other research on performance measurement where naming and shaming and ranking systems are introduced to support quality. The impetus for OC is to be found at the national level of public government. At the local level, social workers view OC as an added burden, at the same time as they express hope of OC as a tool for improvement.

National agencies have dual roles. On the one hand, an agency is supposed to implement and develop the use of OC. On the other hand, the data are used in other parts of the agency’s support regarding domestic violence despite the poor quality of data. Information from the OC is used to provide financial assistance to municipalities. Only when certain procedures are in place will the municipality receive some support to develop methods. Our study shows that the municipalities themselves are well aware that their answers are not always accurate.

This research has identified OC as a normative technique that enhances organizational self- regulation. In addition, OC is assumed to provide benchmarking, continuous quality improvement, the limitation of discretion, and standardized work performance. Performance measurement indicates the OC results in increased management and new methods of control. However, indica- tors do not stand alone; they need to be interpreted, which calls for networking to analyse the results. OC promoters seem, however, to be unable to convince stakeholders that it is an effective tool for managing information and knowledge. That is, knowledge management is realized through constitutive formation. OC, as a modern tool, influences how social work is performed by relying on competition as a mechanism of change (i.e. name and shame), and this strategy seems unavoidable.

Those who do not take part are met with suspicion. Staying outside OC might be an option, but who dares as there might be consequences. Our respondents noted that the media can place undue pressure on municipalities that perform ‘poorly’ by publicizing comparisons.

OC might be a driving force for municipalities that come out poorly in the ranking, encouraging them to invest in services aimed at supporting victimized women. A ranking system could reward municipalities that perform well and punish those that perform badly, but no data support this widely-believed assumption (Hertting and Vedung 2009, 38; Sauder and Espeland 2009). Ranking systems are influential and change the preconditions for organizations that must react.

Organizations are exposed and under external pressure. According to the theory, this pressure

will push poorly performing municipalities to implement routines and methods using strong scienti fic support. That is, OC is believed to be a driving force of governance.

Indeed, OC provides national agencies with data and has succeeded according to the program theory in implementing a new routine in the social services. It is now on the local agenda. This research has shown that surveys and variations in the registration of data indicate validity is low, leaving responders with incorrect information. The data presented on the website are unreliable according to respondents in the two municipalities. OC, however, does not solve the information needs of national agencies since the data provided might be unreliable.

Social work is complex and indicators are unable to show more than partial pictures. The picture in this version of OC does not consist of outcome data or data directly from users of social services.

That is, knowledge about impact is not provided. OC delivers data about routines, processes, and to some extent structural information. Donabedian ’s original idea from 1966 about outcome data has been put aside; instead, OC sends messages about the importance of administration and routines.

OC requires human and financial resources. Professionals working with clients exposed to domestic violence are often already burdened with a heavy workload, but they feel a need to prioritize the surveys so their municipality will not be perceived as performing badly as competition and benchmarking are the sources of evaluation. OC does not provide insight and explanations and does not provide answers to questions about the outcome of social work. Instead of organizing proper evaluations, municipalities must put up with unreliable data gathered from the OC.

Furthermore, OC concentrates on the appearance of procedures within the organization. These procedures do not guarantee that clients receive e ffective services. Indicators of quality can be valuable, but their usefulness requires careful professional judgements.

The limitations of this study rest in its empirical design. As this study relies on only two cases, there is a need for careful generalization; we do not know whether the results would include other areas of social work or even health care. Therefore, the analysis had to focus on theoretical generalization. The research presented here provides a snapshot and the circumstances may have changed since the study began.

Conclusion

We conclude that the results published in OC about interventions in social work in the field of domestic violence are built-in without proof of causality. Implications for the national level include improving the quality of data gathered for OC. Indeed, this will take resources, but if the overall goal is the use of data for governance, there is no other way. Otherwise, the national level might end up with endless discussions using ‘alternative facts’. At the local level, providing resources for valid and real evaluations would improve the services provided to clients. That is, municipalities are now using their limited resources to complete national surveys they do not use themselves. The original goal to use indicators to collect knowledge about people ’s living conditions has now been reduced to a matter of accountability.

OC represents a trend in social work that re flects the global influence of performance measure- ment. It changes the modes of governance from direct governance to governance by numbers.

Today, organizations demand the proof of the implemented processes and e ffects of their inter- ventions. OC changes routines and provides standardized information that a ffect how social work is organized. Our results show that OC can be used as a political instrument for controlling social work and to argue for expanded resources and in politics, validity and correct documentation are of minor importance. Finally, does the use of OC improve the lives of women who are victims of domestic violence? These victims deserve a credible answer.

Disclosure statement

No potential con flict of interest was reported by the authors.

Funding

The Swedish Crime Victim Compensation and Support Authority, grant [04049/2013].

ORCID

Verner Denvall http://orcid.org/0000-0002-2655-2132 Cecilia Kjellgren http://orcid.org/0000-0002-4416-1223

References

Agevall Gross, L., V. Denvall, C. Kjellgren, and M. Skillmark. 2015. “Brottsoffer I Indikatorland: Öppna Jämförelser Inom Socialtjänstens Brottso fferstödjande Arbete.” Socialvetenskaplig Tidskrift 22: 341–358.

Arnaboldi, M., I. Lapsley, and I. Steccolini. 2015. “Performance Management in the Public Sector: The Ultimate Challenge. ” Financial Accountability & Management 31 (1): 1–22. doi: 10.1111/faam.2015.31.issue-1.

Balle Hansen, M. 2016. “Performance Managemant Og Evaluering.” In Evaluering Af Offentlig Politik Og Administration, edited by T. Bredgard, 115 –143. Copenhagen: Hans Reitzels Forlag.

Balle Hansen, M., and E. Vedung. 2010. “Theory-based Stakeholder Evaluation.” American Journal of Evaluation 31 (3): 295 –313. doi: 10.1177/1098214010366174.

Berwick, D., and D. M. Fox. 2016. ““Evaluating the Quality of Medical care”: Donabedian’s Classic Article 50 Years Later. ” The Milbank Quarterly 94 (2): 237–241. doi: 10.1111/1468-0009.12189.

Bevan, G., and C. Hood. 2006. “Whats´s Measured Is What Matters: Targets and Gaming in the English Public Helath Care System. ” Public Administration 84 (3): 517–538. doi: 10.1111/j.1467-9299.2006.00600.x.

Blomgren, M., and C. Waks. 2011. “Öppna jämförelser inom hälso-och sjukvården-en utmaning mot den professio- nella byråkratin? [Open Comparisons within Health Care – A Challenge to Professional bureaucracy?].”

Arbetsmarknad & Arbetsliv 17 (4): 95 –108.

Brunsson, N. 2006. Mechanisms of Hope: Maintaining the Dream of the Rational Organization. Malmö: Liber and Copenhagen Business School Pr.

Chen, H. T. 1990. Theory-driven Evaluations. New York: Sage.

Clarkson, P. 2010. “Performance Measurement in Adult Social Care: Looking Backwards and Forwards.” British Journal of Social Work 40 (1): 170 –187. doi: 10.1093/bjsw/bcn096.

Dahler-Larsen, P. 2011. The Evaluation Society. Stanford: Stanford University Press.

De Bruijn, H. 2007. Managing Performance in the Public Sector. New York: Routledge.

DiMaggio, P. J., and W. W. Powell. 1983. “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizatinal Fields. ” American Sociological Review 48 (2): 147–160. doi: 10.2307/2095101.

Donabedian, A. 1966. “Evaluating the Quality of Medical Care.” The Milbank Memorial Fund Quarterly 44 (3):

166 –206. doi: 10.2307/3348969.

Donaldson, S. I. 2007. Program Theory-driven Evaluation Science: Strategies and Applications. New York: Lawrence Erlbaum Associates.

Erlingsdottir, G. 1995. Förförande idéer –kvalitetssäkring i hälso-och sjukvården [Seductive Ideas – Quality-assurance within Health care]. Lund: Lund university.

Evetts, J. 2009. “New Professionalism and New Public Management: Changes, Continuities and Consequences.”

Comparative Sociology 8 (2): 247 –266. doi: 10.1163/156913309X421655.

Groth Andersen, S., and V. Denvall. 2017. “Data Recording in Performance Managemant – Trouble with the Logics.”

American Journal of Evaluation 38 (2): 190 –204. doi: 10.1177/1098214016681510.

Hanberger, A., and L. Lindgren. 2018. “Evaluation Systems in Local Eldercare Governance.” Journal of Social Work 19: 1 –20.

Hasselbladh, H., E. Bejerot, and R. Å. Gustafsson. 2008. Bortom New Public Management-Institutionell transforma- tion i svensk sjukvård [Beyond New Public Management – Institutional Transformation within Swedish Health care]. Lund: Academia Adacta.

Hertting, N., and E. Vedung. 2009. Den utvärderingstäta politiken: Styrning och utvärdering i svensk storstadspolitik [The Evaluation Tight Politics – Governance and Evaluation in the Swedish Metropolitan politics]. Lund:

Studentlitteratur.

Hood, C. 2012. “Public Management by Numbers as a Performance-Enhancing Drug: Two Hypotheses.” Public Administration Review 72: S85 –S92. doi: 10.1111/j.1540-6210.2012.02634.x.

Høybye-Mortensen, M. 2015. “Decision-making Tools and Their Influence on Caseworkers’ Room for Discretion.”

British Journal of Social Work 45: 600 –615. doi: 10.1093/bjsw/bct144.

Høybye-Mortensen, M. 2016. “Performance Information in Human Service Organizations: Quality and Usefulness.”

Human Service Organizations: Management, Leadership & Governance 40 (5): 486 –499.

Hunter, D., and S. Nielsen. 2013. “Performance Management and Evaluation: Exploring Complementarities.” New Directions for Evaluation, no. 137: 7 –17. doi: 10.1002/ev.20042.

Johansson, V., and L. Lindgren. 2013. Uppdrag o ffentlig granskning [The Mission of Public audit]. Lund:

Studentlitteratur.

Lindgren, L. 2014. Nya utvärderingsmonstret. Om kvalitetsmätning i den o ffentliga sektorn [The New Evaluation Monster. About Quality Measurement in the Public sector]. Lund: Studentlitteratur.

Lindgren, L., M. Ottosson, and O. Salas. 2012. Öppna Jämförelser. Ett styrmedel i tiden eller “Hur kunde det bli så här?”

[Open Comparisons. Governance of Our Time or “How Could It End up like this?”]. Göteborg: FoU i Väst.

Perrow, C. 1986. Complex Organizations: A Criticial Essay. Third ed. New York: Random House.

Pollitt, C. 2006. “Performance Management in Practice: A Comparative Study of Executive Agencies.” Journal of Public Administration Research and Theory 16 (1): 25 –44. doi: 10.1093/jopart/mui045.

Pollitt, C. 2013. “The Logics of Performance Management.” Evaluation 19 (4): 346–363. doi: 10.1177/

1356389013505040.

Pollitt, C., and G. Bouckaert. 2011. Public Management Reform: A Comparative Analysis - New Public Management, Governance and the Neo-weberian State. Oxford: Oxford University Press.

Power, M. 1999. The Audit Society: Rituals of Veri fication. Oxford: Oxford Univ. Press.

Regeringskansliet. 2014. Öppna jämförelser inom hälso- Och sjukvården – Handlingsplan för 2014–2015 [Open Comparisons in Health Care - Action Plan during 2014 –2015]. Stockholm: Regeringskansliet.

Rogers, P. J., and C. H. Weiss. 2007. “Theory-based Evaluation: Reflections Ten Years On: Theory-based Evaluation:

Past, Present, and Future. ” New Directions for Evaluation 2007 (114): 63–81. doi: 10.1002/ev.225.

Sauder, M., and W. N. Espeland. 2009. “The Discipline of Rankings: Tight Coupling and Organizational Change.”

American Sociological Review 74 (1): 63 –82. doi: 10.1177/000312240907400104.

Shaw, I., M. Bell, I. Sinclair, P. Sloper, W. Mitchell, P. Dyson, and J. Ra fferty. 2009. “An Exemplary Scheme? an Evaluation of the Integrated Children ’s System.” British Journal of Social Work 39 (4): 613–626. doi: 10.1093/bjsw/

bcp040.

Shore, C., and S. Wright. 2015. “Governing by Numbers: Audit Culture, Rankings and the New World Order.” Social Anthropology 23 (1): 22 –28. doi: 10.1111/1469-8676.12098.

Skillmark, M., L. Agevall Gross, C. Kjellgren, and V. Denvall. 2019. “The Pursuit of Standardization in Domestic Violence Social Work: A Multiple Case Study of How the Idea of Using Risk Assessment Tools Is Manifested and Processed in the Swedish Social Services. ” Qualitative Social Work: Research and Practice18 (3), 458–474.

doi:10.1177/1473325017739461.

Socialdepartementet. 2015. Öppna jämförelser inom socialtjänsten och hemsjukvården. Handlingsplan 2015 –2018 [Open Comparisons within Social Service and Home Care. Action Plan 2015 –2018]. Stockholm:

Socialdepartemenet.

Socialstyrelsen. 2007. Modell för utveckling av kvalitetsindikatorer i socialtjänsten [A Model for Developing Quality- indicators within the Social services]. Stockholm: Socialstyrelsen.

Socialstyrelsen. 2010. Social rapport [Social Report]. Stockholm: Socialstyrelsen.

Socialstyrelsen. 2014. Öppna jämförelser 2014. Stöd till brottso ffer. Nationella resultat, metod och indikatorer [Open Comparisons 2014. Support to Crime Victims. National Results, Methods and indicators]. Stockholm:

Socialstyrelsen.

Socialstyrelsen. 2016. Öppna jämförelser 2016. Våld i nära relationer. Guide för att tolka resultaten [Open Comparisons 2016. Domestic Violence. Guide to Interpret the results]. Stockholm: Socialstyrelsen.

SOU. 2015. Nationell strategi mot mäns våld mot kvinnor och hedersrelaterat våld och förtryck [The National Strategy against Domestic Violence and Honour-related Violence and oppression], 55. Vol. SOU 2015. Stockholm: Fritzes.

Van Dooren, W., C. De Caluwe, and Z. Lonti. 2012. “How to Measure Public Administration Performance:

A Conceptual Model with Applications for Budgeting, Human Resources Management, and Open Government. ” Public Performance & Management Review 35 (3): 489–508. doi: 10.2753/PMR1530-9576350306.

Vårdanalys. 2012. Grönt ljus för Öppna jämförelser? Vårdanalys utvärdering av öppna jämförelser inom hälso- och sjukvården. Rapport 2012:4 [Green Light for Open Comparisons? Vårdanalys Evaluation of Open Comparisons within the Health Care. Report 2012:4]. Stockholm: Myndigheten för Vårdanalys. doi:10.1094/PDIS-11-11-0999- PDN.

Vedung, E. 2011. “Spridning, användning och implementering av utvärdering [Diffusion, Use and Implementation of evaluation]. ” In Utvärdering i socialt arbete: Utgångspunkter, modeller och användning [Evaluation in Social Wirk;

Starting-points, Models and use], edited by B. Blom, L. Nygren, and S. Morén, 285 –299. Stockholm: Natur &

kultur.

Wandersman, A., P. Imm, M. Chinman, and S. Kaftarian. 2000. “Getting to Outcomes: A Results-based Approach to Accountability. ” Evaluation and Program Planning 23: 389–395. doi: 10.1016/S0149-7189(00)00028-8.

Weiss, C. 1997. “How Can Theory-based Evaluation Make Greater Headway?” Evaluation Review 21 (4): 501–524.

doi:10.1177/0193841X9702100405.