• No results found

Design as sociomaterial navigation: a performative framework for action-oriented design

N/A
N/A
Protected

Academic year: 2022

Share "Design as sociomaterial navigation: a performative framework for action-oriented design"

Copied!
189
0
0

Loading.... (view fulltext now)

Full text

(1)

Design as

Sociopolitical

Navigation

(2)

both of these fi elds, in giving due emphasis to the power that designers have in shaping the technologies that users will use, have tended to underplay the larger organizational context in which design work itself is undertaken. In contrast, this study highlights the organizational accountabilities inherent to design projects.

Design work is conducted within a fi eld of relations between various parties, and designers’

work is accountable to each of these participants with their different stakes, backgrounds, authorities, and/or organizational responsibilities, which often come with contradictory per- spectives, agendas, ways of working, and values. Design as sociopolitical navigation refers to the ways in which designers organize activities in relation to these various actors through- out the design process: navigating socially and politically charged environs is part and parcel of design work.

While PD has provided designers with a host of techniques and practices for including users

in the design of technologically-supported practices, it has not bequeathed to designers a

framework for organizing and analyzing design activities as they unfold in real time. This

thesis draws on dramaturgy and performance theory to create a framework for planning and

analyzing design events that make central the format of the activity, how the people and

materials are prepared to participate, and how their participation prepares them for future

activities.

(3)

Design as Sociopolitical Navigation

A Performative Framework for Action-Oriented Design

Brendon Clark PhD Dissertation

Mads Clausen Institute for Product Innovation

University of Southern Denmark

(4)

Submitted for partial completion of Ph.D degree at the University of Southern Denmark.

 Brendon Clark Mads Clausen Institute Sønderborg, Denmark 2007

Ph.D Supervisor:

Jacob Buur

Assessment Committee:

Thomas Binder Ilpo Koskinen Pelle Ehn

Dissertation on record at Denmark’s Royal Library.

ISBN 978-87-991686-2-0

Cover designed by Kim Holm (www.designastronaut.dk). Front cover

sketch by Niklas Andersson and back cover sketch by Brendon Clark.

(5)

ACKNOWLEDGEMENTS

The work presented here is inseparable from the personal and professional relationships I have been lucky enough to develop over the last several years. I owe special thanks to Mette Kjærsgaard for luring me back to Denmark in the first place and for introducing me to the possibility of a design anthropology. Since then (2002), Jacob Buur has continually provided me with the resources and professional support to pursue my sporadic initiatives. He has also demonstrated to me a conscientious approach to organizing multidisciplinary research activities that has forever changed my view of what it means to engage with others.

Jeanette Blomberg’s valuable critique just before I embarked upon this four-year journey was a needed wake-up call for me that has helped me ever since. To this I attribute a process of depersonalizing my work, while striving toward developing a rigorous research practice.

I am indebted to Ben Matthews for his ceaseless willingness to articulate the details of good research and to find value in mine (or to “mine” mine for value). He and the rest of my colleagues at the Mads Clausen Institute have provided a lively practice environment for which I am deeply grateful:

Karen Anderson, Jette Tofte Iversen, Mads Vedel Jensen, Nico Daams, Werner Sperschneider, Kirsten Bagger, Wendy Gunn, Chris Heape, Marcelle Stienstra, Tom Djajadiningrat, Jared Donovan, Birgit Fabius, Kyle Kilbourn, Larisa Sitorus, Mette Mark Larsen, Rune Christensen, and guests such as Thomas Binder and Daniela Sangiorgi. Also the students of the IT Product Design program (and students from the Interaction Design program in Umeå) who kept me on my toes by challenging me to make sense of my material. And, my colleagues at the Umeå Institute of Design who welcomed me with open arms, especially Niklas Andersson & Mike Stott for taking a chance on me. In Umeå I learned a great deal from the diverse participants of the CUPL project and my project teammates.

Joachim Halse & Jens Pedersen are the core of my PhD student cohort. My

work would not be as exciting or valuable without mixing with them the

personal & professional, theoretical & practical, leisurely & frantic. None of

this would be the same without my amazing wife Diana. Meeting her was

one of the “preliminary results” of my PhD study (we met after my first

year), and clearly the best result. Among everything else, Diana has the

unique ability to make what I say and do feel so very valuable.

(6)
(7)

ABSTRACT

Participatory Design (PD) and the emerging field of Design Anthropology have each demonstrated the essentiality of the involvement of users in the design process and designers' understanding of users' work for the success of the technology under development. However, both of these fields, in giving due emphasis to the power that designers have in shaping the technologies that users will use, have tended to underplay the larger organizational context in which design work itself is undertaken. In

contrast, this study highlights the organizational accountabilities inherent to design projects. Design work is conducted within a field of relations between various parties, and designers' work is accountable to each of these participants with their different stakes, backgrounds, authorities, and/or organizational responsibilities, which often come with contradictory perspectives, agendas, ways of working, and values.

Design as sociopolitical navigation refers to the ways in which designers organize activities in relation to these various actors throughout the design process:

navigating socially and politically charged environs is part and parcel of design work. While PD has provided designers with a host of techniques and practices for including users in the design of technologically-supported practices, it has not bequeathed to designers a framework for organizing and analyzing design activities as they unfold in real time. This thesis draws on dramaturgy and performance theory to create a framework for planning and analyzing design events that make central the format of the activity, how the people and materials are prepared to participate, and how their participation prepares them for future activities.

The performative framework is developed through analyzing a design case titled

Design for Cultural Pluralism (CUPL). The Swedish government supports

efforts to improve integration of children from non-Swedish descent into

Swedish society. CUPL was a nine-month project sponsored by a local

municipality aimed at exploring issues surrounding cultural integration and

developing design solutions to support practices in that area. From the

inception of the project until its end, the design team organized and

participated in a range of activities within their own organization (design

institute), the sponsoring organization (municipality), and the project field

(i.e. schools, libraries, sports clubs). By analyzing a series of activities that

led to the financing of the two proposals of the project, the thesis

(8)

demonstrates the value of the performative framework for both analyzing and planning design activities.

Goffman’s work on the theatrics of everyday behavior, and Turner and Schechner’s work in performance studies, provide a theoretical foundation for analyzing design activities as performances illuminating the use of space, prepared materials, the format of the participation, and the impact of an activity. This broadens the PD focus from who may be involved in an individual activity and the immediate outcome of the activity, to include: (a) how and with whom participants develop their roles and what resources they draw upon to do so; (b) how their roles vary in relation to when, where, and with whom an activity unfolds and how the structure of an activity is created and maintained; and (c) how participants interpret an activity afterward with other people and how they draw upon it as a resource for participation in subsequent activities. At the same time, the thesis argues that the performative framework’s attention to the fluidity of participant roles from one activity to another, the structure and

interrelationship of activities, and the process of assessing how activities give value to their participants, provides a resource for organizing design processes.

Strategic corridor, performance map, and performative task are three concepts developed in the dissertation for planning and assessing action-oriented design activities: A performance map is a tool for analyzing the various levels of interaction among project stakeholders (i.e. design team members, funding decision-makers, intended beneficiaries) based on who organizes and attends an activity and its function in relation to other activities.

Activities at the strategic corridor level are organized to provide project stakeholders a format for collaboratively exploring new possibilities that inform and define the content and structure of subsequent activities.

Performative task is an organizational unit for planning group activities that manifest in shared performances. It can refer to tasks such as an exercise given to participants during a workshop, or on a broader scale such as design team members working toward a presentation or workshop. In addition to introducing a framework for analyzing and proactively

organizing design work and activities throughout a project, this dissertation

opens a (historically) neglected aspect of design work in research to analytic

scrutiny: the negotiation of project resources with stakeholders.

(9)
(10)
(11)

TABLE OF CONTENTS

ACKNOWLEDGEMENTS_____________________________________ i ABSTRACT ______________________________________________iii

PREFACE

A design research exhibition _____________________________ xi

CHAPTER I

User-oriented design and the Scandinavian Approach ________1 The Scandinavian Tradition of Participatory Design___________ 7 Xerox PARC’s contribution to Participatory Design ____________ 8 Expanding the user-oriented design focus ___________________15 Including decision makers in design _________________________17

CHAPTER II

A social perspective on organizational relations ____________23 The study of political processes____________________________25 UMEÅ CASE___________________________________________32 Design institute and municipality partnership ________________34

CHAPTER III

A performative analysis of design meetings ________________41

Developmental sequence of performance_____________________44

Goffman’s dramaturgy of impression management _____________46

Design institute and municipality partnership ________________49

Frontstage: Presentation through a dramaturgical lens ___________52

Performance is “restored behavior” ________________________57

Liminality in Schechner’s workshop-rehearsal _________________57

Backstage: Making the steering committee meeting performance ___62

(12)

Rehearsal: Irene visits the design institute_____________________71 Assessment: steering committee meeting revisited ______________76 Aftermath and post-liminal identity ________________________79

“Steering committee meeting” aftermath _____________________79

CHAPTER IV

Action-oriented design ethnography ______________________85 The integration of ethnography and design __________________86 Ethnographically informed Tangible Analysis Kit ____________ 100 Creating action-oriented representations ____________________102 Format for tangible analysis ______________________________112 Workshop as performative task ___________________________115 Action-oriented representations for tangible analysis ___________116

CHAPTER V

A performative framework for action-oriented design _____ 121 Performance map of social interaction ______________________122 Strategic corridor: Creating liminal spaces ___________________129 Three versions of the rehearsal/performance relationship _______133 Revisiting the social drama in the design partnership ___________136

CHAPTER VI

Design as sociopolitical navigation _______________________ 141 Negotiation process as viable research domain ______________ 142 Participatory design and ethnography as resources for action __ 143 Performative framework for action-oriented design___________ 144 I. Performative framework for action-oriented design __________144 II. Strategic corridor level of interaction _____________________147 III. Performative task ___________________________________148

REFERENCES ___________________________________________ 151

APPENDIX ______________________________________________ 161

(13)
(14)
(15)

PREFACE

A design research exhibition

We arrive at a Swedish city of 110,000 inhabitants of whom 30,000 attend the university. We join this year’s version of the university’s design school exhibition where the bachelor and masters students proudly display their greatest achievements. We find car interiors and exteriors displayed on screens and in models, household products such as an ergonomic golf bag, a sailboat, a vacuum cleaner, mobile phone, and beautiful sketches and renderings of process and product. Each display, whether in a glass case or free standing, is accompanied by a poster or video screen, and name tag.

This year’s exhibit is extra special for a variety of reasons. Most of all, it is special for those in attendance because it is “this year’s” year-end

extravaganza. Opening night has its share of glitches. As 9:00 pm roles around, some students continue to add pieces of glass to their display cases, clear away cardboard or peel away the last pieces of tape from a model hoping that the paint is dry. At the same time, they are fighting the

swarming crowds of spectators with their wine glasses and plates of cheese.

The buzz in the air is palpable as parents, students, local residents, professors, and the media mill around three floors of the design institute.

On the second floor, in the far corner, there is a different type of display

this year. The institute’s design researchers have a presence among the

student projects. Similar to the other exhibits, there are screens, and

posters, and prototypes with arrays of colors. Yes, the posters are different

sizes from the others, but there is also a patch of grass exemplifying an

indoor patio area and there are extra posters that are not necessarily

attached to individual projects. Some people walk into the room taking a

quick glance around and quickly leave; others stay and read through a

comparatively large amount of text, approach a prototype and listen to an

explanation of how it works while the researcher at the side of the table

attempts to get her to please sit down and “Give it a try! Say something

into the microphone!”

(16)

Figure 1. Opening night of the Umeå Institute of Design’s annual exhibition

Those who do take the time to read the posters, watch the videos and participate in an introduction to the prototypes, will learn that the room is the visual display of a nine-month design research project. The Design for Cultural Pluralism (CUPL) project is the design institute and the local municipality’s joint effort to design products and services to improve cultural integration of children in Sweden. The gist of the issues guiding the project, are: “Why do children of non-Swedish descent score lower on standardized tests in Sweden?” “How can design contribute to improving this?” They also read that the designer researchers are involved in

ethnographic research and participatory design with an emphasis upon user involvement in the design process. For some, the reaction to both this process and the room’s exhibition can be caught in the comment “Too many words!”

CUPL has three interrelated projects: a design research project resulting in

three concepts (two prototypes) for supporting mother tongue teaching

and learning (conducted by two interaction designers, an engineer and an

anthropologist); an MA degree project to support free-time interaction with

a gender focus; and four concepts to support teacher, parent, student

meetings in the school environment resulting in the Interaction Design

(17)

students’ semester Design Ethnography and Participatory Design course (See main CUPL posters in appendix).

Fig. 2. Mother tongue teaching practice in Umeå, Sweden

Children in Sweden are entitled to language instruction in their mother tongue if at least one of their parent’s first language is not Swedish. This has created the professional practice of mother tongue teaching. In Umeå forty-five teachers representing twenty-five languages travel throughout Umeå giving language instruction to 1,200 students. The teachers travel to as many as four schools a day giving forty-five to sixty minute lessons in each school. Teachers have the responsibility to build their lesson plans, find relevant textbooks and materials, and shape curricula under

extraordinary teaching conditions. As a result, mother tongue teachers have developed a highly customized teaching process based largely on

improvisation. The Soundbites concepts allow teachers and their pupils to develop their own audio content, making audio tangible for use in and out of the mother tongue classroom.

Fig. 3. Author demonstrating Bitesize Fig. 4. Teacher production process

(18)

Tonight, the two main authors of the CUPL project, an interaction designer and myself, are quite pleased with the number of people trying the

prototypes, their interest, their actions and reactions. We are delighted that there were mother language teachers, the school principle, our main municipality sponsor, some mother tongue teachers, students, and others in attendance. However, over the following two-weeks regardless of the positive responses we receive from various stakeholders such as the municipality sponsors and various guests, we speculate about why more of the forty-five mother tongue teachers did not attend the exhibit. After all, this was a presentation of their practice, and of new interesting products and services that promise to enhance their work experiences. We sent out invitations and personally manned the exhibit room during opening hours.

What does it take to get more people involved in the excitement of our project? The exhibit was a “new and exciting” part of our work practice.

The project was changing the practice of the design school, but the event did not necessarily play the same role for mother tongue teachers. Even if the ideas and outcomes are valuable for them, and were born out of participation in their teaching activities and workshops, and their students’

projects, that does not mean the majority of mother tongue teachers attend such events as a design exhibition, or would feel comfortable in such a place merely because of their practice being represented.

Figure 5. Presenting to project stakeholder. Figure 6. Behind the scenes just weeks before.

Throughout the project, there has been a similar effect as the people

attending the exhibit had when entering the CUPL room. They do not look

at the grass on the floor in the exhibit room and automatically imagine that

it is just like the school children’s outdoor environments. People do not

just look at the prototype and think, “What a great idea!” They may not get

it by reading a poster or two. But, for those who took the time and

(19)

experienced a combination of reading a poster, hearing an explanation and trying a prototype for themselves, their was a transformation into

excitement upon discovering something they appreciate. “Wow! What a great project!” But these voices compete for attention from the project team with others (“too many words”). As the picture in figure 5 suggests, there is a lot more behind the scenes of such an exhibit than the cohesive impression it may portray. What cannot be seen is often even more interesting than what is clearly visible.

I find a great deal about the design exhibit and the CUPL project worthy of sharing. There are so many stories to be told (created). We can learn about the mother tongue teachers involved in the project and the dramas within the design team. There are the student projects and the impact of having to exhibit designs. There is the design research regarding the process that makes them.

In relation to design, I find it intriguing that the two posters about mother tongue teaching practice, and the poster about the five-week project we held with 9

th

grade students, did not precede the design phase of the prototypes. Rather, they took place concurrently throughout the design process. In this same room, where we held many of the design activities throughout the project, we all frantically worked for weeks leading up to the big exhibition. The posters and prototypes took their final shape just as the 9

th

Grade Somali students finished their personal DVD projects (see 9

th

grade project poster in appendix). Each is as much an explanation of the project as any other.

For me, the CUPL project symbolizes the year I spent in Umeå and the numerous experiences and impressions during that time. I would like to be able to tell the stories about how my experience in Umeå provided me with a stark contrast of what design “is” from what I held previously, and how such a contrast sheds new light on the previous years I spent with my colleagues at the Mads Clausen Institute. But that is a different story.

I am quite intrigued though, how having a specific deadline to complete

one’s work and the anticipation of putting it on display to an audience

triggers new ways of interacting with people and materials. I find that such

a drive toward a deadline can be both disturbing and exciting bringing out

the best and worst in us. Now that I have started with the end, I will

unravel one of the stories of how the CUPL project came to be and the

discoveries it has provoked.

(20)
(21)

CHAPTER I

User-oriented design and the Scandinavian Approach

The title Design as Sociopolitical Navigation refers to the ways in which designers organize their own activities in relation to the various actors in the design process. Specifically, I am interested in the type of designers and design researchers who strive toward inclusive design practices with a strong user orientation. In recent years, practitioners with such an orientation fell under such categories as User-Centered Design, Human- Centered Design, Participatory Design, Collaborative Design, Design Anthropology. But they also include Human Computer Interaction, Systems Design, Computer Supported Cooperative Work, and Interaction Design. Projects in these areas are predominately conducted by project teams and often involve crossing a number of organizational and disciplinary boundaries. Whether conducted for industry, for a public initiative, for a non-profit organization, for academic community, or any combination of these, a design project requires its practitioners to participate in a host of social encounters with people from a variety of backgrounds. Often this includes not one-on-one encounters, but encounters with greater numbers of people.

The way that design processes are conceived by those organizing and driving projects, influences what design practitioners emphasize and deemphasize; the issues they find valuable. The techniques and practices practitioners employ to account for what can and cannot be influenced throughout the span of a project, directly influence what can be

“accomplished” in a project. Ultimately however, regardless of the

orientation of the design practitioners and the disciplines represented in a

project team, they must communicate about their work with each other and

with people outside of their project team. This dissertation stems from the

assumption that design with a humanistic/inclusive orientation cannot be

reduced to an artifact, report, or set of specifications and handed to

(22)

developers in an organization. Instead it is dependant upon practitioners’

abilities to understand and organize activities that demonstrate their value to the multiple actors throughout the design process. I argue that a performative framework for organizing design activities provides design practitioners resources for developing materials and activities that support engagements with decision makers and stakeholders necessary for initiating and maintaining the allocation of resources in a user-oriented design project. It is my contention that exploring the high-stakes often involved in designer / decision-maker encounters will also shed light on the value and limitations of action-oriented approaches to participants in the use context, such as potential users and their organizational gatekeepers. To make this case, I draw on empirical work I conducted over eleven months as a guest researcher at the Umeå Institute of Design in Sweden.

Design involves moving from how things are now, to how things could be and should be in the future (Schön 1983 [1991]; Simon 1969; Suchman et al. 1999). Design research projects seek to move from understanding current practice through empirical studies (including active engagement) and theory as a basis for moving from what is known to imagining future practices, often with the introduction of new artifacts to support the development of new practices, or new artifacts that are inseparable from that new practice. Those who identify themselves as “designers,” or playing a part in “design,” are situated in a variety of organizational arrangements and/or work in relation to different organizational frameworks. They occupy a variety of roles and have biographies spanning a range of disciplinary trainings and practical experiences.

The work of design is difficult to connect to a specific location or activity.

However, over the span of a design project, design teams engage in a wide variety of activities and events with people who at different times for different reasons are part of the design team or engage with the team.

These can include colleagues, potential users, stakeholders, sponsors and

others interested in the project’s potential influence, and even people such

as the mail delivery employee, or the print shop technician. Similar to other

practices, design activities include design meetings with those inside and

outside of the host organization, face-to-face meetings with stakeholders,

conference calls, internal and external presentations, and more specifically

to design, learning about the future use domains by means of exploring

secondary data, observing ongoing practices, conducting workshops, user

tests, and a range of other related activities.

(23)

Practitioners working in the Scandinavian tradition of Participatory Design (PD), also referred to as the “Scandinavian Approach” and “collaborative design,”

1

have generated a host of action-oriented techniques and practices for the collaborative design of new technologies (Bjerknes et al. 1987; Ehn 1988; Greenbaum and Kyng 1991; Schuler and Namioka 1993; Suchman et al. 1999). The basic premise for collaborative design to produce something new that supports skilled use practice depends upon three main criteria: 1) designers learn about the users and the use context of the technologies they design; 2) users learn from designers about new technological possibilities and how participate in the process of designing, and; 3) for designers and users to engage in activities together to collaboratively design new technologically supported practices.

The Scandinavian approach’s focus on bringing all three criteria together in activities to support mutual learning allows practitioners from a wide variety of disciplines (i.e. anthropology, philosophy, sociology, psychology, industrial design, theater, and dance) to contribute their techniques for achievement supporting these goals. Some techniques bring all three criteria together within one activity such as a full-day workshop with users and designers, while others are employed over time and favor one or two of the criteria over the other(s). These techniques have been developed in

multidisciplinary teams that often encourage each practitioner to bring the value of her discipline into action during cooperative activities.

At the same time, the transition from one approach to another raises a host of issues in relation to managing the design process. For example, industrial designers and system designers have long used low-fidelity prototypes to demonstrate to users their ideas and to solicit feedback. In the

Scandinavian tradition, however, designers have developed collaborative prototyping techniques that involve users in the development of prototypes and use scenarios to imagine future possibilities with the prototype (Bødker and Grønbæk 1991). There is an important distinction between, on the one hand, introducing a prototype to someone and asking them to try it, or think about it, and to describe or demonstrate their opinion, and on the other hand giving someone prototype materials and exploring with them, in their use context, possible future scenarios of use. At the same time, shifting from “demonstrating” to “cooperating” relies upon aligning a host

1

I will use these terms interchangeably throughout the dissertation. People also refer to

User-Centered Design in the Scandinavian tradition.

(24)

of circumstances to bring together the users and designers in a meaningful activity at the appropriate time during the design process.

Bødker and Grønbæk suggest that, “some organizational problems must necessarily be handled to establish a basis for performing cooperative prototyping in a specific project” (1991: 209). In the subsequent six pages of their contribution, the authors describe the type of work necessary to overcome these problems. They fall under the headings: “establishing project groups;” “setting up prototyping;” ”providing prototypes;”

“maintaining communications;” “the user perception of the process;” and

“getting resources” (1991: 209-216). I make the argument in this dissertation that in order for the value of action-oriented approaches to suffice as a resource for action among practitioners in diverse fields, they must overcome the challenges of illuminating and demonstrating the differences between important distinctions, such as demonstration and collaboration, within the duration of a project

2

in a way that also includes the

“handling of organizational problems.” I find that to ignore the details of working out interactional issues within design projects, face-to-face encounters with project teammates, partners, stakeholders and decision makers are predominantly limited to taken-for-granted formats that favor verbal argumentation and short, one-way presentations supported by Power-Point and/or standard text documents such as research reports.

While the legacy of the Scandinavian approach provides design practitioners valuable resources for developing techniques for staging mutual learning activities with designers and users, and increasingly with stakeholders, the value of collaborative design techniques and practices has not been explored with the same vigor in relation to the formats of activities with decision makers who determine the financial and human resources of design projects, e.g., determining the duration, participants, methods and deliverables of a project.

I believe that the Scandinavian approach’s history as a Marxist, workplace democracy approach, with the alignment with worker unions against management, and the specific focus of direct user involvement, has:

overshadowed the value of collaborative techniques and practices for engaging participants throughout design projects; obscured the organizational work necessary for collaborative activities; and, left user-

2

Schön (1983 [1991]: 62) refers to the “action-present” as the time that something can be

influenced regardless of the duration. The action-present is clearly much different for

many practitioners, especially in relation to practitioners who work exclusively in industry

versus exclusively in academia.

(25)

oriented design practitioners in need of a conceptual framework for organizing, conducting, and assessing collaborative activities in today’s organizational contexts.

I include, as worth giving greater analytic attention, decision-maker interactions as a vital aspect of the design process. Furthermore, I

demonstrate how the Scandinavian approach and anthropological ethnography offer techniques uniquely suited for engaging participants in new

organizational formats for standard meetings. I argue that drawing on dramaturgical and performance, for the organization and assessment of design activities, can demonstrate the value of action-oriented techniques for engaging project partners and stakeholders in the creation and maintenance of design projects, in addition to their value in the PD tradition of engaging users in context. Through this frame, designers intervene in the unfolding practices of skilled users and invite them to participate in the unfolding skilled practice of design, but also their design projects play a role in the unfolding practice of the project partners and stakeholders.

I propose that viewing each encounter between the design team and project participants through a performance framework, design practitioners are guided in developing materials and supportive formats for activities that not only inform the design of better products, systems and services, but that create and maintain the human and financial resources necessary for user-oriented designs. At the same time, creating structure involves

developing a working language and shared techniques with multidisciplinary teammates and colleagues. In summary, there are three main challenges facing user-oriented design practitioners that I explore in this dissertation:



Performing user-oriented design requires working out an array of organizational issues that mediate what is possible in a project.

Procuring resources from decision makers is a key aspect of performing user-oriented design.



While the Scandinavian tradition of Participatory Design has left

practitioners (in academia and industry alike) with unique action-

oriented techniques and practices for engaging multiple actors in

the design process, it has left practitioners without a conceptual

framework for illuminating the value of an action-oriented

approach to important aspects of design.

(26)

I seek to contribute to user-oriented design by drawing on sociological, anthropological, Participatory Design, and performance theory to develop a conceptual framework for organizing and assessing design activities. I develop a “performative framework” that includes concepts for addressing the processual nature of action-oriented techniques and practices, and practical techniques for staging and organizing activities with multiple participants through a design project.

In the remainder of this chapter, I will continue to set the foundation of my contribution by reviewing the main issues in Participatory Design and the value of action-oriented design techniques. After a brief review of the Scandinavian approach, I identify the need for concepts and techniques for design practitioners to include a broader range of social interactions, including those with decision makers, in the purview of malleable activity formats.

In chapter II, A social perspective on organizational issues, I focus upon

establishing my position as an action-oriented design researcher conducting empirical work while participating in a design project. I introduce a focus upon process, change and public expression drawing on political

anthropology. Victor Turner’s “social drama” is a unit of analysis valuable for exploring the relationship between the Umeå Institute of Design and the Umeå Municipality’s service design partnership.

In chapter III, A performative analysis of design meetings, I demonstrate the value of a performative framework as a lens for analyzing design activities. I rely upon a series of dramaturgical and performance concepts to work my way through a detailed analysis of the meetings among design team members, their colleagues, and the gatekeepers of financial resources during project funding negotiations.

Chapter IV, Action-oriented design ethnography, is a demonstration of how I have drawn upon the techniques and practices from the Scandinavian approach and anthropological ethnography as resources for staging action- oriented activities with multiple participants. I develop the concept of performative task as a way of organizing activities with a specific action- orientation.

In chapter V, A performative framework for action-oriented design, I develop a

structural view of the design partnership between Umeå Institute of Design

(27)

and Umeå Municipality, and illuminate the interrelationship of activities that made possible the funding of the CUPL design research project.

And finally, in Chapter VI, Design as sociopolitical navigation, I summarize my main contributions of the dissertation asserting the importance and value of an analytic and prescriptive framework for organizing, conducting and assessing action-oriented design processes that include encounters with decision makers and users alike.

The Scandinavian Tradition of Participatory Design

The Scandinavian tradition of Participatory Design originated out of the workplace democracy movement as exemplified by the UTOPIA project

3

(Bødker et al. 1987; Ehn 1988). The PD researchers developed the Collective Resource Approach (CRA) founded on the basis that to be effective, design efforts must be conducted locally in collaboration with worker unions and should adopt the agenda of the union. The tool perspective in CRA is rooted in the idea that design solutions should be tools that enhance the ability of a practitioner to perform and develop his craft. To achieve this, those craftsmen must participate in the development of the tools they will use. Ehn argued that the foundation of scientific rationality, i.e., “the epistemological assumptions of rationalistic reasoning and ontological assumptions of dualism,” (Ehn 1988: 53) of the Cartesian approach to systems design (based on the body/mind dichotomy) was the linchpin in the political struggle between labor and capital.

What if traditional theories and methods for systems design not only have been politically applied to deskill users, but more fundamentally, theoretically reduce the skills of users to what can be formally described?

(Ehn 1988: 51)

Ehn and the PD practitioners developed a philosophy of design supported by a range of design-by-doing techniques (prototyping, experimental systems design, the use of mock-ups, scenarios) that did not “reduce the practical skill of designers and users to what can be formally described and understood as rationalistic reasoning” but rather grounded designing in activity (1988: 56).

3

While other projects such as DEMOS and the work of Nygaard, with the unions in

Norway, further comprise the foundations of PD, to me UTOPIA draws the most

attention as a symbol of early PD for its appropriate name and the researchers involved

that continue to contribute to the PD community today.

(28)

The early Scandinavian Participatory Design projects in the 1970s championed a Marxist agenda in the middle of the labor/commodity struggle. From this point of view, design researchers aligned themselves with unions in the struggle to democratize the workplace. As in the collaborative prototyping example, the philosophical standpoint provided researchers guidance in determining their research alliances. Greenbaum and Kyng (1991) suggest that the projects represented in their volume, Design at Work, demonstrated an attempt to even out the imbalance among management-driven initiates by siding only with the workers.

Bjerknes et al. describe their “counter-strategy” for taking control out of the hands of management as a way to

“…avoid that skills and knowledge are built into the computer systems instead of being a prerequisite to operate them.” The strategy is also important as an alternative to the development of expert systems—as a means to replace experts—instead of as a means to support experts to do their work in a better way. (Bjerknes et al. 1987: 7)

Xerox PARC’s contribution to Participatory Design During a similar period, in the US, the work of anthropologist Lucy Suchman and her colleagues at Xerox PARC (Suchman et al. 1999 provides an overview of this valuable body of work) explored the social nature of technological use and development predominantly in relation to workplace practices. Suchman’s seminal work Plans and Situated actions: the problem of human-machine communication (1987) argues for the development of

technology that does not simplify and limit its potential use by attempting to solve all the contingencies of human behavior, but rather that

appreciates the current degree and state of human actions and how users rely on local resources to make sense of their tasks to inform their actions.

Suchman, Blomberg, Trigg, Orr and others found a common cause in the Scandinavian approach and worked on numerous publications together with Scandinavian counterparts (e.g., Greenbaum and Kyng 1991). Through the development of conferences and research communities such as

Participatory Design Conference (PDC) and CSCW, they contributed to the development of an international community of researchers and

practitioners, coming from backgrounds and disciplinary traditions of social

sciences such as anthropology, sociology and social psychology, industrial

design and computer science, and they explored the realms of participation

of users and stakeholders in the development of new technologies.

(29)

Three distinctions in user-oriented design

What I refer to as “User-oriented design approaches,” encompass the fields represented by Human Computer Interaction (HCI), Computer Supported Cooperative Work (CSCW), and Participatory Design (PD), Industrial Design as well as the newly formed EPIC (Ethnographic Praxis in Industry Conference). The ethnographic approach from anthropology

4

, with its emphasis upon understanding people, and their activities in the context in which they take place, and a history of telling the story of often

marginalized populations, and PD, with its emphasis on giving voice to marginalized workers in the name of work-place democracy through in- context explorations of possible futures, are similar in orientation. They both rely on their practitioners’ improvisation of methods in relation to the circumstances encountered in the field

5

, and, a learning-by-doing

education

6

. However, there are distinct differences in their applications stemming from their disciplinary histories.

Some have attempted to compare the Scandinavian approach to those of their American or non-Scandinavian counterparts (Muller et al. 1991; Spinuzzi 2002). This was largely due to the “Scandinavian challenge” as setout in Computers and Democracy for systems designers to develop more democratic systems (Bjerknes et al. 1987). It is difficult to generalize about regional differences when researchers such as those at Xerox PARC and the Scandinavians have published together and have influenced each other’s

4

There are many approaches of anthropology. I have been trained in the “four-field approach” of the American (Boasian) tradition that combines social, physical, and linguistic anthropology, and archeology as a basis for a holistic approach to understanding human behavior.

5

A point also made by Blomberg in (Muller et al. 1991).

6

In anthropology, method has rarely been taught directly as an individual topic (Price 2001)

and relies on individual experience in the field in the legacy of Malinowski, who spent

three years among the Trobriand Islanders of Papa New Guinea (Malinowski 1967). The

message has been for students, with their backdrop in theory, to “go there” first, and then

begin figuring out how to make sense of their experience using local logic. At the same

time, design education in Scandinavia as reflected by the University of Southern

Denmark’s Mads Clausens Institute and Malmo’s K3, embrace a Schönian perspective of

learning-by-doing (Chrstiansen 2004; Schön 1987). However, Denmark may also be more

prepared for such approaches due to the influences of the 19

th

century priest Grundtvig

with his strong emphasis upon experiential learning as suggested by Bødker (Bødker

1996). “Grundtvig's anti-elitist perspective on the access to knowledge is, in my mind,

quite fundamental in understanding why Danish society and work life are as relatively

open as they are” (1996: 231).

(30)

work. On the other hand, the discussion raises some important distinctions between the Scandinavian approach and other approaches. I see three main issues at the core of these differences:

the balance of understanding and change

the action-orientation of the construction and use of representations

the role of mutual learning among participants in project work 1. Understanding the situation versus changing the situation

I find that the work coming from PARC demonstrates a practice with a greater emphasis upon “traditional” analytic academic research when compared to the Scandinavian’s emphasis upon action research projects. At the same time, the analytic focus has defined the PARC contribution to PD while the they have drawn inspiration from Scandinavia to reflect upon their own practices by engaging in action-oriented projects. This is

noticeable, to some degree, in Suchman and Trigg’s contribution to Design at Work (Suchman and Trigg 1991). They write about a conversation with Pelle Ehn (1991: 86) during the development of the book. They agreed that some of the authors of the book were reflective practitioners in the Schönian (Schön 1983 [1991]) sense of reflection-in-action, while others were practicing reflectors alluding to a greater emphasis upon reflecting-on-action. In the Scandinavia scene, Schön’s conception of design as a reflective conversation with design materials, allows practitioners from many, or possibly any, disciplinary field to practice in a “designerly way.” He argues that it is through a series of experiments that the practitioner engages in a conversation with whatever may be the situation. The view suggests that the practitioner engages in a series of frame experiments to provoke backtalk rather than possessing an ability to predict how circumstances will unfold.

This notion, however, is dependent upon what the practitioner views as the situation, and how she believes the situation can be changed. The

practitioner relies upon a repertoire of experience to begin experimenting with “screwy situations.” It is only by first “making a move,” trying out a frame, and listening to the back talk, that a practitioner is able to engage in the conversation.

Schön’s formulation supports the Scandinavian approach of viewing the

format of design activities as an object of design in its own right. The cases

that make up the literature of collaborative design, provide a series of

(31)

action-oriented experiments for developing techniques such as the use of low-fidelity mock-ups and other representations as a means of engaging users in imagining future practices (Bødker and Grønbæk 1991; Ehn and Kyng 1991). Similarly, the theoretical contributions to PD are

conceptualizations in better service of including designers and users in the design process. For instance, Ehn introduces Wittgenstien’s language games to conceptualize how practitioners of all types generate locally specific languages and, therefore, that it is not possible to stabilize how people behave and design for them. Instead, designers must engage in language games of the users and invite users to participate in the language games of the designers (Ehn 1988).

This type of unfolding development of understanding with the users in the use context in the service of designing is at the crux of the argument that ethnographic descriptions are not a good resource for design. The moment a design idea is presented, the situation changes, and depends upon continued input from users and designers as contingencies are resolved (Kyng 1993). Schön adds that a practitioner’s priority lies in transforming the situation over that of understanding the situation.

The practitioner has an interest in transforming the situation from what it is to something he likes better. He also has an interest in understanding the situation, but it is in the service of his interest in change. (Schön 1983 [1991]: 147)

However, social scientists have been trained for academic research that favors understanding the situation and has had little academic history in changing the situation, other than the incidental contact while conducting fieldwork.

Researchers are generally trained in practices of inquiry that favor long rigorous analysis before making any sort of claims or contributions.

In many respects, the Scandinavian approach born out of the Computer Sciences, has introduced interventions in the current situation by staging activities that rely on representational techniques such as mock-ups as a means for learning about the users and use context. On the other hand, PARC researchers have contributed with traditional ethnographically-based approaches for building understanding about the current situation before engaging in design intervention activities. Blomberg et al.’s introduction

7

of

7

In many respects, this has been a re-introduction or a recent visible introduction as early

PD sought to get away from ethnographic descriptions.

(32)

ethnographic techniques to PD involved a strong focus on ways of

introducing ethnographic value in multidisciplinary work and acknowledged the challenge of transfer from inquiry to design phases of a project

(Blomberg et al. 1993).

2. Construction and use of representations

While PARC’s video-based analysis has mainly been introduced as a tool for collaborative analysis of field material, often in sessions with other researchers to analyze and theorize about work practice (Suchman and Trigg 1991), and later to engage different actors by co-viewing sessions of watching and commenting about the video (Brun-Cottan and Wall 1995), Buur et al. (Buur et al. 2000; Buur and Soendergaard 2000) have further explored the format for co-viewing activities transforming video into design material by creating video cards representing short video clips as artifacts that reify the contents for participants (users, designers and other

stakeholders) in order that they can be used in idea generation and design activities. I find an important distinction between the attention these two uses of video in the design process give to structuring the activity with the project participants. Similar to the distinction in prototyping techniques that foster demonstrating versus cooperating (Bødker and Grønbæk 1991), the difference lies in the action-orientation of the construction and use of representations. While these differences may not be based on distinctions between the Scandinavian approach versus non-Scandinavian, and may rather be viewed as a sequential development of video use in design starting with Suchman and Trigg 1991, to Brun-Cottan and Wall in 1995, and Buur et al.

2000, I will introduce one more example that highlights the problematic nature of the construction and use of representations.

At this point, I would like to reiterate that I find whereas anthropology has relied upon the improvisation and learning-by-doing, as both a training approach for novice anthropologists, and as a field method, the Scandinavian approach has explored learning-by-doing techniques as a basis for structuring group engagements, setting the stage for collaborative design activities.

Whereas Suchman and her colleagues at Xerox PARC have played a strong role in developing the Scandinavian approach, many other user-oriented design practitioners do not approach the organizational format for

engagement with various actors in the design process, as an integral part of

their approach.

(33)

A more commercial version of ethnographic research developed out of Rick Robinson’s visit to PARC in the early 90s (Wasson 2000) that demonstrates a focus upon analysis rather than designing in context as a basis for design. He co-founded E-lab and together with his colleagues developed a form of ethnographic research to inform the creation of strategic visual models for clients (Wasson 2000; 2002) often referred to as experience models (Jones 2006). The ethnographic process for producing experience models is much closer to the traditional ethnographic research process whereby field material is collected and analyzed to produce an output that does not change once it is presented

8

.

E-lab’s influence on American commercial work epitomized by EPIC2005 and EPIC2006, seems to reinforce the idea of a specialized ethnographic practice, even if it encompasses ethnographers and designers working together to inform design strategy with research. The interaction between users and designers does not embrace the collaborative design criteria as celebrated in the Scandinavian approach. For example, Jones, S. (Jones 2005) demonstrated how, in his consultancy work for the BBC’s iCan website, he and his employees conducted ethnographic fieldwork to build an

experience model of grassroots movements in England. While they were satisfied with their findings and their clients found the research “fascinating and informative” (2005: 34), when clients used it as a basis for designing a website to support grassroots campaigning, the results did not provide a valuable website to support grassroots campaigning. Jones’ company conducted the user tests and found the design was insufficient for supporting the sociality of grassroots campaigning. In his paper, Jones reflects upon the aspects of industry praxis and the over-use or misuse of theory and an over reliance on the client’s intentions when conducting research; but the interactional structure of the relationship between the researchers and designers, and designers and users, are not addressed as a problematic issue (I will explore this point in greater detail in chapter IV).

On the other side of the spectrum, with considerations from the

Scandinavian approach, Diggins and Tolmie (2003) express the problematic nature of the construction and use of ethnographic output through analyzing the “organizational features” identifying the form, use and how open or closed they are as critical features. They argue that it is not solely

8

I make these statements on the available material about E-lab’s work. I realize that working

methods in industry are often not reported thereby leaving the impression they were not

practiced.

(34)

the content of ethnographic representations, but the “situated work to make them meaningful” in collaborative work that is important (2003: 147).

3. Mutual learning among participants

The application of ethnography in design or design ethnography has raised great concerns with purists who decry the misrepresentation and watering down of ethnography by amateurs’ “do-it-yourself ethnography” (Forsythe 1999) or a similar critique of amateurism by what one ethnomethodologist refers to as “scenic ethnography” (Button 2000). The argument states that the illusion that social research has been conducted is more damaging than working with an explicitly non-researched understanding. Trained

ethnographers are versed in a wide variety of the theoretical and

methodological discussions including the politics of representation symbolized by the writing culture debate (Clifford and Marcus 1986) and similar debates in work related to design (Star and Griesemer 1989; Suchman 1995). They develop their craft cognizant of such issues. As Forsythe argues,

professional ethnography is invisible work with its subtleties often lost in the appropriation by amateurs, especially those trained in the natural sciences, due to its reliance upon seemingly straightforward everyday activities such as talking to people in interviews and watching people. There is a tendency for novices to take what people say and do at face value without the training to situate it in the complexities of the sociopolitical context.

Bødker and Iversen (2002) heed a similar warning to those who misconstrue the subtleties of PD by getting caught-up in the “initial fascination with users.”

These issues are not specific to ethnography or design, but rather are at the core of why user-oriented design approaches are deemed valuable in practice. As Suchman argues, the further someone is from a practice, the more they simplify that practice (1995). At the same time, she suggests that there are practical advantages to “black box” the details of other people’s work in order to minimize how much we worry about the details of their work (1995: 58).

Interdisciplinary work necessarily involves structuring activities that allow

people to benefit from approaches not their own. The lessons from

collaborative design in the Scandinavian approach with a focus on users and

designers have demonstrated how structuring activities through the

construction and use of complimentary representations can play a vital role

in stimulating mutual learning that is the basis for transcending from what

(35)

is known, to discover what is new. Just as representations for collaborative design necessarily abstract the details of what they represent to allow people to progress in their activity at hand (Kyng 1993), structuring activities for mutual learning across professional and disciplinary boundaries demands balancing the need to expose enough detail about each for meaningful interdisciplinary work without paralyzing the activity with the over-exposure of details. In addition, conducting in context

activities is sensitive work when exposing sensitive details of others in their politically charged field of tension (Swartz et al. 1966: 8).

Expanding the user-oriented design focus

There is increased demand to explore the detailed organizational formats for collaborative activities as user-oriented design approaches draw interest from people from a wide variety of disciplinary and professional

backgrounds who do not necessarily share the PD agenda, but rather seek to develop user-friendly solutions or products that support use practice or appeal to users.

Increasingly, the results of PD research, in terms of an understanding of the relationships between work and technology and the tools and techniques applied, are being integrated into design professionals’

resources for action. However, the concern for worker participation in design that drives PD researchers, has also been challenged in recent years by economic conditions that predominate on the international scene, where efficiency is emphasized over quality of work life and where the power of worker organizations is declining. Looking for new ways of connecting with workers (in addition to union participation) and new strategies for engaging managers and design professionals in cooperative design, some PD researchers have begun to reorient their efforts somewhat by cooperating with people situated throughout the organizational hierarchy and not soley workers and their unions.

(Kensing and Blomberg 1998: 168)

However, the lessons from the Scandinavian approach provide practitioners with valuable resources for the social interaction necessary for engaging various participants throughout organizational contexts. There has been very little attention paid to the development of techniques and practices for creating and maintaining the human and financial resources to conduct user-oriented design. As the drive for efficiency and the decline of

traditional PD allies in industry have put extra pressure on practitioners to

(36)

justify the value of their work, as suggested by Kensing and Blomberg, there are greater benefits for including the decision maker in the format of activities within the purview of design research and practice.

When the issue of project creation and maintenance is broached, it is often to explain why a project did not produce the desired results. In the

UTOPIA project, reportedly workers lost interest when it became evident that management would not take the recommendations further (Ehn and Kyng 1987). It is often in relation to the lack of support or lack of understanding that user-oriented approaches fail to “gain traction” within organizational contexts (Jones 2005; Rönkkö et al. 2004). Furthermore, those who do address the issue, often do not consider the format of the encounter with decision makers as a malleable aspect of their work to the same degree that as the interactions with other research participants.

Instead, there is a tendency to focus upon how to improve representations used to support one-way verbal and visual argumentation to a more-or-less passive audience (Tunstall 2006).

User-oriented design practitioners are equally challenged to avoid reducing the value of their own “invisible work” to what can be “formally described”

to their colleagues and stakeholders as they once were with the skill practices of users. This issue continually surfaces throughout design projects, but is nowhere more evident than during the high-stakes

negotiations that determine the financial and human resources for a project.

Often in the first meeting among project team members, negotiating with stakeholders, supervisors, or their design colleagues, either from the same or different disciplinary backgrounds, the lessons of structuring activities with users and designers in PD and design ethnography have not been applied (and written about) to the format of activities with decision makers.

I would also like to highlight, that while PD practitioners have been good at describing what they do and the responses of participants of their

activities, they have been at pains to account for the implications of their efforts. There is a tendency to favor action over assessment coupled with a lack of training and tools to conduct meaningful assessments. Muller’s quote epitomizes this point:

We have attempted to obtain detailed questionnaire data from end-users

and from developers on the use of PICTIVE. However, when we faced a

choice between doing project work and collecting data, we have generally

(37)

erred on the side of less data and more productivity. Therefore, our assessment is limited at present. (Muller 1993: 224)

Often there is a push for a scientific rationale for assessing success such as attempts to rely on quantitative tools that favor attempts of objective conclusions (for relevant discussions, see Emerson and Pollner 1988;

Suchman and Jordan 1990).

Including decision makers in design

Regardless of the idealistic stance that may be driving a project, whether it is financial return or emancipating workers or users, or addressing issues of marginalized populations, or looking to develop user-friendly products, there is a very pragmatic question of how to bring about support for a project and how to sustain a project that is grounded in multiple practices.

This is not likely to be an academic question that can be answered outside of practice and applied to practice, but rather an organizational reality that must be grappled with in every project by those involved regardless of whether it is an industry-related project or a project based in academia.

Grappling with this reality is the sustained focus of this thesis.

I seek to expand the purview of design activities to include the structure of engagements with the gatekeepers of human and financial resources as a key aspect of introducing user-oriented design practices in various organizational settings. In many cases, stakeholder support is necessary prior to the introduction of any of these practices. I believe for a project with innovative or new practices to “gain traction” within an organization among actors (including decision makers, designers and users) in various professional contexts, it is vital to roughen the surface on which we wish to gain traction, namely by defining the problems of the formats for

engagement in order to socialize participants to appreciate the value of new perspectives.

I have argued through the introduction of three types of action-oriented

activities—collaborative prototyping, using videos as design material, and

defining organizational features of ethnographic output—a subtle, but

important, difference between passive and active orientation of working

methods. I have suggested that the practice of exploring alternative formats

for activities and the active representation forms to engage participants are

techniques and practices born out of the PD tradition (e.g. visual and

physical representations, and bodily movement, provotypes and scenarios). In

(38)

the remainder if this chapter, I introduce how action-oriented practitioners have explored techniques and practices for approaching the format of activities with stakeholders. I end the chapter suggesting that there is a lack of conceptual and practical tools for practitioners to plan, organize, and assess explorations of stakeholder activities in the course of their project work.

Games, such as board games with their pieces, rules, turns, and goals, are good examples of representations that support an active format for social engagement among all the players. Design games have been a part of the PD toolbox for their organizational features for some time. Ehn suggests that games are valuable resources for engaging project participants in imagining future practices for their family resemblance of the games people have played before (Ehn and Sjögren 1991). Going one step further, Ehn suggests that Wittgenstein’s language games metaphor provokes us to stage games for designers and users to develop new language games of the future. Other action-oriented practitioners have used games as

organizational tools for staging activities with a broad range of participants in the design process.

Iversen and Buur (2002) describe three years of action research using design games in educational and industrial settings for building design competence for novice designers and experienced designers alike. Through describing how students and industrialists are introduced to designing games, describing some examples of the games they develop, and providing a list of their reflections, they argue that developing design games builds design competence in novices.

The development of games (and other strategies) was part of their teaching

practice with students as well as their research practice with industrial

partners. It is their contention that project-based teaching does provide

students an opportunity to use design methods in action, but that it does

not allow them to build a repertoire of design practices. They claim that

students focus more on sticking to the rules of the method, than focusing

on the situation they are facing. They look to design games to overcome

the obstacles of teaching design. Based on the Habraken tradition of design

games and Schön’s proposal for a reflective practicum (Schön 1987), Iversen

and Buur propose that their design is a game course, where students play and

develop their own games, contribute to students’ learning a collaborative

design vocabulary by modeling design situations, and enabling students to

explore “real life design.”

References

Related documents

Stenbom S, Cleveland-Innes M, Hrastinski S (2013) Examining a learning-driven relationship of inquiry discerning emotional presence in online math coaching. In: Proceedings of the

organization? How could knowledge be better managed in the organization than it is today? How could this be

Arbetet kommer kretsa runt fyra huvudområren, Att skapa en kultur för ständiga förbättringar, ledningens engagemang, medarbetarengagemang och anställdas inställning

The overall aim of this thesis was to provide better understanding of the underlying factors related to health maintenance in very old people, with a focus on medical conditions,

överenskommelsen om internationella transporter av lättfördärvliga livsmedel och om specialutrustning för sådan transport (ATP), som utfärdades i Genève 1970 och trädde i

I min modell passar inresande från Finland in medan inresande från Sverige inte gav signifikant påverkan och därför inte heller fick plats bland mina förklarande

To demonstrate simulation of immaterial, business-related properties in functional products, a model designed to predict the cost and delivery time of carrying out the

Intervjuerna visar att den historia de får förmedlad hemma skiljer sig åt den historia skolan förmedlar och kommer med alternativa förslag för att göra plats för utomeuro-