Designing interactions for data
obfuscation in IoT
Carlos Mario Rodríguez Perdomo
Interaction Design Twoyear master's Programme 15 credits Spring 2016, Second Semester
“I don't want to live in a world where everything that I say, everything I do, everyone I talk to, every expression of creativity or love or friendship is recorded.” [13] Edward Snowden,The Guardian , 8 July 2013
Acknowledgment
I would like to thank Anuradha Reddy for her valuable guidance during this process and for encouraging me to explore beyond my own expectations. Also, I would like thank David Cuartielles for his always enlightening advices.
To Jody Barton, Sena Çerçi and Luke Richards, I would like to thank for their substantial feedback and help.
Abstract
This project explores the internet of things (IoT) at home, especially the aspects related to the quantity and the quality of the of data collected by the smart devices and the violation of the users’ privacy this situation represents, since with the help of machine learning algorithms, these devices are capable of storing and analysing information related to the daily routine of each user at home. Therefore, this research enquires new ways to raise the user's’ awareness about the flow of the data within the IoT at home in order to empower them and give them back the status of administrators of this context by designing devices that are capable of obfuscating the data before it leaves the home.
During this process, several methods were used together in order to reach the outcomes. From the use of annotated portfolios to evaluate the state of the art related with the field, to video sketching as a useful and quick tool to embrace the user’s perspective in parallel with the use of cultural probes in order to test some conceptual scenarios and find new ways to explore this project based on the experiences of the participants.
As a result, this project’s outcome is based on the use of the materialization of the data as the proper way to bring the abstract process that happens in the background closer to the user's reality in order to display how this data is actually flowing through the environment and at the end generate a calltoaction to guide the user into the execution of the obfuscation of the data.
This project opens up the discussion within the interaction design field about the way we are communicating with the technology and if it is the proper way to do it when this technology coexist with the user at home. Additionally, it questions the way in which the interfaces should be designed in order to create a transparent dialogue between the users, the objects and the vendors. Keywords: IoT, critical design, privacy, data obfuscation, HCI, tangible interaction.
Table of Contents
Introduction 1 Part 1: Research Framework 1.1. From ubiquitous computing to IoT at home 3 1.2. Privacy and security 4 12.1. Administrator permission 4 1.2.2. Data storage 5 1.3. Related projects 6 1.3.1. Ethical Things 6 1.3.2. Lightbeam 7 1.3.3.Poetic Router 8 1.3.4. Apply Magic Sauce 9 Part 2: Theoretical Framework 2.1. Critical design in the IoT environment 11 2.2. Annotated portfolios 12 2.2.1. Materialization Visualization 13 2.2.2. Unconscious Awareness 13 2.3. Annotated Portfolio analysis 14 Part 3: Research Methods 3.1. Desktop Research 17 3.2. Annotated Portfolio 18 3.3. Ideation 18 3.4. Prototyping 19 3.5. Video sketching 20 3.6. Cultural Probe 21 3.5. Sketches 21 Part 4: Design Process 4.1. Desktop Research and Annotated Portfolio 23 4.2. Concepts 24 4.2.1. Data Obfuscation 24 4.2.1.1. Substitution (Lie) 26 4.2.1.2. Nulling out (Hide) 26 4.3. Cultural Probe 26 4.3.1. Cultural probe outcomes 29 4.4. Prototypes 30 4.4.1. MQTT 304.4.1.1. MQTT exploration 31 4.5. Fictional scenarios 34 4.5.1. Temperature and humidity 35 4.5.2. Light 36 4.5.3. Fictional scenarios Outcomes 36 4.4. Sketching Iteration 37 4.5. Final Iteration 40 4.6. Reflection 42 Part 5: Conclusion 5.1. Discussion 44 List of Figures 46 References 47 Appendix 49
Introduction
IoT devices have been around for more than 20 years now, but the inclusion of this technology at home has been spreading massively during the last decade without any concern about how the users could react when they live with an artifact that is always surveilling, recording and analysing their daily routine, or about the need to reveal the quality and the quantity of the data these devices are gathering in order to create a transparent dialogue between the system and the users.
Right now, the current relationship between the users and the IoT vendors is based on how to provide a better automatization service without revealing how these accurate solutions use the data or even when they get that data. This situation creates a lack of awareness from the user perspective about the complexity of the interoperability between devices, since it isn’t only about turning on and off the light when the user leaves home. It is about how the interconnected information such as location, energy consumption, sport habits, eating habits and more data coming from different sources can be used by the vendors to predict events that may happen in the future in regards to the user’s behavior.
For instance, the anticipatory shipping [1] is a patent owned by Amazon, and is based on the connection of multiple layers of information such as the user’s location, the user’s previous searches, the searches made by others around the user’s area and information about the prevalent trends, which enables Amazon to ship products to the closest delivery hub even before the user takes the decision to buy it in order to reduce the shipping time.
This may seems like a good deal; however these kind of data analysis systems pose a threat to the user’s privacy and it could be more intense if we translate this scenario into the IoT at home where the amount and the quality of personal sensitive data increases.
In this case the users not only reveals their names, address, likes and financial information, but also habits, routines, personal ritual and more information that could fall into the wrong hands, for this reason these kind of data analysis systems could have a tremendous impact on the user’s life.
At this point, during the big moment of IoT, it is imperative to ask about the implications of this technology and reduce the selective exposure we are immersed in, whereas people 1 prefer avoiding the fact that we have been watched and someone else is using our personal data for some purposes we are not even aware of. We have to admit that this is an uncommon situation and never before we were exposed to the fact that we have to live with such sophisticated pieces of technology that go beyond the boundaries of what we define as home.
For this reason, this project aims to evaluate some data obfuscation strategies as a way to empower the users in this unbalanced system, by giving them the chance to execute tangible malfunction operations that will contaminate the data to be stored and analysed at the vendor's server.
The main motivation for this project, aside from empowerment of the user, is to raise questions about the awareness of the data and the privacy issues that the ubiquitous aspect of this technology raises since these devices work as invisible watchers of the user’s routine, without giving any chance to hide from them. This situation is already happening within the internet as infrastructure and if the interaction design field doesn’t start working on the ways to solve it, our future will be guided by algorithms that control the systems in our lives.
Part 1: Research Framework
1.1. From Ubiquitous Computing to IoT at Home
At the beginning of the computer era, the idea of microchipcontrolled machines everywhere seemed like a futuristic illusion for users; however as Moore's law has predicted, since 1965, the power of microcontrollers has been increasing in a predictable rate, making what was once thought a fictional scenario is a reality nowadays. In the same way, Gillian Crampton Smith claimed that there are many more computer chips on the planet than there are humans [36], therefore researchers and designers should start thinking about this novel reality we find ourselves in.
The initial understanding of this new reality started with the creation of a group at Xerox PARC, which has been trying to understand the possibilities of a fully connected environment since the late 80s. Back then, the members of this team coined the concept of calm technology, which then was called ubiquitous technology as: " a world richly and invisibly interwoven with sensors, actuators, displays and computational elements embedded seamlessly in the everyday objects of our lives and connected through a continuous network
" [42]. Nowadays, this world is a reality that we call IoT, and lives among
us, acting based on the concepts of seamlessness and interoperability as its core.
The idea of a continuously invisible and seamless interconnected world creates a lack of awareness from the user’s point of view about their context. This is a phenomena that Weisser explains as a group of tools that don't interrupt the actions of the user, but rather gives them the opportunity to focus on the big picture, almost like a magical disappearing act [42] which is an important feature within the smart devices especially at the home environments, where some robust products already coexist with the users and in this way share the responsibility of controlling the environment along with the user. Then, via some sophisticated machine learning algorithms, these devices would have the capability to map the daily routine of the user, in order to provide an imperceptible, pervasive and efficient service.
The market of domestic IoT devices is growing everyday, populating the home environment with automation systems, surveillance systems, light control systems, coffee makers, VR headsets, remotecontrolled audio systems and this list can continue since the main goal of these devices is to provide the perfect customised service to the user. However, from a broader perspective, all of the IoT systems at home are based on a similar structure formed by two main contexts, firstly the home as the place where all the interaction with the user and devices happens, secondly the servers where all the data is gathered and the artificial intelligence runs, and both of them interconnected through the internet. For this reason, the main focus of this project will be the home environment where the users still have the chance to have an active role with the quality and quantity of data they want to share with these external agents involved in the system. However, it is important to understand that at
to gain access to these services, and that's the only touchpoint where they take notice about the data they will be sharing through these devices. Figure 1. Infrastructure of IoT at home
1.2. Privacy and Security
The issues related with the privacy and security in the IoT at home could be divided into two main situation. First, the gap between the understanding of the nature of the information within the IoT world and and the tangible dimension that the user’s world has. Second, the way the data is store nowadays and how it is use or could be potentially use.
1.2.1. Administrator permission
As soon as the IoT met the home environment, where the users are used to having complete control over the elements around themselves, this pervasive technology has created an interesting scenario for new products and services which will drag the users into a situation where they don't have to worry about the functional aspect of their context anymore since all the smart devices provide the perfect setup for a comfortable situation.
Even though this image seems suitable, it brings to the table several consequences where the users lose the control and accessibility to the infrastructure of the context and become spectators in an environment where they are supposed to be the administrators; instead now, in the IoT homes, the real administrator is a sophisticated engine running in a server miles away from the user and their control.
This distance relationship between the user and the technology happens in a world of encapsulated devices or black boxes where a perfect working piece of technology will make that the users take the functionality for granted, focusing only on the inputs and the output, while forgetting everything about the internal complexity of the device. Therefore, the more this technology works seamlessly, the more opaque and obscure it becomes.
This situation raises, some technical security problems related to the device itself, like the research that Daniel Buentello showed during the Black Hat conference [14] where Daniel and his team found a way to hack the Nest thermostat in just 15 seconds, making clear that behind every IoT devices there is a regular computer, which in this case was an ARM processor that could be modified with a new firmware in order to create an invisible but interrupted transfer of data that could be sent to third parties, with intentions other than the original.
These kinds of vulnerabilities are important within the whole situation of IoT due to the fact that common users can’t have access to the embedded systems of the devices, while there are some special users who can find a backdoor and use the data in other ways. However, these aspects are completely related to the security of the device from a hardware perspective and won’t be evaluated in this project, due to the reason that the main focus is the relationship with the data management and the violation of user’s privacy through this technology viewed from the interaction design perspective.
1.2.2. Data storage
From the data storage perspective, there are many cases that show how insufficient the security strategies applied to some servers may sometimes be. For instance, in 2011, the servers of Sony PlayStation were hacked and as a result of this attack, more than 77 million accounts with sensitive personal information were exposed [29], revealing information such as full name of the user, home address, email address, IP address, credit card number, date of birth, birthplace, telephone number and username. If we translate this case to a scenario related to the IoT at home where multiple levels of the users’ data have the potential to be exposed and the quality of this data could generate a deeper impact on the user privacy, since in this case it can extend to information about the daily routine of the user, creating a bigger gap in their personal security.
This research seeks to create a statement based on the three main questions asked by Edwards et al. as follows; " What kinds of affordances do we need to provide to occupantusers to make the system intelligible?
home?
", " Where will the locus of interaction be in a system that exists in no one place, but
rather represents the sum of many interoperable (and changing) parts?
" [7].
These questions summarize two issues that are unsolved in the IoT, viewed from the user's perspective. On the one hand, it reveals the need to understand the degrees of control and freedom the user needs to be aware of, now that they are coexisting with smart data collectors, which can create a physical impact into their environment. In other words, the current situation of the IoT devices at home is trying to introduce services that cause total isolation of the user from the functional aspect of the home, taking away the possibilities and responsibilities of new kinds of affordances that the users could have.
Also, it is possible to question the interoperability features of IoT. For instance, the Nest devices can exchange data with other smart devices, which could have more sophisticated2 sensors or actuators and creates a complex fabric of data, which not only provides a better service, but could also become a surveillance tool for many others interested in this data.
This data could even be use in legal situation, for example, the section 215 of the USA Patriot Act , claims that the Government " 3 may make an application for an order requiring the production of any tangible things (including books, records, papers, documents, and other items) for an investigation to protect against international terrorism or clandestine intelligence activities
" [8]. This means that if for some reason the pattern of behaviour of any
user changed in a way the authorities claimed to be suspicious they can ask for data that isn't only related to a specific environment, but now they can also access to complex data related to the routines of the users at their homes, or indeed any other smart environment they might frequently inhabit.
For all the situations explained before, this project will work with two concepts closely related with this technology, affordance of data and awareness about the data
, in order to explore
new ways to reveal how intrusive these devices are in a context where users should have control but don’t.
1.3. Related Projects
Many projects were evaluated during the research phase of this project however only four will be mention in this document since they exemplify the four opposites that will be used in the annotated portfolio, materialization/visualization and awareness/unconscious.
1.3.1. Ethical Things
This project by Automato Farm [9] is a fan which is able to take the decision of whom it should point at based on the opinion of a group of people connected to the Amazon Mechanical Turks application who have access to some information the users can set by
2 The Nest Devices [24] is a family of IoT products consisting of a thermostat, a smoke alarm, and a camera. 3 The Patriot Act [8] is a piece of legislation created by the United States Government ostensibly in order to create protection against terrorism.
turning four knobs on the sides of the device, which describes aspects such as religion, degree, gender and age. Then, this information will become the criteria to take a moral decision of to whom it should help.
The aim of this project is to criticise the fastdevelopment of autonomous systems which in some cases are in charge of solving situations that need some degree of human morality, such as self driving car or lethal battlefield robots. Automato Farm claimed that most of the time these discussions about ethics and technology always involve extreme cases, however they try to transpose this discussion to other situations such as the smart devices in home environments, where there are devices that can gather substantial information about the user but, are these devices able to take the right decision in such cases like if a ‘smart’ coffee machine should make a cup of coffee to its user upon their request, when it knows about their heart problems.
Figure 2. Ethical Things by Automato Farm
1.3.2. Lightbeam
Lightbeam [21] is a MozillaFirefox addon that uses an interactive visualization to show which websites are interacting with the user while they browse the web and who is getting data from this process, due to the fact that the communication with one specific server could often be tracked or observed by a third party. The Mozilla Foundation claims that tracking data is not bad, since it helps to raise the awareness of the data shared by the user while empowering the user by giving a transparent experience throughout the use of the web.
This tool is partly owned by the Mozilla Foundation, a nonprofit organization that promotes the values of an open Web and understands the growing relevance of online privacy, therefore they are part of some initiatives such as stopwatching.us [39], a project trying to gather organizations, individuals and business to protest against the NSA’s surveillance apparatus and the abuse that it represents to the basic rights. Figure 3. Lightbeam visualization
1.3.3. Poetic Router
This device designed by Saurabh Datta [30] is middleman router connected to the home network and creates an access point which other devices can connect to, and that way it can monitor all the traffic going through. After listing all the communication occurring, it collects the HTML text from the pages in order to create the body of a poem, which is then sent to a FM transmitter, which recites the poem generated before by using a Text To Speech algorithm. The motivation of this project is to create discussion beyond the the privacy of the communications in IoT and try to view the data around the smart devices at home from an artistic perspective.
Figure 4. Poetic Router by Saurabh Datta
1.3.4. Apply Magic Sauce
In 2013 Cambridge University’s Psychometric Centre created Apply Magic Sauce [2], a web service which collects all the digital footprints from the user’s Facebook accounts and uses this data to predicts a psychological profile of them. This service can predict things like the psychological age, the psychological gender, five big personality features such as openness, conscientiousness, extraversion, agreeableness and neuroticism. It can also predict life satisfaction, sexual preference, political orientation, religion orientation, professional interests and relationship status.
This service is a psychodemographic prediction based on the user’s Facebook Likes and creates a visualization of how others perceive them online. Even though they claim this picture may not be entirely accurate, it is important to highlight the kind of awareness this tool can provide based on the idea of the digital persona we are creating everyday on Internet.
Figure 5. Apply Magic Sauce structure
Part 2: Theoretical Framework
2.1. Critical Design Approach to IoT at Home
To raise a critical perspective, this paper will use the three design strategies described by Khovanskaya et al. [20] and at the same time, the concept of the counter functional things by James Pierce et al. [28] will be considered in the way the functions of the objects could be reinterpreted to explore new relations with the user.
Make it creepy, make it strange, and make it malfunction are the three strategies that this project will use in order to understand the relationship between the critical design and the current situation of the IoT at home in order to create awareness about the flow of data back and forth from the smart devices, and to explore new affordances for the users to interact with the devices around the home. Each strategy will be explained by its influence on the current infrastructure and the motivation behind the use of them.
The use of the first strategy, make it creepy, will try to tell the stories of the users based on the information collected by the devices involved in the IoT data collection at home. The main goal in this case is to reveal the interoperability not only between the different sensors installed at home, but also those that the users carry everyday, like the GPS of the mobile phone, the wearable devices that measure steps, heartbeat, sleep routines and so on. The main motivation of this strategy is to create a creepy big brother situation of an external element writing the biography of the user’s life.
The second strategy, make it strange, aims to create a strange situation for the user by displaying the image that the devices at home have of the user in order to create a disruptive relationship between them, and that way the user will try to change their routines if they disagree with that. In this particular case, the goal is to show the possible outcome that the machine learning algorithms have from the user, and also to explore the emotional impact and the social pressure that data could have on the user.
For the malfunction strategy, the user will have the possibility to contaminate or manipulate the data gathered by the devices before it leaves the home network in order to give some control back to the user. However, in the end, it is possible that all this corrupted data will have an impact on the service provided by the actuator within the home, such as, incorrect temperature routines, incorrect lightning behaviours and other situations related to the quality of the data sent to the servers for processing.
After the analysis of these strategies, it is important to notice a constant overlapping between the three of them in relation to the flow of data within the system and how the structure of the hierarchy of data in IoT leaves always the user at the bottom of the pyramid. Hence, in order to empower them and give them control over the use of the data, we can
conclude that within the devices in the home contexts is the gateway at the home network, and more particularly the router, the most relevant element of the system for this purpose since it is the last touchpoint the user could have with the data before it is transfer to the vendor’s servers.
Thus, the router will become a “counterfunctional thing” defined by James Pierce as “ a thing that exhibits features that counter some of its own ‘essential functionality’ while nonetheless retaining familiarity as essentially that thing
” [28], for this reason, this object combined with
the new situation that the critical design perspective will provide as a result during the design process is the primary tool to explore the empowerment of the user.
Figure 6. Khovanskaya et al. strategies applied to the project
2.2. Annotated Portfolios
To understand a broader perspective of the context where the project will coexist, I created two annotated portfolios which helped me to place the chosen related project across two axis. The X axis goes from materialization to visualization in order to map the solutions used by other projects in relation to the used medium, and the Y axis goes from awareness to unconscious in order to grasp the relationship each project tries to address between the user and the data.
The first annotated portfolio tries to understand the relation previously described and the second one overlaps the finding from the first annotated portfolio with the results of the Khovanskaya et al. strategies applied to the project described in the previous point of this section in order to overlap the results of the first brainstorming approach with the context where the project will be placed.
Also, it is important to mention that not all the projects are in the same research field, since some are more related to an artistic perspective of the situation, while others try to reveal the technical or cognitive aspects of the IoT home situation.
2.2.1. Materialization Visualization
This range correspond to the X axis in the following diagram and it reveals the influence of the mediums and its relation with how the user could understand the flow of the data within a network or an specific place through an object.
The projects at the Materialization side are directly related with the TUI (tangible user interface) which according to Hiroshi Ishii [18] corresponds to give a physical form to the digital information, making easier to the user the manipulation of bits. This means that in the case of the IoT devices the information gathered by the sensors will create a reaction in the environment, which is the way how they work however the chosen projects explore a different understanding of the interpretation or the reaction of that collected data.
On the Visualization side, the projects try to address the quantity and the complexity of the data transferred through internet mostly using GUI (graphic user interface) components such as button, windows, graph and diagrams.
The main goal of this axis is to understand the contracts between these two way to represent the information, some could be mainly a graphic representation which could give a wider access to different layers of information or could be tangible which could have a more meaningful impact in the user’s life without expressing the whole complexity of the data. Hence, this axis is based on the affordance and manipulation of the data.
2.2.2. Unconscious Awareness
The Y axis explores the transparency of the use of the information in relation with the output viewed from the user perspective. Even though, all the project explore the use of the data from a critical perspective, its position depends on the clarity of the conveyed message.
The upper side of the graph have projects with the interest of make the transfer of information transparent to the user and try to reveal the gaps of the current situation and also reveal the infrastructure where the data comes from, while the lower side of the graph has projects that try to create a re interpretation of the data, without revealing the infrastructure behind.
Both perspectives display the range in which this project will coexist since depending on the analysis of the context and the user, the aim of the final message could have different nuances.
Figure 7. Annotated Portfolio
2.3. Annotated Portfolio analysis
Ethical Things [9] is a fan which can decide whom to help first based on a collaborative decision from the Mechanical Turks application. The aim of this object is to criticise the lack of a moral criteria in algorithms revealing the inability of the technology to take decisions based on the ethics of the situations. This project is located at the materialization/awareness corner of the annotated portfolio, because it uses a physical object that will modify the tangible world based on a piece of software. Furthermore, the fact that they decided to use a banal object raises more questions about the invisibility of the IoT technology, which is a concept that also affects the Y axis.
At the visualization/awareness corner, there is Lightbeam [21], a Mozilla Firefox extension that helps the user see who is tracking their Internet traffic through an interactive
visualization, revealing normally invisible communication to the average user. Even though this project doesn't affect the real world directly in the same way the previous project does, Lightbeam is giving the users the opportunity to go deeper into the communication and help them understand how their internet traffic has been tracked and eventually modify their habits and increase the concerns of who is surveilling our internet activity and especially if we agree with the surveillance aspects that this situation raises.
The Poetic Router [30] works as a "man in the middle" router, where all the communication could be analyzed, tracked, and in this particular case, the router lists all the links, sends a request to to all the found links and then with the returned information creates a poem that is read by a text to speech voice coming from a radio. This project is located in the materialization/unconscious corner of the annotated portfolio, firstly because it gives a new technical dimension to the router as an object, but also extends the relationship with other elements around it, increasing the explored capabilities using similar communication protocols in this case Wifi and radio frequencies. However, it keeps some aspects of the invisibility of the data due to the fact that even though it tries to create something out of it, it’s still something that the users will never be relate to, unless they are familiar with the whole conceptual and theoretical background behind.
Finally, the project at the visualization/unconscious corner is an experiment by the Cambridge University called Apply Magic Sauce [2]. This project tries to calculate the digital footprint of Facebook users by acquiring their information related to likes or status and compare it with more than 6 million different social media profiles and in the end they can create a profile of the user where they can define some personally aspect. In this case, it is important to understand the amount of data the Internet users leave behind every time they browse the Internet, and how these digital persona created by these data could affect the real life of the users. This service is relevant to the project because it brings to light the picture we create by sharing data on internet and raise the question whether we agree with this picture that the Internet users will get from us or not.
To summarize, these projects try to reveal new ways to raise awareness about the data shared by the users and the multiple possibilities that can emerge by the the analysis of it. With this particular situation, this paper will seek to evaluate the current interoperability of the various IoT devices within our homes, and their intrusiveness in the users’ lives by creating critical scenarios, which will propose new ways of affordance in order to empower the homeowner, and give them back the status of administrator of their own private space. In order to place this research in the current stateoftheart, it is important to locate the strategies described above within the critical design approach to IoT at home section in the same context of the related projects to understand their differences and similarities.
Figure 8. Annotated portfolio overlapped with Khovanskaya et al. strategies.
In this second annotated portfolio, the “Make it Creepy” strategy, which creates the biography of the user by using invisible data, is closely related with the materialization concept used by the Poetic Router, since both of their outputs are narrative structures. However, the closest link between the user and the output described by strategy used in this project could create an interesting reaction in the user’s actions and habits. At the other end of the spectrum, Lightbeam, and the “Make it Strange” strategy share a similar input in a sense of the interoperability with different results; however, both of them impact the user’s behaviour and look for a more conscious use of the technology. Meanwhile, the “Make it Malfunction” and the Ethical Things try to empower people and create an uncommon relationship between the user, the data gathered by the sensors, and the service provided by the device. Besides, it also creates an interesting way of affordance where the technology is simply a medium, and not the main core of the interaction.
Part 3: Research Methods
3.1. Desktop Research
The desktop research phase is useful as the first stage of the process to understand the main features of the topics, which will be overlapping within the project. So, at the beginning, it is relevant to define an open brief or a question for the evolution of the project or the way Salmond et al. [33] explains as “Define the problem, solve the problem
”.
This stage is where the designer must explore ideas executed before, who or what are the parts of the system and how they participate within the current situation. In the case of the IoT, these are the companies that manufacture the products, researchers, the user, and even those who with malicious intentions want to manipulate the system; and on the critical design side, there are design studios, researchers and solutions coming from special user such as the members of DIY community.
After the formation of a brief which can give a broad perspective of the field, the next step is to be more specific with the situation, which will become the core of the project, therefore, it is important to dig deeper and apply a literature search method [19] in order to find published information that can influence the output of the project. In this step, it is important to access relevant archives or start by reading mainstream writers about the topic, and from there use the references they use to continue with the research.
Then, it is important to create a boundary examination [5] in order to refine the problem and find the particular topic within the field which the project will work with; however, since this is an evolving process, the boundaries could change throughout the design process. To be more specific, since the first analysis of the field, the designer must write a statement which will evolve during each step of the process in parallel with a list of keywords and adjectives which will help to build the criteria in order to expand the spectrum of future solutions.
Also, it is important at this point to make a sampling with experts [5] by sharing the project concerns with them and to understand their reactions and their feedback. This step is useful to grasp how provocative the focus of the project is and its relevance within the interaction design field.
The result of this process is the research question and some additional keywords which will rule the project; however, the desktop research phase cross the design process since the first day until the end of the project for this reason, the designer should be open to refine the statement constantly.
3.2. Annotated Portfolio
The annotated portfolio is a useful tool to set the context where the project will coexist, constrained by the keywords and the research question.
This method is described by William Gaver [12] as the way to present at the same time the particularity and multidimensionality of design process without leaving behind the demands of theory, therefore in projects that overlap several fields, this method works perfect to reveal the intersections between them.
It is vital for this method to recognize the annotated portfolios not only as a visualization tool, but also as a critical tool of the current situation, since part of the outcome of this process is to find out how the research question will fill a gap within the current design landscape; a gap which could have been previously studied by others, but from different perspectives or by using different mediums.
This method will help the designer be critical about the assumptions found in the previous steps; therefore, at this point, the research question could have some changes. However, as I mentioned before, this part of the project aims to understand the field and then finding the right place to locate the research.
The result of this stage is a complete overview of the state of art and a graphic piece that shows where the project is placed. Additionally, the research question at this point is fully grounded and ready to be challenged by the other stages of the design process.
3.3. Ideation
The ideation process is constantly overlapping with other processes during the project, because, since the first stages of the project where the research question starts to emerge, some vague ideas on possible ways to solve the problem emerge as well, hence all these ideas should be collected and added as notes to a design diary where the designer collects the raw data of the process before it is analyzed.
Figure 9. Design Diaries.
Also, during the prototyping phase where there is a direct contact with the technical aspects and the users, the ideation process will help to connect the research question and the criteria built during the first stage of the project with the outcomes of the prototype.
The ideation method is a constant process in the designer's mind, however it is essential to keep this creative process organized using several methods such as a checklist [19] where the designer needs to prepare a list of questions based on the research statement in order to evaluate the project. Additionally, another way to organize this process, is by the creation of a selected criteria [19] which is a set of objectives that have to be satisfied in order to continue with the process.
Since the ideation process is always happening throughout the course of a project, it is difficult to define an expected result, however this process and the one related with the selection of the idea are important checkpoints to realise the quality of the project based on how generative and challenging for the field the outcoming ideas are.
3.4. Prototyping
Since the interaction design field tries to merge topics, the prototyping stage becomes crucial to test where and how they intersect, for this reason, the use of the Houde and Hill model [15] is useful to find which aspects of the evaluated situation the prototype will affect.
The Houde and Hill model reveals the complexity of the prototypes and the interconnection between different aspects, therefore it is essential to create a list of questions or issues the prototype will answer based on three specific subject: Role, Look & Feel, and
Implementation. The next step is to prioritize those questions and try to find some links between them by placing the aim of the prototype in the model.
Then, after understanding the function of the prototype within the design process and the building and testing phase, the designer must gather the results and judge the project using the criteria developed during the previous phases and the research question from a critical perspective. Also, since a prototype is an interconnected system within itself, some other questions and issues will lead to new decisions.
Figure 10.Houde and Hill model applied to the technical prototypes
The result of this phase is hard to define since this method will try to narrow down during the building process, but it will open up once again during the of analysis stage, so it is also important to try to set some boundaries or priorities based on the external factors of the project, such as goals, time and resources.
3.5. Video sketching
Video sketching is a helpful tool to understand from the user’s perspective what is the relations between the user, the context and the object involved in a specific situation and on the contrary of the video scenarios, this particular method is meant to use the simplest and quickest tools around.
John Zimmerman [43] define four main advantages when video sketches are used in a project. First, Video sketches help to decide how a concept works at a low level of details which leads to the second advantage which will help to focus on the interaction design reaction. Also, it reveals the relation with the context and the usability issues the design
concept could have and based on the last three advantages this tools become an important way to communicate and share the concept.
This last advantage overlaps with the fact that the tools used to build the video sketch are usually simple and quick which give it a sense of a work in progress and at the moment of been shown and shared to a group of participants they will feel more open to discuss and point the mistake and in this sense it becomes a perfect tool for ideation within a workshop .
3.6. Cultural Probe
The cultural probe is an ethnographic strategy coined by Bill Gaver et. al [10] influenced by the International Situationist, a french artistic group from the late 1950s to the early 1960s that developed useful techniques between the artistic perspective and the ethnography approach such as psychogeography and la derivé (drifting).
The aim of the cultural probe is to provoke the participant’s imagination in order to trigger questions about specific aspects of life, values and thoughts.
This strategy involves a set of objects and instructions that will guide the participant into actions and situations that could challenge established ways of their lives or even reveal new perspective and values about their context. This approach gives the opportunity to get information from the user’s perspective without disturbing their context or even their expressivity.
An important part of this strategy is to create a proper level of engagement at the beginning by meeting the user at the context where the probe will take place, that way by checking the context the cultural probe will have a better fit in the flow of the user’s daily life and at the end of the process it is also important to have a meeting with the participants in order to understand how the probe changed or disrupted their lives.
This last part is the trigger for what Gaver defined as UserCentered Inspiration since the cultural probe could open unknown perspective from the participant side, which enrich the project perspective or even change and ground different directions about the project.
3.7. Sketches
Sketching is an amazing tool to generate and iterate ideas and solve problems, it is also an efficient way to communicate concepts, not only because it is fast but also because it opens up the opportunity for others to participate in the project by discussing or commenting.
Mike Rohde [35] talks about the importance of this process more as visual thinking rather than by its aesthetic purposes and by this way he encourages people to avoid sentences like “I can’t sketch because I’m not an artist”. Rohde also highlights the importance of sketching
as an addition to the designer’s repertoire, next to the software and hardware tools, since every tool opens up the opportunities to new working spaces.
Sketching has three main features that make it especially useful for the designer. First, the variety of ideas that this process can generate quickly provides the perfect tool in a process where the iteration and the exploration aspects of the design are the main concern. Second, the possibility to explore the alternatives since using this method isn’t the constraints by the possibilities of other mediums therefore, it helps to evaluate solution and detect in an early stage potential problem during the production stage. Finally, the last feature is that it is the perfect tool for discussion due to the unfinished look of its outcome, enabling and encouraging others to comment without any concern about destroying the work of long hours.
Part 4: Design Process
4.1. Desktop Research and Annotated Portfolio
At the beginning of the design process, the aim was to understand the big picture about the IoT and its different perspectives, therefore I started my research scanning the reports from the IOTAP (Internet of Things and People Lab at Malmö Högskola) [16]. This part of the process helped me to find out what could be the most problematic aspect within this interconnected and complex field. As a result of this process, I found out about different issues such as configuration, user interface, cognitive issues between the user and the devices, among other things. However, privacy and the management of the data was the most promising aspects to overlap with a critical perspective of interaction design.
After taking the decision to focus on this specific topic within the IoT field, I used the references of the reports previously mentioned to dig deeper into this subject. This is where I found most of the referenced literature used in this project and also by sharing the project with the people involved within the field, I was suggested some new perspectives and further readings.
In parallel, I listed different design studios and projects that criticise the current situation in IoT such as Automato Farm [9], Iohanna Nicenboim [26], Unfit Bits [41] and Dunne & Raby [6]. Some of them are more focused on the technical exploration, which opens up the possibility to change the current situation through hacking the system and by using the vulnerabilities of it. Yet, on the other hand, there are projects closer to a more humanistic approach, which try to reveal the flaws in the dynamics between humans and technology.
In order to create the landscape where this research will coexist, I created an annotated portfolio with the most relevant projects I found in the previous step by placing them into a two axis graph which revealed the contradictions between the medium, from materialization to visualization, (X axis) and the level of awareness of the user related with the sharing of data, from awareness to unconscious (Y axis). This process helped lead to an understanding of the different features, which I had aimed at with the project, but the most important finding was the understanding of how others are working with or criticising the awareness and the affordance of the smart devices.
Also, during this part of the process I looked for several tools which could help me understand functional aspects such as the flow of the data and the manipulation communication with tools like Mosquitto [22] and Node Red [25] some of which will be further explained in the prototyping section.
Also, some services such as Spacebrew [38] and Shiftr.io [34] which are interesting ways to display the nature of the network and even though the target group for these services are
users with some knowledge in the technical aspects, it could be simplified for the common users to grasp its functionality, since the graphic interfaces of both projects reveal the need to create simple and understandable interfaces for this topics.
After the process of understanding the current situation within the particular chosen field, it was important to match the theoretical aspects and the related projects using the same strategy of annotated portfolio, but in this case, the main goal was trying to understand the similarities and differences between these two worlds. In this case, I used the chosen project in the first annotated portfolio and matched them with the strategies related to critical design found within the theoretical research. The outcome of this step is the creation of a proper criteria in order to continue with the next phases of the project.
4.2. Concepts
After the definition of the three strategies by Khovanskaya et al. [20] mentioned in the Theoretical Background section and its application to this project, I decided to work with the “Make it Malfunction” strategy. I chose this after a process of discrimination based on the criteria made by two concepts, awareness of the shared data and affordance of data.
The possible outputs mentioned before with the other two strategies try to raise the awareness by the visualization or the materialization of the data, but those strategies never give any new way of control of this data to the user. Therefore the chosen strategy which is tightly related to the control or in this case the uncontrolled data, fits better however it is important to refine this strategy in order to solve the research question fully.
The concept of malfunction is mostly attributed to the uncommon behaviour of technology, which drives the user into a situation where they lose all control over the expected output, since the technology is mostly encapsulated and the users only have access to the input and the output of the system. Therefore, for the sake of this project and in order to focus on the user side, the term of malfunction will be modified to data obfuscation which will act as the underlying concept and its different methods will be used as concept strategies.
4.2.1. Data Obfuscations
Data obfuscation is the process of hiding or modifying data, however when this process is executed, it is important to remain some usable data for future processes. When this technical process overlaps with the current situation of IoT from the user’s perspective, it is clear that as soon as the user acquires the smart device and agrees to the terms and conditions, it is impossible for them to avoid the intrusiveness of the builtin sensors. Therefore, at this point we must ask, what if the user has the possibility to control the flow of data in order to be empowered within the situation and balance the current hierarchy?.
The obfuscation technique comes from the engineering terminology and it is important to say that there are many strategies to execute this process such as shuffling, number variance,
encryption, masking out and many more. However, for this project I decided to work with substitution and nulling out since they are the most used strategies and both of them can be related to two common human actions, lying about something and hiding information from someone else.
Before going deeper into the two conceptual cases, it is important to explain the concept of the expert system and which is the perfect operation scenario is for this project. This scenario tries to understand how the artificial intelligence system is set within the vendor’s servers; however, I don’t have any proof that this structure corresponds to the actual one because this information is classified and it is impossible for the user to access it.
The artificial intelligence system in IoT is made by an expert system which is able to simulate the decisions made by a human expert in a specific topic using ifthen rules. These kinds of system live inside the vendor's servers and use the data coming from the user, overlaps it with data from other sources such as electricity consumption statistics, weather forecast and so on, and then execute the rules in order to take the decision which will return as the action, such as turning on/off the lights or changing the temperature setting of the air conditioning system.
The expert systems are made by two subsystems: 1. the knowledge base where all the unstructured information is stored, and in the IoT at home situation is filled with the raw data coming from the sensor inside the smart devices, 2. the inference engine which is the part of the system that use all the stored information or knowledge, and by applying some rules can create new knowledge. However, it is important to say that sometimes there is an additional element which filters the data in order to identify their source or even their quality, therefore for the perfect scenario where my project will develop upon, the filter will be before the inference engine.
4.2.1.1. Substitution (Lie)
This data obfuscation method allows substituting data with authentic looking values in order to preserve the authentic look and feel of the data base. In this case, it is important to understand the nature of the values such as the range of the information (maximum and minimum) and the average value over time.
This strategy gives back the status of administrator to the user since it is possible for them to have some degrees of freedom in the quality of the data that will be stored in the knowledge base. Thus, the responsibilities of some of the operations performed by the actuators of the system will be shared by both the users and the vendors.
Also, we can compare this strategy with the action of lying, since humans use this action as a way to keep themselves secure by creating a false statement that is presented as if it were true. Therefore, lying to the knowledge base is a defence mechanism that the users perform when they can’t hide from the surveillance features of the system.
4.2.1.2. Nulling out (Hide)
Nulling out is the method of sending a null value to a database, sometimes it could be understood as deleting but is actually more related to filling a space with a value that could be the unexpected result of an operation or presumably the expected result from an operation that has not been executed by the system yet.
This clarification is important when we try to overlap this concept with the action of hiding, since the efficiency aspects of the current IoT systems could understand this strategy as a malfunction from the sensors side; however, it could open up the opportunity for new degrees of freedom from the user’s interaction point of view.
For instance, in some cases, the user doesn’t want something to be registered by the sensors because it could destroy the map of data created on the server’s side, and since there isn’t any chance to turn off the devices or switch between a private or public state, it is important to empower the users with tools that can help them manipulate and control the quality of the data during the transference amongst the home and the servers.
4.3. Cultural Probe
In order to understand when the users are willing to lie or hide information to the IoT devices, a cultural probe was deployed during one week with 5 users, between 24 and 40 years old who live alone or share the apartment with another person.
The dynamic of the probe was based on the fictional scenario of a home full of smart IoT devices, such as a smart refrigerator, a smart lock, a smart lighting system and a smart TV. This four smart objects were chosen not only because are related with daily activities but also because are part of the most used devices at the home environment nowadays.
The cultural probes consist on an small card with a pen placed next to the object that could become smart, that way the users will ask themselves the question about lying or hiding information at the moment they are interacting with it.
Figure 12. Cultural probes installed in one of the participant’s house
The structure of the cards had five elements. Here is explanation of its the structure but the design of all the cards is part of the Appendix A:
● The situation: At the topleft of each card there is question that helps to introduce the participant into the situation where the smart device could be useful.
● The pro: This sentence explain the features that the device can provide to the user in order to solve the situation.
● The con: This sentence explains how the feature described before could be use by someone else against the user’s privacy.
● The survey: This section has three columns each with a sentence at the top. So, every time the participants use the device they have to ask themselves if based on the previous information they would prefer that someone else knows about it
(the
participant has no concerns about sharing the data),to lie to the device
(they want to
obfuscate the data by substituting it), t
o hide information to the device (they want to
obfuscate the data by nulling it out) and fill one space in the column.
● The news: At the topright corner of the card there is a QR code with a URL which will guide the user to a news articles that describes a particular situation where the security of the related smart device was exploited.
Figure 13. The parts of the cultural probe
Before the implementation of the cultural probes a meeting with the householders took place in order to inform them about the dynamics of the experiment and to talk with them about their current relation with smart devices.
After the deployment of the cultural probe, the participants were interviewed again in order to understand their answers and if the cultural probe had a long term impact into their relation with this technology. Since the focus of the interview was to create a comfortable and open conversation the use of a rigid questionnaire was dismissed, instead the conversation was guided following some keypoint such as: global comments about the week, talk about each device, talk about the pattern of the answer for each device, when this pattern change and why?, did the cultural probe generate more question about the topic?, did the cultural probe had an impact on other aspects of the daily life?.
For the sake of documentation of this project, the last interviews were recorded and uploaded to a private SoundCloud[37] account which are part of the Appendix B4. 4 The link to SoundCloud are part of the Appendix however some participants agreed to be part of the cultural probe only if the recordings were used for academic purposes, therefore to access to this recordings please contact the author.
