Applications of Artificial Intelligence in Lighting Systems for Home Environments

(1)

Applications of Artificial Intelligence in

lighting systems for home environments

Mikołaj Dobrucki

mikolaj.dobrucki@gmail.com

School of Arts and Communication (K3)

Interaction Design, Master's Programme (One-year) 15 credits

2020, Spring semester

(2)

Abstract

Artificial Intelligence, being recently one of the most popular topics in technology, has been in a spotlight of Interaction Design for a long time. Despite its success in software and business-oriented cases, the adoption of Artificial Intelligence solutions in home environments still remains relatively low. This study reflects on the key reasons for the low penetration of AI-based solutions in private households and formulates design considerations for possible further developments in this area with a focus on artificial light sources. The design considerations are based on literature review and studies of multiple home environments gathered through qualitative interviews and context mapping exercises. Health influence of lighting, multi-user interactions, and privacy-related and ethical concerns are taken into account as the key factors. The considerations have been validated with participants of the study through user testing sessions of a digital prototype that virtualises a home environment and explores some of the common light usage scenarios. The study argues that despite multiple efforts in this direction during the past three decades, the future of Artificial Intelligence in connected, intelligent homes does not lie in smart, autonomous systems. Instead, Artificial Intelligence can be arguably used to simplify and contextualise interactions between humans and their home environments as well as foster the development of parametric solutions for private households.

(3)

1. Introduction

1.1 Design context

Artificial Intelligence (AI), as a research area, has been receiving a wide coverage within design communities for a long time. Several research studies from the late 90s and early 2000s anticipated high adoption of AI solutions in various fields of everyday life, including both business and personal applications by the year of 2020 or earlier (Kidd et al., 1999).

Despite successful implementations of AI in software and certain professional industries, adoption in home environments is still relatively low (Liu et al., 2019). It can be argued that the transition towards intelligent houses has not happened as quickly as it was anticipated due to insufficient accuracy of AI algorithms, thus due to technical limitations. However, it is also believed that the low adoption of such solutions in home environments is hampered by privacy and design-related issues (Rogers, 2006). This study approaches the topic as a design problem and explores how Interaction Design perspectives could help in developing better applications of AI for home environments. Lighting systems have been chosen as a primary case of the study, based on their popularity as network-connected devices and their crucial role in home-based activities.

1.2 Research focus

This study focuses on the analysis of human behaviours in home environments related to the usage of artificial light sources and seeks potential applications of AI solutions within their scope.

When analysing human behaviours in home environments related to the usage of artificial light sources, three crucial factors are taken into account as a part of this study. Firstly, the study analyses people behavioural patterns in the context of multi-personal interactions. As the current research and state of the art in the field stays highly focused on single-user interactions and design solutions, this study will consider the role of social frameworks and habits. Further, the mutual influence of household members and potential areas where conflicts of interest may occur will also be within focus. Secondly, the study heeds that AI could be used for optimising psychological and wellbeing-related factors through parametric design solutions for home applications. Therefore, the influence of light on human health is taken into account. Finally, as the study is purely focused on people’s activity in private spaces, privacy and ethical concerns are also included as vital factors.

1.3 Research questions

The research question this study seeks to answer is:

How can Interaction Design perspectives contribute to the development of Artificial Intelligence applications in lighting systems for home environments?

The research question is supported by the following sub-questions:

How can Interaction Design methods be employed in design of lighting systems that support people’s mental health and wellbeing?

What are the ethical and privacy-related implications of using Artificial Intelligence in lighting systems for home environments?

(7)

1.4 Process and aim of the study

To answer the above research questions, a literature review in related areas was made, followed by a series of background studies to achieve a better understanding of the discussed topics and formulate a set of design opportunities and hypotheses. Based on these hypotheses, a digital prototype was created. The prototype was tested with multiple participants in order to investigate possible opportunities for AI applications for lighting systems in private households.

The aim of the study is to provide insights into everyday usage of artificial light sources in multi-person home environments and to identify opportunities for applications of AI. Through analysis of the generated insights, the study identifies common patterns and behaviours related to the focus of the study as well as crucial factors that should be taken into account when designing lighting solutions for private households that employ AI as one of its technologies.

1.5 Contributions to the field of Interaction Design

In a reference to the literature review of common trends related to applications of AI in home environments from the past three decades, the study attempts to explain why the current state of the art within this field has not matched the predictions. Considering further efforts towards implementation of those predictions as either troublesome or redundant, the study seeks alternative applications of AI that would have a higher potential of positive influence on life quality in multi-person home environments.

Through the analysis of multiple social and design conflicts related to the usage of artificial light sources in home environments, a few design considerations are proposed. The considerations, aimed as knowledge contribution in the field of Interaction Design, can be used as guidance for future developments of AI applications for home environments.

1.6 Limitations of the study

The study was conducted within a period of 9 weeks, starting 30 March and ending 28 May 2020.

Within this period a literature review was limited to the material available free of charge through the Malmö University Library and open online resources.

The conducted studies were limited to a group of 7 participants aged 23-65. Most of the participants were young adults living in multi-generation households counting 2 to 5 people. 4 of the participants identified themselves as women and 3 of them identified themselves as men. All participants originated from Northern European, Eastern European and Eastern African ethnical backgrounds. The diversity among participants was important to ensure variety of family models and habitual patterns related to usage of light sources in home environments.

Due to the situation of the COVID-19 pandemic, all studies were conducted using remote methods and online resources, including remote interviews and sessions based on digital prototypes, without the use of any real in-home implementations and methods of direct inquiry. The prototype was limited to two scenarios of a single multi-user household due to time and technical limitations of the study.

(8)

1.7 Ethical considerations

The study involved 7 human subjects who took part in interviews, context mapping exercises and user testing sessions. All participants gave consent to participation in the study being aware of its scope, goals and methods and have been informed about the right to opt-out at any point. No personal data, images or recordings have been collected throughout the study.

1.8 Structure of the report

This report summarises research knowledge crucial to the focus of the study, describes the design process behind it, and elaborates on possible applications and further steps.

Section 2 “Background” consists of 5 subsections. Each subsection is dedicated to a different field of knowledge related to the research focus. Each of the discussed topics is supported by examples of related works, studies and commercial products and technologies.

Section 3 “Methods” describes and explains the choice of the methods used throughout the study. The first part of the section introduces the concept of Research through Design, the fundamental methodology of the study, followed by the methods used for background studies and data collection. Subsequently, the design methods used to create and test the final prototype are described.

Section 4 “Process” explains what happened during the study in chronological order. It focuses on design activities that resulted in the creation of the final prototype and presents the findings and observations related to particular stages.

Section 5 “Results” elaborates on the findings of the study through a set of proposed design considerations. Each design consideration is related to the original focus of the study and is examined in the light of previous research and literature review.

Section 6 “Discussion” takes a critical look at the presented findings and talks more about what could have been done better and what would be the next steps if the study were to continue. It also summarises possible contributions of the study to the field of Interaction Design, based on the findings and insights generated throughout the study.

(9)

2. Background

2.1 Artificial Intelligence

This section of the report covers knowledge related to the topic of AI, relevant to the aims of this study. Subsection 2.1.1 describes a relationship between developments in the area of AI and ubiquitous computing as an essential concept related to the implementations of AI in home environments in the fields of Human-Computer Interaction (HCI) and Interaction Design. The next two subsections discuss the reasons for the low adoption of AI solutions in home environments as the main motive behind the focus of this study. 2.1.2 summarises technological issues and challenges, whilst 2.1.3 describes related design problems. Finally, subsection 2.1.4 explains the difference between the theoretical concept of ubiquitous computing and smart technologies as a supreme trend in commercial products for home environments.

2.1.1 Artificial Intelligence & ubiquitous computing

The discussion on AI in the context of home environments during the past 30 years have been inseparably connected with a concept of ubiquitous computing (UbiComp). Mark Weiser, who coined the term in the late 80s (Markoff, 1999) describes ubiquitous computing as an “idea of integrating computers seamlessly into the world” (Weiser, 1991).

Following Weiser’s lead, multiple sources predicted ubiquitous computing to become a dominant trend in technology during the following 15 to 20 years. According to International Data Corporation, development of computing was supposed to transition from the age of personal computers into the times of distributed devices between 2005 and 2020 (IDC, 1996). Those predictions were closely followed by Mark Weiser himself who in 1996 introduced a new concept of calm technology (Weiser & Brown, 1996). Weiser expected that “the results of the massive interconnection of personal, business, and government information will create a new field, a new medium, against which the next great relationship [between people and technology] will emerge.” The realisation of this concept anticipated not only rapid advancements in miniaturisation and raise of the computing power of electronic devices but also major innovations in the field of AI.

The concepts of ubiquitous computing and calm technology were hardly left unheard. They made a profound influence on the works of multiple research centres, companies and governments (Streitz, Kameas, & Mavrommati, 2007). Interconnected during the following years with the concept of Ambient Intelligence (AmI) (Cook, Augusto, & Jakkula, 2009), research and development in this area resulted in a plethora of solutions for all possible industries including home, healthcare and business (Sadri, 2011), often commonly referred as Internet of Things (Gubbi, Buyya, Marusic, & Palaniswami, 2013).

2.1.2 Ubiquitous computing at home: technological perspective

Nonetheless, while the popularity of UbiComp and AI solutions among public and business subjects appears to slowly gain its momentum (‘IoT Signals. Summary of research learnings’, n.d.), adoption in home environments still stays relatively low and definitely a far cry from the Weiser’s vision of the home full of “clocks that find out the correct time after a power failure, microwave ovens that download new recipes, […], paint that cleans off dust and notifies you of intruders, walls that selectively dampen sounds” and so on (Weiser & Brown, 1996, p. 5). For that, there are multiple reasons.

On the one hand, there are certain technical difficulties related to setting up smart building systems. For businesses, operating on a larger scale and with bigger budgets, it is much more efficient to outsource a setup

(10)

of a smart system to industry professionals. What would seem too costly for most families, is often within a comfortable reach of both smaller and bigger companies.

On the other hand, if we compare private homes to offices, designing a UbiComp system for a home environment is typically much more challenging than in business-oriented cases. To work properly and effectively, such a system has to be driven by a certain set of objectives. In offices, these goals are usually easy to define, and in the vast majority of cases boil down to energy usage optimisation and improving employee’s performance through automation of certain routines and ensuring optimal working conditions. But when it comes to private households, such goals tend to be much more obscure and indeterminable. Making adequate decisions is not only a matter of optimising revenue but a matter of personal health, ethics and relationships (as explained later in 2.2).

This also finds its reflection in academic studies. Most of the current research in the area of AI focuses on business applications (Liu et al., 2019, p. 4). Disproportion between the coverage of home- and business-related studies is most likely business-related to the fact that during the last decades, business-business-related studies were both in higher demand and easier to fund by commercial partners interested in the case. Nonetheless, privacy, limited access to private households and baffling complexity of related ethical concerns also made this area much harder to penetrate and therefore could potentially hamper researchers’ interest.

2.1.3 Ubiquitous computing at home: design perspective

Multiple non-technical issues should also be noted as reasons for the low adoption of UbiComp solutions in home environments. As Michael C. Mozer noticed, “Smart homes have failed to become a reality for two reasons. First, inhabitants are fairly satisfied with traditional home controls. Second, the obstacle to understanding new interfaces is high.” (Mozer, 2005, p. 2).

Many calm technology concepts never became popular not because of their technical complexity but plainly because of lack of people’s interest and/or sufficient market to justify mass production. Creating a “microwave oven that download new recipes” (2.1.2) hardly sounds like a challenge. “Clocks that find out the correct time after a power failure” do exist (Brown, n.d.). Nonetheless, a vast majority of houses have neither. There is simply not enough value in replacing a perfectly working clock with a smart one for most people to be bothered.

What is making people interest even frailer is the amount of effort required to properly set up such devices. Often, they require a lot of patience and technical know-how to get them to work. As a result, common awareness of a high risk of failing in such a process and fear of unknown decrease people’s interest.

On the contrary, some of the Weiser’s ideas would undeniably be useful but are not that easily feasible. This is where issues related to AI emerge of crucial importance. Creating “walls that selectively dampen sounds” sounds intriguing but how exactly would that idea work in practice – nobody knows (further analysis of this concept in 2.2.2). If we take a look at recent UbiComp solutions in lighting systems, they also prove to be highly challenging to implement (Karapetyan, Chau, Elbassioni, Azman, & Khonji, 2020). What is commonly mentioned, smart systems to date are still simply not “smart” enough to fulfil Weiser’s visions (Rogers, 2006). Multiple new technologies and concepts, such as fog computing (Bonomi, Milito, Zhu, & Addepalli, 2012) are under development to tackle these issues but we are still far from reaching the levels of “smartness” Weiser was craving for.

To sum up, the current progress in the field of AI solutions for home environments is struggling with both technical and non-technical issues. On the one hand, we have plenty of so-called “smart devices” that hardly anybody cares about or wants to use. On the other hand, the most remunerative applications still seem out of our reach. At the same time, we are highly unsure what the future of an intelligent home should actually look like, and if we need intelligent houses at all. As Yvonne Rogers argues, “progress in UbiComp research

(11)

has been hampered by intractable computational and ethical problems” and “we need to begin taking stock of both the dream and developments in the field” (Rogers, 2006, p. 406).

2.1.4 Ubiquitous computing vs. smart computing

One distinction that deserves a clarification here is a difference between ubiquitous computing and smart

computing as these terms, even though related, should not be used interchangeably. It is especially important

for the understanding of this study, as while all its technological assumptions are based on the current state of the art of smart computing, the goal of the study is to produce design considerations applicable first and foremost to AI-oriented and ubiquitous computing projects.

In a world full of smartphones, smartwatches and smart TVs, we are so used to the concept of “smart” that we rarely ask ourselves what this term stands for. Smart system not being smart enough to “selectively dampen sounds” is rather an obvious reference to the capabilities of an AI to perform such actions. However, in many common cases, being “smart” has absolutely nothing to do with AI. The main differences between smartphones and regular, “dumb” phones boil down to stronger hardware capabilities and ability to install third-party applications. Modern TVs earned their claim to be called “smart” just by an addition of an internet connection. The same thing happens to all the other so-called smart home devices.

Talking of smart homes, the understanding of being “smart” usually goes a bit further. We not only expect a smart home to give us a remote control of light bulbs and kettles but also to offer us some level of automation. Smart home frameworks such as Apple HomeKit, Xfinity Home or Google Nest (‘iOS - Home’, n.d.; ’Google Nest Connected Home’, n.d.; ‘Xfinity Home Security’, n.d.) allow us to create various automation models. Most of them rely entirely on binary operators triggering certain sets of predefined actions (e.g. “switch on the lights in a hall and kitchen when I entered the front door” or “lower heating power when nobody is home for more than 3 hours”). Well-designed automation can be both convenient and more sustainable. Nonetheless, programming actions based on predefined variables still has nothing to do with AI.

As a result, when talking about “smartness” in a context of AI and contemporary smart devices, we are ultimately referring to two utterly different meanings of this term. To fulfil Weiser’s visions, we would need a home not only to perform predefined scenarios but also to independently create and modify its own. At a glance, this may still seem like a technical problem – after all we just need our AI systems to get “smarter”. But at this stage, “smartness” means much, much more than better connectivity and computational power. It means understanding people’s moods, feeling and emotions. It means usage of algorithms based on behavioural science as well as consideration of complex privacy and ethical issues.

2.2 Privacy and ethical concerns

2.2.1 UbiComp and privacy

Building AI solutions for private spaces always brings privacy implications. If we get back to the early days of UbiComp discussions, Marc Langheinrich rightfully predicted that computers will become more and more ubiquitous and invisible, so “we are going to have a hard time in the future deciding at what times we are interacting with (or are under surveillance by) a computing or communication device” (Langheinrich, 2001, p. 273). This leads to serious privacy concerns as it leaves a vast field for personal data collection without consent.

Langheinrich does not advocate for total, perfectly closed privacy and security. Instead, he offers us a vision of moderate privacy that prevents “unwanted accidents – data spills of highly personal information that people who have never asked for it suddenly find at their doorstep” (Langheinrich, 2001, p. 281). His

(12)

perspective was found very useful for the development of this study. Building a UbiComp home system without any data collection seems almost impossible or at least highly limited. At the same time, privacy concerns should be taken very seriously and lead to the least possible associated risk.

10 years later after Langheinrich’s works, the discussion continues, still very current and needed. At that time, Paul Dourish argues that privacy is “not merely ways in which information is managed but also ways in which social actions are achieved” (Dourish & Bell, 2011, p. 160). Dourish says that all the privacy-related behaviours must be put in a relationship to their social and cultural context. That is yet another contribution valuable to the focus of this study, as social and cultural contexts of privacy are more important at home than anywhere else.

Finally, it is worth evaluating one of the recent works following Langheinrich’s efforts. The last few years cast a lot of new lights at digital privacy, especially considering both large-scale media scandals such as Facebook– Cambridge Analytica data scandal (Wong, 2019) and widely commented attempts towards new privacy laws (The EU General Data Protection Regulation (GDPR), 2017).

Influenced by these events, more designers started to advocate for avoiding data-collection and a greater focus on what data are being stored and how. Daphne A. Muller’s and Pierre Lévy’s argue that moving the weight of decision making from human to machine is not necessarily a boost for comfort (Muller & Lévy, 2019). They warn us against the future home turning into a black box, “in which the users lose control over the communication between the devices, and get surprised or even feel haunted by them.” This does not mean that the authors are completely against the usage of AI solutions per se. What they propose is more a matter of reduction of data collection, for example through limiting it to a bare minimum of a “training phase”, so the data are collected only when the algorithm is being trained and deleted immediately afterwards.

2.2.2 UbiComp and ethical implications

Giving control over our houses to AI is not only a matter of privacy issues but also generate serious ethical implications. To better understand why, the study proposes a closer evaluation of an earlier example of “walls that selectively dampen sounds”. Dampening sounds might be useful, especially when you have an annoying neighbour or a hyper-active teenager in your own home but obviously, not all sounds should be dampened. From the simplest cues about other people’s behaviour (“I won’t go to the bathroom now as I hear someone’s already there”), through calling family members for dinner, to being able to quickly react on worrying sounds, there’s plenty of cases where certain sounds are crucial to stay. However, what is usually trivial for a human brain to assess (“Is it my husband working out or my grandpa falling down the stairs?”) pose insurmountable problems for the most advanced AI algorithms. And, even if we were able to train a smart system to correctly identify a full variety of possible sounds, we would still stand in front of substantial ethical and moral dilemmas.

First of all, identifying and evaluating sounds would inevitably mean smart systems “listening” to us everywhere and at all times which already may sound concerning from an ethical point of view. Moreover, introducing autonomous decisions based on what a system “heard” makes it even more problematic. If your daughter is loudly crying in the middle of the night, should this sound be dampen so you can conformably sleep or should it be “left as it is”? Or maybe a smart system should send a notification about a possible need for intervention? Can a person deliberately “mute” their walls so they can cry without a risk of being heard? Can they regulate what kinds of sounds are being dampened in the whole house? If so, what if another household member has different preferences? How to resolve such conflicts of interest?

For now, arguing if AI will ever start to understand this kind of problems is barely a technical question and much more of a philosophical dispute. From such a perspective, the gap between the original UbiComp concepts and the current reality of smart devices is even more undeniable. Reflecting on the current state of

(13)

the art in the field, we could say that while Weiser expected us to build a world of empathic, wise technologies, so far we are still struggling with making them just a little bit smarter.

2.3 Smart home technologies

2.3.1 Smart home: commercial products

It is important to notice that various efforts towards automation of indoor amenities date much earlier than the concept of UbiComp. For example, controlling the qualities of artificial light autonomously was a subject of an American patent from 1993 which claims the rights to a concept of “Lighting system[s] for controlling the colour temperature of artificial light under the influence of the daylight level” (U.S. Patent No. 5,721,471A, 1993). Following the trends mentioned in section 2.1.2, the patented solution was meant for office spaces with no indication of possible private applications. However, it is still a proof of interest in such areas dated as long ago as the origins of calm technology. The patent is currently held by the US Philips Corp, one of the biggest producers of smart home lighting products.

Currently, the primary smart products system offered by Philips is Philips Hue. The system focuses entirely on lighting. Philips Hue light sources include light bulbs, light strips and lamps. The system allows controlling light temperature, colour and brightness. It also offers multiple automation features, mostly dedicated to switching on and off certain configurations based on a given factor. For example, the light can turn off automatically when you leave the home or slowly brighten up during the morning to wake you up. It is possible to turn the lights on and off at sunset and sunrise by sharing the location of your home with the system. It is not possible though to sync the temperature of the white light with the temperature of daylight although you can achieve such an effect on your own with a bit of tinkering. Philips Hue offers configurable scenarios related to mental health and well-being such as light settings for falling asleep, waking up, concentration or meditation. Some of the Philips Hue features consider multi-user control of the system, e.g. other household members can share their location with the system so you will not switch off all the lights when leaving the home with other people inside. None of the available solutions is autonomous though, nor does it make any use of AI technologies.

(14)

Another popular smart home system available on the market in Nest. Nest products include a wide variety of smart home solutions such as cameras, doorbells and smoke/CO detection alarms. However, their most famous and acclaimed product is admittedly the Nest Learning Thermostat. Nest’s thermostat significance marks the fact that it was the first widely adopted smart home product that learns from human behaviour and autonomously builds schedules fitting the preferences of inhabitants. The excellence of Nest’s innovation (very unique at the time it was released to the market) lays in a fact that it does not require any trackers or sensors to program its behaviour. Instead, it learns from manual changes made to the settings over time.

In 2017, IKEA, the world's largest furniture retailer, also entered the market of smart products by introducing their own smart lighting and sound solutions. IKEA’s products also offer an ability to control light qualities such as colour and intensity. However, their automation capabilities are even more limited than Philips’. Again, no AI-related solutions are involved.

2.3.2 Smart home: research projects

Multiple research projects have been conducted in recent years that analyse struggles and possible opportunities for the AI usage at home (Harper, 2003). One of the most notable projects is the Adaptive Home (Mozer, 2005). Similarly to the Nest Learning Thermostat, “the intelligence [of the Adaptive Home] arises from the home’s ability to predict the behaviour and needs of the inhabitants”. Implementation of the project covered a wide variety of amenities such as lighting, heating and kitchen devices. The intelligence of the home is based on relatively simple, solid assumptions such as “When the inhabitant awakens at 4 a.m. and climbs out of bed, the home predicts a trip to the bathroom, and the bathroom light is turned on”. However, all the examples given by Mozer focus on single-user scenarios. Mozer does not discuss conflicts of interest between inhabitants but makes a point about conflict of interest between inhabitant’s comfort and sustainability which formed one of the crucial hypotheses of my work at a later stage.

Another interesting project realised in similar times was Aware Home by The Georgia Tech (Gonçalves, 2001; Kidd et al., 1999). The project again was focused on single-user interactions. Authors say that their priority was to create an intelligent, automated home whilst preventing information overload, avoiding invasion of privacy and creating practical UbiComp applications for the everyday user. The project was focused on senior adults and anticipated observing and learning behaviour model. The authors argue that we can teach an algorithm the preferences and patterns in people’s behaviour by analysing their past actions. The general findings from the project are very similar to findings of the Adaptive Home project. What makes them unique is a focus on design for the elderly. Kidd clearly illustrates that at least in some contexts data collection for learning behaviours is well justified. His insights in this field influenced some of the hypotheses of this study.

Worth mentioning is also a House_n project that illustrates MIT’s efforts in the area (Intille, 2002). What is interesting about this initiative is that its goal was not to automate actions but to help people make the right decisions. We can assume that computers will get much smarter, but people will most likely not. As a result, “there is a fundamental problem here: the more complexity the algorithms consider when making decisions, the less transparent those decisions will be to the homeowner” (Intille, 2002, p. 81). At the bottom line, Intille’s home is not a typical smart home but a teaching home – a home that teaches its inhabitants how to control their own environment.

What is common for all the projects mentioned above is that they are highly focused on single-user situations. Analysing these examples led to the conclusion that the aspects of UbiComp research that require more attention are multi-user interactions and social implications.

(15)

2.4 Multi-user interactions and social implications

Focus on individual interactions makes it much easier to conduct studies and get clear research outcomes. Unfortunately, by omitting a multi-personal aspect, it is easy to stay oblivious of crucial problems and nuances. As long as identifying optimal settings for an individual can already be extremely hard to achieve, if we try to mix preferences of multiple people in a single space at the same time, the challenge becomes even greater.

Most of the households around the globe have multiple members. It is inevitable for multiple people living under one roof to have different needs, limitations and personal preferences. Also, the personal context of multiple inhabitants can vary at a single point of time. This leads to unavoidable conflicts of interests which not only create design challenges for multi-user interactions with a shared system but also makes UbiComp solutions much harder to optimise and teach predictable patterns.

2.4.1 Multi-user interactions at home

Creating complex, robust UbiComp systems for home environments face non-trivial design challenges. Many of them still have not been properly solved in any existing products. Designing such systems for interaction with multiple users of different needs, habits and limitations is undeniably tricky, even if the system is not supposed to be autonomous at all. As documented within background studies (4.1.2), multi-person households are expected to experience various conflicts of interest and situations of unwanted mutual influences. To some extent, improving the experience of such situations is a design problem. Various design approaches can be taken to support people in achieving more flawless use of shared products, for example through making actions of others more visible and therefore helping people to take each other into consideration when interacting with a home.

An important example of how such an approach can be implanted comes from Niemantsverdriet et al. who approached this challenge using a framework of social translucence (Niemantsverdriet, Broekhuijsen, van Essen, & Eggen, 2016). The framework is based on three rules: visibility, awareness and accountability. Authors argues that awareness of each other’s actions, other’s intentions and the ability to predict the effect of your interactions on others are crucial to building a complex, multi-user smart home system. Authors present a case of implementing social translucence through design of a lighting system that is closely related to the focus of this study. Nonetheless, none of their works anticipates any involvement of AI which is being explained through unfeasibility of building such solutions. From the perspective of this study, I am more than inclined to agree that “creating a system that can interpret and predict the context and successfully coordinate interactions for all users in all situations is virtually impossible” (Bellotti & Edwards, 2001). However, I would argue that this is not the only role that AI could play at home and this is definitely a perspective I am lacking in Niemantsverdriet’s work. The alternative approaches are presented within the results of this study.

2.4.2 Social implications of UbiComp

Edwards and Grinter analysed various challenges related to implementing UbiComp solutions at home during their research conducted for Xerox Palo Alto Research Center in 2001 (Edwards & Grinter, 2001). One of the challenges they describe are social implications. Authors suggest that changing the ways our homes work not only save us time and labour but also change our social habits and behaviours. They compare introduction of UbiComp solutions into the home to the introduction of such novelties as washing machines and indoor bathrooms. This point is crucial when it comes to analysing AI applications in home environments as it is important to consider not only their potential most direct consequences but also how they can change our much more complex behaviour patterns and social habits in a long run.

(16)

To better understand what could be the scope of such consequences in case of lighting systems, we first need to understand what is the actual role of light in our everyday life at home.

2.5 Influence of lighting on human health

Artificial lighting at home is crucial for carrying out everyday tasks during the dark hours and by complementing daylight in the areas and/or during the times when the access to it is limited. Apart from practical purposes, lighting can also have a tremendous impact on our mood and mental health (Bedrosian & Nelson, 2017).

Nowadays, the majority of people living in western urbanised societies spend most of their lives indoors. Most of this time is spent at home. Considering the current circumstances of a COVID-19 pandemic, these numbers may reach new historic highs in the foreseeable future. Therefore, it is not an exaggeration to say that optimising the living conditions through lighting technologies can have a profound impact on people’s lives as a whole.

Research at the intersection of technology and human health is crucial to the development of modern medicine and healthcare. However, it also has a long history of being problematic and controversial. Given the importance and troublesomeness of these topics, this study also takes them into consideration as vital factors too important to ignore.

2.5.1 Light ergonomics

Allowing some level of simplification, we can say that domestic lighting is used for two main purposes: to produce a comfortable atmosphere and to provide a good environment to facilitate visual tasks (Nakamura & Karasawa, 1999). Modern smart lighting systems are trying to make the best possible job at facilitating these tasks by allowing their users to control various qualities of artificial light such intensity and hue, as well as to control the penetration of natural light inside the building. As for positioning, hue, and intensity of light being the most common adjustable parameters of artificial lighting found in both research and commercial projects (2.3.1; 2.3.2), these variables are also within the main focus of this study.

Studies in the area of light qualities such as hue and intensity trace back to the times before the Second World War, with the Kruithof curve being the best-known concept related to artificial lighting and psychology from that period (Davis & Ginthner, 1990). Kruithof suggested that there is a certain spectrum of white lighting that people perceive as visually pleasing. Despite the obvious shortcoming of evidence and accuracy in his findings, the idea itself is still commonly exploited in the design of lighting solutions.

What is problematic about the Kruithof’s theory is that the “ideal” lighting intensity and temperature not only depend on many external factors such as seasons and time of the day (Kakitsuba, 2016) but also are very subjective and vary from person to person. This leads us into a potential conflict between the ability to customise a lighting system to fit personal preferences and compromising the needs of multiple people at the same time. This finds a reflection at the later stage of this study.

Insights on this subject can also be drawn from related studies, covering the influence of other artificial light sources such as screen light. According to Scott Monteith, “blue light exposure may […] increase alertness, and interfere with sleep […]. Areas of concern in mental illness include the influence of blue light on sleep, other circadian-mediated symptoms, prescribed treatments that target the circadian system, measurement using digital apps and devices, and adolescent sensitivity to blue light.” (Bauer et al., 2018). Those and other findings resulted in the creation of various software solutions trying to reduce exposure to blue light during the usage of screen-based interfaces. One of the most advanced solutions currently available on the market is TrueTone technology build-in all recent Apple devices. The algorithms behind iPhones and Macs screens are

(17)

not only trying to reduce the amount of emitted blue light but also to adjust the colour and intensity of displays to match the ambient light of surroundings so that images on the screen appear more natural. These findings are important from the perspective of this study as they suggest that similar solutions might be developed for home lighting in the future, which may result in nontrivial design problems that this study is trying to address.

2.5.2 Light and mental health

The influence of artificial light on our health and wellbeing does not end at qualities such as hue and brightness. According to Bedrosian and Nelson, the biggest aberration that artificial light brings to our lives is not related to light qualities but to timing and periodicity of exposure. It disturbs predictable periods of light and dark driven by sun and moon which influences our cardiac system and interferes with our natural day and night cycles. This can have a crucial impact on our sleep behaviours, hormone system, cellular functions and gene expression (Bedrosian & Nelson, 2017).

This is more than important in the context of designing lighting systems for multi-user interactions. We need to remember that whatever actions related to artificial light we are performing, they influence not only our health but also the health and biological rhythm of people we are living with.

From a plainly biological perspective, we are only starting to understand how light factors influence human health. Contemporary research of mammalian retinas shows that apart from sensors responsible for visual effects, our eyes also contain a much different kind of photoreceptors that regulate Non‐Image Forming (NIF) body functions such as circadian rhythms, alertness, well‐being and mood (Huiberts, Smolders, & de Kort, 2016). In other words, our eyes “see” more than just the visual pictures we directly experience. These findings help us understand other light-related phenomena about the human body, including the fact that generally people prefer daylight to artificial light much more when it comes to optimising their comfort, performance and satisfaction.

From these findings, we can seek novel design opportunities. For example, we can draw a conclusion that making indoor environments as close as possible to the experience of natural light can have a positive impact on human health. This may be of special importance to people spending an extensive amount of time indoors. Here, AI could be used to adjust lighting and create a sense of continuity between the inside of a room and the outdoors. As indicated by Yoon and Ishida, “we can experience a sense of continuity between the inside of a room and the outdoors by adjusting the illuminance and colour of the artificial lighting” (Yoon & Ishida, 2000). Such ideas are important for the direction of this study as they offer possible applications of AI solutions within home environments.

(18)

3. Methodology

This section briefly describes design methods used throughout the process.

3.1 Fundamental methodology: Research through Design

The underlying approach to this study was grounded in the practice of Research through Design (RtD). John Zimmerman defines Research through Design as “a research approach that employs methods and processes from design practice as a legitimate method of inquiry” (Zimmerman, 2010). As the main reason for Research through Design coming into the spotlight of the research community Zimmerman sees a unique RtD ability to solve “wicked problems”. I believe that finding a proper approach towards the usage of AI in the most private aspects of people’s lives is exactly one of such “wicked problems” that “cannot be easily reduced” (Zimmerman, 2010, p. 310), hence a choice of this methodology.

What is unique about RtD in comparison with other design methodologies is that rather than focusing on creating a design artefact as the main contribution, RtD’s goal is producing new theory for design. Therefore, the main goal of this study was to produce research-based design considerations that could contribute to the theory within a field of Interaction Design. These considerations should be able to provide guidance and be applicable in other design projects related to creating multi-user interaction in home environments that include the applications of AI.

3.2 Methods for background studies & data collection

In order to produce valuable design consideration, a digital prototype was built and tested with the study participants. The prototype was meant to provoke thoughts on potential usage of AI in home environments. The experiences from using the prototype were meant to elicit a discussion and lead to opportunities for finding new Interaction Design paradigms.

To achieve a better understanding of the lighting systems in home environments before creating the prototype, a series of background studies were conducted, including interviews and context mapping exercises.

3.2.1 Qualitative interviews

A series of qualitative interviews were conducted at the beginning of the design process to gather data about the usage of lighting in home environments with multiple household members. Interviews, as a method of enquiry, vary from structured, through semi-structured, to unstructured, depending on how they are prepared and conducted (R. Edwards & Holland, 2013). In the case of this study, all the interviews were following a structured or a semi-structured approach as their main goal was to gather data and observations rather than evoke wide, deeper discussion at this point of the process.

All the interviews were conducted remotely due to the circumstances of the COVID-19 pandemic. As the study does not include any field studies or other research methods conducted onsite in the home environments of participants, all interviews paid special attention to the details and characteristics of the discussed spaces. Furthermore, all the interviews were discussing the usage of lighting in the environments which the author of this study was already familiar with (was there physically in the past) what helped to correctly interpret interviewees’ responses.

(19)

The list of questions asked during the interviews, as well as an example of a set of responses can be found at the end of this report, in Appendix 1 (8.1).

3.2.2 Context mapping

Context mapping exercises were conducted with the same set of participants, as a follow-up of the qualitative interviews. According to Sanders, the main focus of context mapping is to “gain deeper insight into the needs and dreams of prospective users of new products” (Visser, Stappers, van der Lugt, & Sanders, 2005). Following this argumentation, during this study, the main goal of context mapping exercises was to get a better grasp of people’s tacit and latent knowledge and therefore reveal insights that participants would not be able to verbalise themselves during more direct methods of enquiry such as qualitative interviews. As a sensitising tool for the context mapping exercises within this study, workbooks were used. Similarly to the concept of Cultural Probes (Gaver, Dunne, & Pacenti, 1999), workbooks are meant to be used as playful objects that participants can interact with alone in their own time. As described by Sanders, workbooks are “small booklet[s] with open-ended questions to answer, things to draw, such as ‘draw a diagram of the things you did while travelling to work this morning’.” Due to the pandemic circumstances, all artefacts were prepared digitally, using Google Sheets.

The template of the workbook, along with an example of a set of responses can be found at the end of this report, in Appendix 2 (8.2).

3.3 Methods for developing and testing final prototype.

3.3.1 Sketching

Sketching was chosen as a fundamental method of generating ideas for a prototype. According to Buxton, “one of the key purposes of sketching […] is to provide a catalyst to stimulate new and different interpretations” (Buxton, 2011, p. 115). In the scope of this study, sketching was used to develop various perspectives on the topics being investigated and repeatedly reframe analysed problems. Sketching with various digital technologies was also helpful to find the most appropriate medium for the design prototype developed during the next stage.

3.3.2 Prototyping

To avoid visiting other people’s household, the final prototype of the study was prepared in a digital form and tested remotely. The prototype focuses on “Role” side of the system with a partial inclusion of a “Look and feel” aspects (Houde & Hill, 1997).

The goal of the prototype was to be tested with users and facilitate discussion rather than develop any actionable design solutions. The final prototype was developed using an online design editor, Figma. Examples of screenshots of the prototype can be found at the end of this report, in Appendix 3 (8.3). 3.3.3 User testing

The prototype was tested and discussed with multiple study participants. The way the testing was conducted was heavily influenced by the principles of contextual inquiry (Beyer & Holtzblatt, 1998, p. 41-67). It also can be connected to the practice of A/B testing (Azevedo, Alex, Montiel Olea, Rao, & Weyl, 2018) as users were able to compare multiple lighting configurations and explore multiple scenarios of how AI could be used in the presented stories.

Testing was based on completion of staged activities performed by participants in a virtual home environment inhabited by multiple users where the lighting system could be controlled by AI. The virtual

(20)

home environment should be understood as a computer simulation that a single participant can interact with and perform certain actions miming everyday in-home activities related to the usage of light sources.

3.4 Methods for synthesis of collected data

For the synthesis of collected data, methods such as Insight Combination and Concept Mapping (Kolko, 2010) have been used to formulate the final design considerations of the project. They introduced a critical look at the insights gathered during the earlier stages of the process.

As many of the responses gathered through qualitative interviews and context mapping exercises repeated certain patterns, behaviours and observations, the synthesis process allowed to identify common themes and prioritise collected data. It was a crucial step leading to better understanding of relationships between lighting systems in home environments and their users which allowed the formation of final design considerations.

(21)

4. Process

The underlying structure of the design process was inspired by a popular Double Diamond model (Figure 2). The first part consisted of in-depth literature studies on the chosen topic and related research fields. Next, several research methods were employed to gather data. From that point, information was synthesised into more general observations and statements. At the following stage, a prototype was built, preceded by a research of available technologies and prototyping approaches. After choosing the most appropriate solution, the prototype was developed and tested which allowed validation of the hypotheses and formulating conclusions of the study.

4.1 Exploration

4.1.1 Literature review

Considering the focus of this study as related to multiple knowledge areas, literature review turned out to be a crucial part of its initial stage. Thorough studies of Interaction Design literature on AI, ubiquitous computing and interactive technologies in home environments created a groundwork for this review. It was followed by a research of available smart home solutions and related research projects which, as a result, helped to choose artificial lighting as primary case study of the project. Subsequently, a review of related works on lights ergonomics and the influence of light on physical and mental health was conducted. Those readings also highlighted the importance of privacy and multi-user interactions in the context of the study which eventually helped to narrow the focus and define appropriate research questions.

(22)

4.1.2 Background studies

After completing the literature review, the focus of the study shifted towards gathering original insights about the usage of light technologies at home.

The background studies have been conducted on a group of 7 participants including:

✴ A 25 years old participant of Somali nationality living in a flat with parents and five siblings ✴ A 65 years old participant of Polish nationality living in a flat with her daughter

✴ A 25 years old participant of Scottish nationality living in a shared student flat with one other person ✴ A 24 years old participant of Polish nationality living in a house with parents and grandparents ✴ A 24 years old participant of Danish nationality living in a house with her brother and parents ✴ A 21 years old participant of Polish nationality living in a flat with two siblings and their grandmother ✴ A 29 years old participant of Polish nationality living in a flat with her mother

All studies have been conducted remotely, within a total period of 2 weeks in April and May 2020.

Interviews conducted at this stage, as well as context mapping exercises, provided a deeper understanding of how people use lighting in their homes and what role it plays in their lives which was a vital insight for choosing a further direction of the study.

Firstly, fieldwork studies helped to get a better understanding of people awareness of light qualities and functions. The points listed below summarise the most important takeaways from the interviews:

✴ Light temperature and intensity were important factors for all participants. Participants could notice a significant difference in their mood and efficiency depending on a type of light used

✴ Participants considered warmer lights as meant for mood creation, whilst colder lights as more applicable to tasks that require focus and precision

✴ All participants stressed the importance of natural light as vital for their wellbeing and performance during cognitively challenging tasks.

Secondly, the studies identified the most common functions and applications of artificial light as the following:

✴ Supporting locomotion during the times of insufficient natural light ✴ Facilitation of visual tasks

✴ Creating moods and atmospheres ✴ Supporting feeling of safety

✴ Security reasons (e.g. creating an impression that someone is still awake during the night)

✴ Improving interpersonal communication, both in real life and remotely (e.g. ensuring that interlocutors can see their facial expressions and body language)

✴ Supporting the needs of domestic animals (e.g. leaving a light on “for the dog” that does not like darkness)

✴ Improving focus and concentration (e.g. keeping the lights on not to see better but to feel more concentrated and energised).

(23)

Finally, based on the studies, it was possible to identify the most common patterns, issues and conflicts related to the usage of light in multi-person home environments:

✴ Among people living together for a longer time, it is typical to develop various unspoken rules and habits that, even though undocumented, everybody follows (e.g. “the last person leaving a room should switch the lights off” or “if I put my headphones on, others know they shouldn’t change the lights as I need to focus”)

✴ Lights are commonly used for subtle social cues (e.g. guessing other inhabitants’ mood based on the light setting their used or waiting for the other person to leave and switch off the lights in a certain room so you can use it without interacting with them)

✴ Most of the participants’ interactions with lighting are repetitive and follow patterns completed without thinking about them; many light-related behaviours are semi-regular and predictable

✴ Conflicts of interest usually occur not because of different personal preferences (e.g. one person preferring a warmer light than the other) but because of misalignment in day cycles and task-related requirements (e.g. one person wants to watch tv in near-darkness while the other wants to read with the lights on).

4.2 Challenge definition

4.2.1 Discovered conflicts and opportunities

Analysis of the data gathered during the first part of the study resulted in a definition of a set of repeating conflict themes and potential design opportunities. A summary of discovered conflicts is presented in the diagram below (Figure 3).

Personalisation ↔ serving needs of multiple users: People have very different needs and preferences related to the usage of artificial lighting. It can be assumed, that with further development and higher adoption of AI solutions in home environments, much more personalised light experiences will be possible. However, the more precisely personalised an experience becomes, the more its parameters drift apart form the preferred parameters of other people. In other words, the more powerful lighting systems get, the harder it will be to compromise the needs of multiple people at once.

personalisation user’s comfort better context & transparency algorithms’ complexity learning technologies simplifying interaction

serving needs of multiple users sustainability

individual’s privacy feeling of trust & control data collection concerns

reducing communication between users ↔ ↔ ↔ ↔ ↔ ↔ Figure 3: Conflicts related to the focus of the study

(24)

User’s comfort ↔ sustainability: People’s personal comfort rarely goes in pair with maximal sustainability. In the case of light, it can be illustrated on the following example: having multiple lights, sometimes the most comfortable would be to keep all of them switched on. On the contrary, the most sustainable solution would be to limit their usage to the bare necessary minimum. AI could be employed to work on these goals but it is questionable which value should it treat as a priority.

Better context & transparency ↔ individual’s privacy: Transparency of usage in home environments mostly refer to being able to see footprints of other people’s actions, history of past events and the live status of all amenities. On the one hand, AI could help to provide as much of such context as possible to other members of a house. For example, they would know not to switch off the light after you have deliberately switched it on. On the other hand, transparency of actions can lead to privacy issues and introduce new kinds of behaviours (we behave differently when aware we are being “watched”).

Algorithms’ complexity ↔ feeling of trust & control: The more complicated the algorithm of the AI is, the more difficult to understand its decisions become. Even if decisions are appropriate, we may still be unable understand them which will influence our feeling of control in the house. This can have a major negative influence on our feeling of comfort and safety.

Learning technologies ↔ data collection concerns: Building AI-based, learning devices usually requires collecting massive amounts of user data. In the times of heated discussions on privacy in technology and notorious scandals and data breaches, it is more important than ever to question if such an approach is the right thing to do. The industry is working on making the algorithms smarter, but usually, a smarter algorithm means more data collection.

Simplifying interaction ↔ reducing communication between users: Using AI for creating a well-working multi-user system of lighting in home environments can reduce the number of necessary user actions. For example, we no longer need to remember about switching off the light in the kitchen because the AI will do it for us. Reducing the number of actions and problems seems good at a glance but does not necessarily have to be. Solving problems and discussing conflicts with other people is one of the cornerstones of interpersonal interactions. By reducing the number of ‘things to discuss’, we are reducing the number of interactions between people living together.

4.2.2 Design framework

Based on the collected data, and discovered conflicts and opportunities, a design framework was created to serve as a base to the next stages of the study. It visualises study’s assumptions based on knowledge absorbed at that stage with a goal of validating these assumptions through prototyping and user testing.

The model represents the key factors that should be taken into account when designing home lighting solutions and the relationships between them. As presented in the diagram (Figure 4), actions related to usage of artificial light sources in home environments are primarily driven by two sets of needs: practical and psychological. Decisions based on these needs are influenced by a number of factors, including biological preferences, ergonomic needs, but also social factors and habits, and privacy and ethical considerations. The actions taken can be divided into two major categories: state changes and background adjustments.

State changes should be understood as visible, perceptible changes to the state of home amenities such as switching the lights on and off or changing crucial settings and preferences. Background adjustments should be understood as minor adjustments to light qualities that stay in the background (are either not distracting or unnoticeable) but that influence the quality of experience with artificial lighting through a more optimal influence of light on such factors as mood, health, wellbeing, work performance, etc.

(25)

4.3 Prototyping & testing

4.3.1 Sketching and idea generation

Design framework along with the insights collected during the interviews and context mapping was crucial to the ideation on solutions for user interactions with smart lighting in home environments with sketching techniques. Through hand-drawn sketches (Figure 5), storyboards and storytelling, the study attempted to imagine various everyday situations in which AI can be used for facilitating appropriate usage of indoor lighting in multi-user contexts. The goal of this part of the process was to choose the most interesting scenarios that could be developed in the final prototype.

Based on the work conducted in this regard, a few basic assumptions for a prototype were formulated.

Figure 5: hand-drawn sketches and notes created as a part of

technological exploration presenting development of programmable virtual prototyping space

(26)

The prototype was meant to be a virtual model of a 3-person household consisting of two parents in their late thirties and their teenage daughter. A participant using the prototype was supposed to play the role of a wife and mother.

Two main scenarios for a prototype were chosen:

1. The first scenario investigates people’s behaviour and lighting-related habits in a context were participant’s actions may unwillingly impact on other members of the household and there is no option to discuss or negotiate the changes.

2. The second scenario draws on a typical occurrence of conflict of interest related to lighting in a home environment which happens when differences in time-defined needs of different household members are not able to be easily fulfilled at the same time.

The content of the tasks facilitated through the prototype can be found in a table below.

Both scenarios are based on the stories collected during background studies and selected by importance of repeating themes among participants. Only two scenarios were chosen due to time limitations of the study (1.6).

4.3.2 Exploration of technologies

Part of the sketching process was dedicated to the exploration of technologies and finding the best medium for the prototype. A few digital prototyping tools were considered and tested, such as Adobe XD, Framer, UXPin, ProtoPie and Figma as well as programming technologies, gaming frameworks and 3D modelling tools such as pixi.js, phaser, WebGL and Blender. Creating a full prototype in pure Javascript, HTML and CSS and creating an original framework for handling isometric geometry was also considered as one of the options.

Eventually, Figma was chosen as the main prototyping tool. The decision was driven by technological factors. Figma is not only free for individual use but also is a web-based application that works well in all modern web browsers and operating systems. This allowed conducting the tests remotely on participants’ machines, which turned out to be an important advantage over other competing solutions.

Time and weather Circumstances Task

Scenario 1 6.30am, -2°C

Dawn, 12min to sunrise Sky with light clouds

Bedroom You just woke up. You need to take a shower and dress up without waking up your partner. Choose appropriate light settings.

Scenario 2 5.15pm, 3°C

Evening, 2h past sunset Heavy clouds

Living room You came home from work and want to relax while watching tv. Your daughter is studying at the kitchen table. Choose appropriate light settings.

Applications of Artificial Intelligence in Lighting Systems for Home Environments