COMPUTATIONAL THINKING FOR ADULTS- DESIGNING AN IMMERSIVE MULTI-MODAL LEARNING EXPERIENCE USING MIXED REALITY

(1)

COMPUTATIONAL

THINKING FOR ADULTS

DESIGNING AN IMMERSIVE MULTI-MODAL

LEARNING EXPERIENCE USING MIXED

REALITY

LENARD GEORGE SWAMY

(2)

ABSTRACT

With the rapidly developing interest towards programming, there’s also a growing demand for the creation of new tools and techniques that helps adults with a non technical background learn programming. Due to the lack of tools and techniques that are more appropriate for adults, many initiatives today end up using tools designed for kids to aid their endeavour of helping adults acquire programming skills. In this research project, I highlight the drawbacks of using tools designed for kids, describe what it means to have a computational skills and propose a new approach to help adults with a non IT background acquire computational skills. The result is a functional prototype of a tool that is shaped by this new approach.

(3)

1. INTRODUCTION

In today’s digital world, digital competence is increasingly gaining prominence and a necessary skill to have in the 21st century. The increased exposure to technology raises a need for understanding how the digital world works, in the same manner as we get to know the physical world (Heintz & Manilla, 2018). Consequently, this need to expose people to a more profound understanding of technology, leading towards it’s proficient use, has become one of the most prominent and primary motive for various domains, such as healthcare, transportation, finance and education. Being a student myself and having a profound interest in improving the quality of education in developing countries, the scope of this research project encomposes the educational domain and explores how both kids and adults are exposed to technology by introducing them to programming and computer science.

Some countries have introduced CS as a subject of its own while others have decided to integrate it with other subjects, by making programming an interdisciplinary element

throughout the curriculum (Heintz & Manilla, 2018). This newfound flourishing interest across the globe has engendered active research in the creation of new tools, methods and

products that fosters computational thinking for kids and teenagers (between the age of 8 to 16 years). A Majority of these research projects focus on using technology as a tool to kindle logical thinking, making it easier for them to understand and learn how the computer works (Scratch, n.d ). Most commonly used tools such as Scratch, Tynker, Gamefroot and Google Blockly help kids to learn how to code through games and storytelling.

Similarly, with the proliferation of software applications across various domains in the

professional realm, there’s a great demand for a workforce that has a sufficient proficiency in handling complex systems. Today, 90% of jobs require some level of digital skills, while more than one third of the labour force has an extremely limited ability to use ICT productively (UNESCO, 2017). As this demand rapidly rises, adults with a non technical background have been identifying the need for acquiring computational skills citing reasons such as salary, the intellectual challenge and wider job opportunities. This recent shift in interest has also led to the emergence of various initiatives, who have been exclusively helping adults from varied backgrounds to acquire computational thinking skills. As the inventory of tools, methods and products for adults is not as ripe as compared to the one for kids, these initiatives end up adopting tools designed for kids such as Scratch, Google Blockly & Microbits in their endeavor to spread computational thinking for all adults.

Having a background in programming, Computer Science and as a student of interaction design, this scarcity of tools and methods exclusively for adults piqued my interest towards identifying & designing more methods and tools that would better suit the adults. Making this as my research focus, I first critically analysed the contemporary approaches adopted by the initiatives both from a designerly perspective and a technical standpoint. Once familiar with the current techniques along with my past experience of acquiring computational thinking

(6)

suitable and accessible for all adults and begin my explorations of the various tools and methods that this approach fosters in a designerly manner.

1.1. Contemporary Approaches

I embarked this research by first gaining a profound insight on the existing initiatives and their respective approaches in propagating computational thinking for all adults. As this research project was conducted in Sweden, I decided to narrow the plethora of initiatives down to the ones present in Sweden for a start. Having the ease of getting in touch with some of the founders of the below mentioned initiatives helped me gain a profound understanding on the technical, social and cultural impact of the initiatives.

In 2012, the Swedish government established the Digitalization Committee

(Digitaliseringskommissionen) with the task of providing guidelines for the future of work related to digitalization in Sweden (Heintz & Manilla, 2018). One of the committee’s reports highlights the need for the school system to put larger focus on digital competence

(Digitaliseringskommissionen, 2014). This lead to the emergence of some of the most prominent voluntary initiatives such as TeacherHack, Kodeboken, MakerSkola and CoderDojo who exclusively focus on teachers, parents and adult hobbyists without a non technical background.

Teacherhack is a project carried out by Karin Nygårds and Terese Raymond that aims to find the digital dimension in particular subject at schools. The multidisciplinary nature of

programing allows teachers to “hack” their curriculum where by making it more engaging through the introduction of programming (Teacherhack, n.d.). TeacherHack provides tutorials through their website teacherhack.com on how a subject teacher can effectively integrate programing with school subjects through the use of Scratch and Microbits. Likewise,

Kodboken.se also helps those who want to get started with programming and digital creation. It's completely free to use and one doesn’t need any prerequisites to start. It is primarily aimed for kids and young people but they do have materials for parents, teachers and educators. MakerSkola is another project by RISE interactive whose aim is to contribute to the development of new subject matter specific methodology based on the creative use of emerging technologies. _{Challenging young people to explore the boundary between analog} and digital resources also means combining theoretical and practical work, in line with what happens in for example crafts, but in an even broader context (MakerSkola, n.d.)_{. The project} provides opportunities to develop and disseminate best practices in the field of maker culture between teachers, schools and local education authorities, which over time has the intention to improve our schools’ educational activities in general and provide input for future

curriculum development._{Extracurricular activities such as CoderDojos, code camps,} after-school clubs and makerspace activities are organized to give children, youth and teachers access to informal learning opportunities.

The aforementioned initiatives have a lot in common in the tools they use and their approaches. Scratch, Blockly and Microbits the most prevalent tools used to teach basic concepts of programming for adults. They adopt the strategy of learning by doing where learners combine code and easy to use electronics to build simple toys or digital artefacts.

(7)

These initiatives help adults learn either in groups at an educational institution with a presence of an instructor or individually at the comfort of homes or desired workplaces.

1.2. Research problem

This critical analyses of the initiatives and their approaches thoroughly through the lenses of an interaction designer and a programmer, familiarised me with not just what existed but highlighted the inherent problems with respect to the underlying ideology, the technologies used, interaction of users with the tools, and the accessibility for all adults.

The bottom-up approach and its shortcomings

The common ideology forming the backbone of all the methods is the bottom up approach. Here, the learner starts the learning process by first understanding the fundamental concepts in programming and proceeds to solve increasingly complex problems. A more detailed description of tht bottom up approach is found in appendix A. This learning approach is prevalent mostly in schools, where a certain programming language (mostly object oriented like C, C++, Java or block programming language ) is taught as a part of their curriculum. As kids are more ready to learn anything new, they don’t need a significant reason behind their pursuit of acquiring new knowledge. Along with the physical presence of an instructor to provide professional aid, the bottom up approach is perfectly well suited for kids.

However, this is different in the case of adults. As an adult, I truly believe that one must have a certain intention or a reason behind pursuing something new and intellectually challenging such as programming, especially outside a educational institution. It’s this underlying reason or intention in achieving a certain goal that provides the vigour with which one mentally pushes oneself in the advanced stages of programming. Without this reason one might even question the need to acquire programming skills. The problem with the bottom up approach is that It can be problematic when one lacks or isn’t sure about their intent or purpose to learn programming besides fundamental requirement to attain digital skills. For adults who use these tools at home in the absence of an instructor, self direction is an important element for a better learning out come.

Tools built for kids are not suitable for adults

Most of the tools (such as Scratch, Blockly, Microbits, LegoBlocks) used by the initiatives are the ones designed for use primarily by kids. These tools employ colourful visuals, playful elements- puzzles and storytelling, block programming (see Appendix A) and easy to use electronics. This approach appeals to the kids but not to the average adult.

As adults, we like to build moderately complex artefacts and not just solve simple puzzles and create digital stories. The type of digital artefacts adults like to create are a combination of multiple connected components having a certain degree of complexity. Inorder to build a complex artefact, one must be able to visualise the bigger picture ie, the different

(8)

programming isn’t enough to solve these complex computational problems and one must switch to textual programming.

Technology and interactivity

In terms of the underlying technologies, most of these tools are deployed as desktop applications. As a result, these tools are digitally constrained to a two dimensional surface. Microbit uses both Scratch and Blockly to allow one to develop functionalities for its

hardware. Both Scratch and Blockly, comes with an interactive digital interface that is primarily graphic, – in other words, visually placing a higher emphasizes on how one

interacts with the code and not the hardware components. This way one fails to consider the embodied interaction of the digital artefact with the system in which its embedded in. This also neglects the of a rich set of bodily movements with respect to interacting with hardware causing a nuanced cognitive disconnect in understanding the correlation with the software and hardware. Having a good cognitive fluency, that is, a good, synchronised correlation between cognition and bodily movements is crucial to a good learning experience. Therefore one must equally consider and enhance interactions with both hardware and software

simultaneously

1.3. Research Focus

After a thorough analysis of the Initiatives and their inherent problems, It was clear that a new approach was necessary in order to foster novel and appropriate tools for adults. Therefore, I propose a new approach - The top down systems thinking approach, followed by the area of focus of this project influenced by this approach.

The Top Down - Systems thinking approach

As adults we would want to make and build digital artefacts that are meaningful in certain aspects. For example we would like to build an artefact that would serve a particular purpose for us or to others in a real world situation. In order to create a moderately complex digital artefact and to motivate oneself to learn something as challenging as programming, one should start by identifying and defining the problem in simple words and situating it in the real world. Starting with defining the problem first before diving into the technical details is

synonymous to the “Top Down” approach.

In today’s increasingly complex and connected digital world, it’s also necessary to know how multiple components in a digital artefact(s) are connected in a system. Computational thinking in this aspect would be a thorough understanding of how information flows between these constituent components, how they interact with each other and how they ultimately affect the system they are present in. To be a proficient programmer doesn’t only mean to be efficient in solving computational problems and writing good code, but it also means to be able to visualise and determine the implications of the application in the real digital world. Having this birds eye view is a sought after, characteristic aspect of a good programmer. This is

(9)

Augmenting the top down approach with systems thinking results in the the Top Down - Systems thinking approach as shown in Fig 1.1. This approach would not just help the user in identifying and breakdown the problem into its constituent subproblems more efficiently. I believe this systems thinking approach will provide meaning to pursue programming by making the user aware of technical, social and cultural impact of the digital artefact they’re trying to create.

Figure 1.1 - The top down - Systems thinking approach

As shown in figure 1.1, the user is first given a task to solve. Allocating a task becomes the problem to solve, giving the learner an intent to achieve a certain outcome. He then proceeds to solve the problem step by step by first identifying the components required. Using this information then learns to connect the components physically with each other. Having this birds eye view of his system, in step 4 he identifies the sub problems in terms establishing digital flow of information between the components and how the artefact would interact with the environment. One the subgoals are identified, he moves on to converting these subgoals into computer executable code by writing it in pseudo-code. By step 6, he would have a good grasp on the problem and be able to determine the appropriate programming language, tools and methods required to solve the task. With this added meaning he proceeds to learn the fundamentals of programming required to solve each of the subproblems.

(10)

Research question

This new approach opened a vast design space that requires to be explored in a designerly fashion. But given the limited time period I decided to primarily focus on Step 1 - Step 3, which is this is the exploration stage of the learning process. As shown in figure 1.1, This is where the user explores and fishes for information regarding the components and connecting these components.

By addressing the inherent problems with the existing tools such as the absence of a rich interactive learning experience and its unsuitability to adults , this novel ideology calls for the creative use of novel technologies such as mixed realities to create a rich learning

experience in the realms of computational thinking. Based on this, I formulated my research question as follows.

“ How will mixed reality aid in the multimodal exploration of hardware and

software and provide an immersive learning experience ”

(11)

2. LITERATURE REVIEW

The first step of the research process was to become well informed by the state of the

knowledge relevant to the area of my interest. This helped me familiarise myself with existing approaches to teaching programming and their respectives strengths and shortcomings. I also had to gain a basic understanding of new concepts such as constructivist learning theories and systems thinking approach. More readings were done to identify how tangible user interfaces could be enhanced to support multimodality through aesthetic interactions by treating the computer as a social object. Figure 2.1 shows the distinct elements that i believe are critical to computational thinking.

Figure 2.1 - the building blocks of an effective approach to propagate computational thinking

2.1. Computational thinking for all

In an effort to instil computational thinking for teachers with varied backgrounds, Heintz & Mannila (2018) started the process by first introducing them to programming using contemporary, well established tools such as Scratch and Pyonkee.

Being the first project in Sweden on helping adults on a larger scale with acquiring digital skills by using tools designed for kids, their insights and findings were coherent with my

(12)

Moreover there were a few other new insights garnered from their projects that aided my comprehension of the plausible implications of using these tools & methods.

One of the key interpretations of their process outcome was that the teachers who learn Scratch (or other similar tools), learn it as a tool and not as a realisation of common programming concepts. When they have learned the tool, they feel that they know

programming (Heintz & Mannila, 2018). This became evident when they start discussing the programming concepts with the teachers or show how to do the same thing in another language, such as Python, where the aren’t able to follow. Scratch in all its greatness also seems to lure people into believing that they know more than they do, which is something we have to be aware of and try to mitigate (Heintz & Mannila, 2018).

The implications of this could be that programming languages becoming a hurdle and not an easy opportunity to solve a computational problem in a different way. the true objective behind helping them acquire digital skills is for them to have the ability to think logically which is a common denominator to different programming language and to the way the computer works. Emphasis should be given to semantics and not to the syntax of programming.

There is a need for a clear vision at the leadership level and resources that make it possible for teachers to not only take part in a limited number of workshops, but also to learn more and experiment on their own and together with colleagues (Heintz & Mannila, 2018). Tools like Scratch requires the presence of a physical instructor to help the users learn

programming. Tools that feeds into the intuition of its user’s making it more usable will

enkindle and enhance self directed learning, thereby reducing the need of a human tutor and increasing the engagement level of the learner.

2.2. Constructivism and Multimodality

Constructivism as a paradigm or worldview posits that learning is an active, constructive process, where learner is an information constructor, actively constructing or creating their own subjective representations of objective reality (Learning theories, n.d.). New information is linked to to prior knowledge, thus mental representations are subjective. In any case, we are active creators of our own knowledge. To do this, we must ask questions, explore, and assess what we know. his learning theory is best suited for adults over the age of 25 as a majority of them would have acquired substantial knowledge that they could work on and reflect. Leveraging on this acquired pre existing knowledge and appropriately channeling it for them to apply it with the new knowledge is the key element to the creation of novel tools.

This theory engenders the Constructionist Learning approach, where new knowledge is constructed inside the head of the learner through the act of making something shareable outside of their head (Ackermann, 2010). The maker/DIY movement with its emphasis on learning through direct experience, hands-on projects, tinkering and invention, is based on constructionist learning ( Stager, n.d.). This is also referred to as the Somatic learning style where instructional material presented in a particular somatic style - visual,auditory, or

(13)

that combining multiple somatic modalities when presenting content creates more effective communication and learning. The idea here is that physiologically, lasting learning requires that neuronal pathways be formed within the brain. Using multiple modes in presenting learning content (text and image, sound and image, etc.) stimulates several brain locations simultaneously and contributes to the development of the required neural path whereas a single mode presentation results in less stimulation (Lohr, 2003).

Most of these papers highlight the richness of modalities while making or crafting. Having the same richness while exploring a subject is also instrumental to learning. The dependency on computers, dulls down the modalities of exploration to just visuals and sound which should be changed. To come up with novel approaches, this calls for using the interaction driven approach, where one starts off by exploring how people would interact with the artefact and then proceed to determining the features and the suitable technology that can support the interactions. This way one can create a rich set of interaction vocabulary.

2.3. Designing for Gesture and tangible interactions

In their book, Designing for gesture and tangible interactions, Maher & Lee (2017) zooms into the human side of things - cognitive effects and development on the way people think, when people interact with computers. They advocate the move towards a more embodied form of interactions by incorporating tangible and gesture interactions thereby detaching ourselves from touch screens and physical artefacts such as the mouse and the keyboard. Using a mouse and other elements can diminish the richness of interaction where as tangible UI can enhance and provide more freedom in the form of hand gestures broadening the bandwidth of interactions which falls to favour of multimodality (Maher & Lee, 2017).

Tangible interactions

Tangible interactions places emphasis on the use of real world physical objects also known as tangible objects. A key aspect of tangible interactions is its support for concurrent access and manipulations (Fitzmaurice, 1996). By moving and grasping tangible objects we can manipulate digital content. Another interesting property was the spatial reconfigurability of devices. Tangible objects are discrete, spatially reconfigurable physical objects that represent and control digital information (Fitzmaurice, 1996).

On the contrary, the design issues highlighted by Maher & Lee (2017) was certainly

something critical to the use of tangible interactions. One of them was designing the interplay between virtual and physical element. This involves determining which information is best represented when displayed physically and when displayed virtually (Maher & Lee, 2017). Another key insight was the lack of strong conventions and general interaction models that represent a set of interactions with tangible user interfaces (Maher & Lee, 2017).

(14)

the use of remote actuators, sensors and displays will allow people to interact at a distance leading to a more sociable multiple use interactions.

Gesture based interactions also have a few design issues that needs to be taken into account. A real problem of fatigue can occur if the user does repetitive gestures for a long period of time. This calls for designing ergonomic gestures that reduces these repetitive actions and improves the learning rate at the same time (Maher & Lee, 2017). Reducing the error of inaccurate gesture recognition was another design challenge. For example, a swipe gesture can be done with a single finger or with more than one finger. The system should be able to determine both the action as a swipe gesture and shouldn’t throw an error.

Towards embodied interaction

Combining the set of key points and design challenges with both tangible and gesture based interactions provided insights that influenced the design of the interaction model of the tool. Spatial reconfigurability opened up a design space that pays close attention to the nuanced cause and effects in the user created system when the components are rearranged. With respect to selecting what information needs to be displayed, it should be meaningful from the user’s perspective causing no cognitive disfluency. Another design challenge I was made aware of the lack of standards for a TUI. Therefore to design gestures that has a natural feel to it while interacting with the system, I had to analyse each user oriented task closely and with utmost care. When it came to gesture recognition, the tool must be trained to

acknowledging subtle movements such as holding and flipping a tangible object at the same time and differences between moving the fingers and moving the hand.

2.4. Aesthetic interactions

The concept of Pragmatist aesthetics highlights the difference between the aesthetics of use and aesthetics of appearance by acknowledging that functionality and clarity is not enough to meet human needs and desires when engaging with interactive systems ( _Petersen, Iversen, Krogh & Ludvigsen, 2004 ). Using their established foundation for an aesthetic approach to interactive system design, I extended the two central aspects of aesthetics - socio cultural approach to aesthetics and designing for mind and body to better fit the scope of this research project.

Socio cultural approach

Our ability to engage in an aesthetic experience is based on our social context, manifested in a personal bodily and intellectual experience prolonged beyond the immediate experience. Aesthetic is not a priori in the world, but a potential that is released in dialogue as we experience the world (_Petersen, Iversen, Krogh & Ludvigsen, 2004).

In her work Suchman (1987) highlights the problems with human computer interaction by - are. contrasting the difference between plans and situated actions. As mentioned by her, plans are a predefined set of actions where as The coherence of situated action is tied in

(15)

essential ways not to _{a priori}prescriptions. but to the action's particular circumstances (Suchman, 1987). According to her,

“ _{The ability to interact gracefully depends on a number of relatively independent skills: skills}

involved in parsing elliptical, fragmented, and otherwise ungrammatical input; in ensuring that communication is robust (ensuring that the intended meaning has been conveyed); in

explaining abilities and limitations, actions and the motives behind them; in keeping track o f the focus of attention of a dialogue; in identifying things from descriptions, even if ambiguous or unsatisfiable; and in describing things in terms appropriate for the context.“

This made me account for erratic behaviours by the learners at different stages in the

learning process, instead of implementing a predefined set of instructions that follows a linear path. This notion towards treating the computer as a social objects also influenced in the design of how the tool will communicate or give feedbacks to the user which is more linguistic rather than mechanistic.

Designing for mind & body

In a pragmatist perspective, aesthetic experience is closely linked not only to the analytic mind nor solely to the bodily experience; aesthetic experience speaks to both. According to pragmatist thinking, the aesthetic experience encompasses the immediate sensational auditory, visual and tactile qualities of artefacts and the intellectual process of appropriating the artefact (_Petersen, Iversen, Krogh & Ludvigsen, 2004). It is the system's capacity to excite imagination that potentially will reward the user an aesthetic experience comprised of both a bodily sensation and an intellectual challenge. As explained in the previous section, aesthetic interaction with a TUI in the educational context means that a designer must choreograph a natural set of gestures that has a positive effect on the learning rate.

2.5. Spatially augmented reality

Augmented reality provides us with an interactive experience of a real-world environment whereby the objects that reside in the real-world are "augmented" by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory,

(16)

Figure 2.2 - various modes and levels through which overlay digital information onto physical objects

Figure 2.2 shows the various modes and levels through which one can overlay digital information onto physical objects using head mounted, hand held and spatial see through displays (Bimber & Raskar, 2005). But by using a projector, we can directly integrate digital information onto the user’s physical environment and this specific variant of the A.R

commonly known as Spatially Augmented Reality or SPAR. A characteristic aspect of the SAR is that it doesn't require any hand held or head mounted displays as the projector can be physically fitted in the environment than being on the body (Bimber & Raskar, 2005)_{. This} gear free and multi - collaborative lends itself to be an important characteristic of tool made for improving productivity and collaborative learning.

As this research project slightly inclines towards tangible user interfaces and embodied interactions, the use of SPAR was more relevant to this project than the other forms of AR. With the use of SPAR, there are no widgets to manipulate, no sliders to move, and no displays to look through or wear,. Instead, we walk around objects, moving in and out to zoom, gazing and focusing on interesting components, touching or modifying using our hands or hand-held tools, all at very high visual, spatial and temporal fidelity.

The primary components of a SPAR system are cameras, depth sensors and projectors. More than being easy to use and assemble it also lends itself to a low cost way to build something. Projector-based SPAR allows possibly higher resolution and brighter images of virtual objects, text or fine details. Since virtual objects are typically rendered near their real-world location, eye accommodation is easier.

(17)

2.6. Systems thinking

As this project uses an ideology that revolves around inducing the perspective of a system’s thinker as a part of the learners cognitive frameworks, the following inherent properties of a Systems Language are embedded into the design of the tool.

Focuses on “Closed Interdependencies.”

The language of systems thinking is circular rather than linear. It focuses on closed interdependencies, where x influences y, y influences z, and z influences x. Identifying a starting point(s) of cause and how the output manifests itself and has an effect. This cause and effect awareness teaches and helps to build efficient systems (Goodman, 2015).

Is a “Visual” Language

Many of the systems thinking tools—causal loop diagrams, behavior-over-time diagrams, systems archetypes, and structural diagrams —have a strong visual component. They help clarify complex issues by summing up, concisely and clearly, the key elements involved. Diagrams also facilitate learning. Studies have shown that many people learn best through representational images, such as pictures or stories (Goodman, 2015). A systems diagram is a powerful means of communication because it distills the essence of a problem into a format that can be easily remembered, yet is rich in implications and insights.

Allows Examination and Inquiry

Systems diagrams can be powerful means for fostering a collective understanding of a problem. Once individuals have stated their understanding of the problem, they can collaborate on addressing the challenges it poses. And by focusing the discussion on the diagrams, systems thinking defuses much of the defensiveness that can arise in a high-level debate (Goodman, 2015).

Embodies a worldview

Systems thinking looks at wholes, rather than parts, that recognizes the importance of

understanding how the different segments of a system are interconnected (Goodman, 2015). An inherent assumption of the systems thinking worldview is that problems are internally generated.

(18)

3. STATE OF THE ART

3.1. Lumen

Figure 3.1 - Screenshot from CIID website (CIID, n.d.)

Lumen is a Mixed Reality storytelling platform that lets people immerse in alternate realities in their natural space through machine learning and projection mapping technologies. It tries to explore the creation of a new kind of media that takes advantage of the physical world while overlaying a layer of digital fiction on top of it.

This Concept prototype that greatly inspired and influenced the technical design of my product. With the use of a camera, mini projector and machine learning algorithms, I learnt how one could augment any interface with additional information and functionalities in a simple low cost fashion.

3.2. metaDESK

Figure 3.2 - Screenshot from MIT Tangible Media Group website (MIT, n.d.)

MetaDESK is a table with a computer screen as the table top. It is a prototype developed at the MIT media lab’s Tangible media group. “ The metaDESK integrates multiple 2D and 3D

(19)

graphic displays with an assortment of physical objects and instruments, sensed by an array of optical, mechanical, and electromagnetic field sensors “. with the use of physical object as both icons (also referred as phicons) and widgets, one can manipulate virtual space with physical objects available in the real world.

Being one of the first few attempts in connecting the real with the virtual world, a key design insight gained from this project was the physical embodiment of GUI (graphical user

interface) widgets such as icons, handles, and windows by coupling it with the everyday physical objects and the digital information that pertains to them.

3.3. Lantern

Figure 3.3 - Screenshot from Lantern by Nord projects (Nord projects, n.d.)

Lantern is an open source DIY project by Google in collaboration with Nord projects. _Using an Ikea lamp, laser projector and Android Things, Lantern explores the relationships between physical surfaces and digital content — augmenting real-world objects and environments with glanceable, meaningful data. _{Using a mobile app and existing services such as spotify,} processing for android and google calendar one can transforms any surface into an ambient interface

Similar to the working of Lumen but differs in purpose. As Lantern is an open source project, I used Lantern’s source code and physical form design to quickly build a functional prototype used for this research project.

(20)

3.4. Loopy

Figure 3.4 - Screenshot from Loopy (Loopy, n.d.)

Loopy is a web tool that allows one to create models, visualise and simulate a system. Here users can create various entities / nodes/ components integral to the system, draw

relationships between these components. These relationships are based on how one component affects the other. After creating a connected system one can simulate an interactive system

One key aspect of this tool is the ease at which one can quickly create a model however complex. Having the interface of popular paint applications, one uses a pencil tool to draw circles and connections between these circles. I borrowed the visual language used in this tool that identifies components, the ease with which one could make connections and how one could make a simple interactive simulation using simple digital interface components such as sliders and buttons.

(21)

4. METHODOLOGY

According to Nigel Cross (2007), there are things to know, ways of knowing them and ways of finding out about them. Just as how scientific ways of knowing uncovers the truism of facts designerly ways of knowing uncovers the problems of the real world embedded in a socio cultural context (Cross, 2007). Design inquiry is_{a form of inquiry built on a belief system that} is a combination of practicality, ingenuity, empathy, reflectiveness and a concern for

appropriateness. This is similar Nelson’s & Stotlerman’s (2003) qualities of design inquiry. It is composed of_{reflective, abstract or conceptual inquiry, action oriented approaches with a} design goal of creating the not yet existing (Nelson & Stotlerman, 2003). The methods I pursued for this thesis was influenced by both the underlying belief systems and approaches towards handling wicked problems.

To garner rich insights into the design problem within the limited amount of time i adopted the participatory mindset throughout the process. A Participatory mindset can break down the disciplinary and/or cultural boundaries between a design researcher and a user, moving the traditional user centered design process to a design process based on collective creativity (Sanders & Stappers, 2012). Combining multiple distinct qualitative methods such as user interviews, contextual inquiries, fly on the wall observations into a appropriate number of Codesign sessions was an efficient way to carry out the design inquiry given the limit time period of this project. These sessions consisted of say, make and do activities that proved to be quite effective in terms of participant engagement and gathering rich insights. It was also important to be completely open to what is emergent during these codesign sessions.

Analysing the gathered data & interpretation is part of our endeavour to grasp the conditions and contexts that exist in a design situation which will set the stage for conceptualising new designs ₍Nelson & Stotlerman, 2003). Desired data and the problem framing gives direction to the interpretations and analysis and using qualitative judgement, is where one decides what must be in the foreground and what goes in the background. Some of the methods used at this stage of the process followed Kolkos synthesis methods such as concept mapping and insight combination.

Design offers a way of problem solving through synthesis as opposed to that of science through analysis. While science has a problem focused strategy wherein scientists try to first discover the rules design can take on a solution focused strategy wherein the designers just focus on trying to achieve the desired results (Cross, 1982)_{.As there isn’t one right solution} for a wicked problem, I employed a solution focused strategy in my ideation and prototyping phase. Generating a substantial amount of ideas and concepts also helped me to

understand the nature of the problem at hand. Having an interaction driven approach influenced the solutions which were focused on the nature of one’s interaction with the system. the sketching and rapid prototyping phase were implemented using off the shelf components such as arduino, raspberry pi, webcam and projectors.

(22)

4.1. DECIDE Technique

It’s used as guidelines in guiding the evaluation activities, in terms, selecting the suitable techniques and practices(Preece, Rogers & Sharp, 2002).DECIDE stands for

1. D_{etermine the goals} 2. E_{xplore the questions}

3. C_{hoose the evaluation approach and methods} 4. I_{dentify the practical issues}

5. D_{ecide how to deal with ethical issues}

6. E_{valuate, analyze, interpret and present the data}

The DECIDE framework was used for designing the workshops and aided me in structuring it in a systematic manner. Using this framework I identified the goals for each workshop, select the appropriate and relevant codesign methods for the workshop and analyse the different kinds of data (drawings, artefacts, writings & collages) collected from the workshop.

4.2. Collective creativity & co-creation

The challenges of solving wicked problems and exploring fuzzy opportunities are growing, making the necessity of collective creativity more and more important (Sanders & Stappers, 2012). People as experts of their experience, plays a large role in knowledge development, idea generation and concept development. Their roles in the design process are changing from, what was formerly known as “end-users” becoming acknowledged as co creators. (Sanders & Stappers, 2012)

Generative exercises/ sessions

We want people to reflect on and express their needs and values in order to explore future scenarios of use. But needs and values are abstract qualities that people are not often used to talking about directly. Therefor insights into the deeper layers of understanding requires that the participant has been thoroughly involved in the problem or situation for sometime (Sanders & Stappers, 2012). By providing tools for ideation and expression and through Say do and make techniques, immerses them over a longer period, where they can become more sensitive to their awakened memories and associations and have the opportunity to gather stories that illustrate things they find interesting or worthwhile. The most helpful perspective from which to organise tools and techniques is people centered, focusing on the activities of the participants rather than those of the researcher or the data.

Layering the levels of knowledge

I wanted the people to reflect on and express their needs and values in order to explore future scenarios of use. Needs and values are abstract qualities and people are not often

(23)

used to talking about them directly. According to Sanders and Stappers (2012), we can think about values by linking them to situations or occurrences, that is, to stories.

Using the “day in the life” (as shown in figure 4.1) approach, a designer can generativity bubble out the needs an and values of the participant. The exercise is defined in detail in appendix A. First someone is asked to describe the steps that occurred in one specific day. This forms the ‘layer of facts’. When that has been completed, they are asked to explain which of these activities they enjoyed (the high points) and which they hated (low points), forming a ‘layer of valence’. In the third step, he or she is asked to indicate, for each high and low point along the story, why it was high or low. The explanations about why reveals the ‘layer of needs and values’. The strength of the layering approach is that it invites people to first create a complete story, then evaluate it, and then find reasons for their evaluations. (Sanders and Stappers, 2012).

Figure 4.1 - The day in the life exercise can be used to layer people from stories to a description of their needs and values (Sanders & Stappers, 2012).

The design workshops

There were two codesign sessions in total. The first session was again split into two exercises where the first exercise focused on how people use search engines online tools and their own experiences and knowledge to construct or identify solutions to the problem given to them and the goal. The second exercise was to identify the pain and pleasure points of their experience in trying to solve a technical problem which was influenced by the “day in the life” exercise.

(24)

of the second codesign session was to discover the the various types of information a novice user needs about a single component and how they would use this information to connect the components with each other. I was also curious about the learner’s general process flow of fishing out information using various resources. Using sensitizing packages that consisted of say & do techniques using words and images, I was able to dig deeper for their information needs and got a good overview of their process of fishing out this information.

4.3. DIKW scheme

Analysing involves interpreting the data, making comparisons to theories and to other data, searching for patterns and determining how well they fit , generalizing findings to a broader scope and finding evidence to support your conclusions.

Figure 4.2 - A simple model to guide analysis, based on Ackoff’s DIKW scheme

In explaining the process of analysis and positioning all these activities, I found it helpful to frame them in a variation of the DIKW scheme - the letters D, I, K, W standing for Data, Information, Knowledge and Wisdom, which distinguishes the levels of sensemaking (Ackoff, 1989). This DIKW scheme helped me to structure and explain how to deal with data

gathering and analysis. As shown in the left hand side of the figure,

● Phenomenon - _{It is something that happens in the world and making sense of the}

phenomenon is the object of the study. People finding it difficult to connect components and making it difficult to come up with specific subgoals

● Data - _{Date can be in the form of photos, videos, notes and things that people make}

or leave behind._{Data itself has no meaning and meaning is actively chosen by the} researcher through interpretation

● Information - _{In this step the data is processed for information regarding the}

(25)

● Knowledge - _{knowledge is generalized and abstracted from the individual data and}

information bits about which its is made. Having this knowledge can predict further events and further data can be extracted from the evidence

4.4. Kolkos synthesis

During synthesis, designers attempt “to organize,manipulate, prune, and filter gathered data into a cohesive structure for information building” (Kolko,2007). Synthesis reveals a cohesion and sense of continuity; synthesis indicates a push towards organization,reduction, and clarity. (Kolko, 2007). Sensemaking of the collected data requires a set of well structured methods. Amongst these, concept mapping and insight combination were two methods more relevant for the data collected.

A concept map is a graphical tool for organizing and representing knowledge. It “serves as a kind of template or scaffold to help to organize knowledge and to structure it, even though the structure must be built up piece by piece with small units of interacting concept and

propositional frameworks” (Novak & Cañas, 2006). Breaking down the rich collective insights generated by the codesign workshops into its distinct elements and identifying a good

relationship between these informed and created a mental model of the problem space.

Insight combination first demands the articulation of individual design insights, and then forces a structured and formal pairing of insights with existing patterns (Kolko, 2010). This pairing creates a new design idea that has a strong connection to both established best practices and to problem-specific research data.

Both concept mapping and insight combination was used within the DIKW scheme to process the data collected from the codesign workshops during the design process.

4.5. Interaction driven approach

As this research project aims at defining a rich vocabulary of interactions for the efficient use of spatially augmented reality within the educational domain educational, the interaction driven approach was the ideal method to pursue during the ideation stage. According to

Maeng, Lim, & Lee (2012), the interaction driven approach makes one aware of finding the product element to apply movement, semantics of movement, thinking about products with similar movement, contexts related to actions and possible form factors (physical structure). With to this approach I was able to conceptualise or extract ideas of new interaction methods through properties of product materials used for a tangible interaction design, sketching out a use scenario rather than ideating isolated components of the system, articulating

sophisticated qualities of interactivity for promoting the design of aesthetic interaction.

(26)

5. ETHICAL CONSIDERATIONS

The project follows follows the ethical standards discussed in Good Research Practice (Swedish Research Councils 2017). As the research project primarily relied on user participation in the codesign workshops, the integrity of the participants is of great importance. Therefore, ethical considerations include:

• General Awareness: During the workshops participants were informed about the topic and goal of the research and what will happen at each stage

• Voluntary participation: People participate free from coercion. Participants are free to withdraw at any time without negative impact for them. Explanations are not required. • Informed consent: The participant is fully informed about the evaluation being conducted and is aware of the purpose of the project and how the findings might be used.

• Confidentiality: Identifying information is not made available and is excluded from any report or publication.

• Do no harm: The evaluation process should not harm in any way (unintended or otherwise) the participants. Harm is understood as both physical and/or psychological

(27)

6. DESIGN PROCESS

The shape of the design development process has changed in response to the shifting foundations in the design research landscape (Sanders & Stappers, 2012). As shown in figure 6.1, a large front end also known as the fuzzy front end, has been growing and gaining in importance. In the fuzzy front end, there is no clear path on how to proceed and there may be many divergent paths to explore before any patterns can be discerned (Sanders &

Stappers, 2012).

Figure 6.1 - The shape of the design development process has changed in response to the shifting foundations in the design research landscape

The design process started of by first establishing a basic ideology - top down systems thinking, based on the analysis of the research problems and literature filling in the gaps with theories and practices. This step helped me to identify the broader scope of my research project and determine what could be (or should not be) designed. Addressing the challenges of wicked problems and fuzzy pathways are done through collective forms of creativity and generative design thinking (Sanders & Stappers, 2012). As wicked problems don’t have a well defined strategy to traverse their unpredictable landscape, one must keep in mind this fundamental nature of the wicked problems while addressing them. The design process primarily consisted of two codesign sessions to gain insights and an understanding of the problem. The codesign sessions were a combination of various methods of qualitative research such as interviews, observations, activities and discussions. The 3 participants (Between the age of 25 - 30) selected for these workshops didn’t have any technical knowledge about programming nor the use of electronic components.

This was followed by analysing the workshop results using the DIKW and Kolko’s method of synthesis and exploring various design opportunities the SPAR technology engenders. The process ends with a conceptualising phase followed by building the first prototype.

(28)

6.1. Codesign Session 1 - Introduction to the top down

systems approach

The primary goal of the first codesign session was to introduce the user’s to the top-down systems thinking approach and identify user’s pain and pleasure points between step 1 - step 4 in the basic framework diagram shown in figure 1.1. That is given a computational problem to solve

● how does one apply his/her pre-existing knowledge about the world with the aid of search engines, video tutorials and help portals ?

● What type of knowledge one poses with respect to technical components and computational thinking ?

● How does one understand and analyse the components of their systems ?

● For a problem to be solved efficiently, one must be able to break down a problem into smaller subproblems.

● What was the nature of their subgoals ?

Using the decide framework, I formulated the above mentioned questions that I wanted to explore, that ultimately shaped the goal of this codesign session. Insight into the deeper layers of understanding requires that the participant has been thoroughly involved in the problem or situation for sometime (Sanders & Stappers, 2012). Therefore codesign session was split into two design exercises where first exercise acted as a sensitizing activity that primed them for the second one. The first activity introduced them the top down systems thinking approach and the second activity focused on eliciting their high and low points between Step 1 - step 4. The structure of the first activity was influenced by the basic framework itself and the second activity was primarily influenced by the layering exercise (Sanders & Stappers, 2012). See appendix B for a detailed description of this codesign session.Figure 6.2 shows the results from both the activities.

(29)

Figure 6.2 – Top-Left (a) design exercise 1 worksheet from an individual user; Bottom-Left (c) Design exercise 1 worksheets from all participants; Top-Right (b) Design exercise 2 worksheets

Insights and conclusions

In step 1 of the design exercise I could see that the participants found it really comfortable when they were given a problem to solve. One of them mentions

“It was nice that you gave a problem to us. We could easily start solving

them and proceed with the exercises “

This proved my assumptions wrong. Initially, I was under the impression that they can solve or make something that based on their personal liking. This lead to realisation that they wouldn't be able to imagine a problem space without having a fundamental know how of the components involved. Therefore giving a task that is has some sort of familiarity and

(30)

In Step 2,_{there was friction in terms of while fetching information regarding the components} while using the internet tools. For example, one participant searched for motors on google and got back results that showed them various kinds of motors. Some of these motors looked different although they all worked the same way but it did confuse the participants. Therefore they just resorted back to asking me ask me if they had any technical doubts regarding the components, as they were confused from the multiple choices that google provided. They reason they resorted to the human instructor was because of the specificity of the answer given by the instructor.

When it came to connecting the components in step 3, the lack of prerequisites regarding the individual working of the components made it difficult for the participants to connect them. They wanted to know how information is processed at each node of their individual system and how information flows between these nodes. From the nature of their subgoals in step 4, it was relevant that the participants didn’t acquire significant information regarding the

components. This piqued my curiosity as I myself wasn’t aware of the users information needs and therefore set the stage for codesign session 2.

6.2. Codesign Session 2 - Information needs & task flow

analysis

The effort to solve one aspect of a wicked problem revealed another set of problems, which determined the goals for the 2nd codesign workshop. In step 4 of design exercise 1, The inability to identify subgoals was primarily due to lack of appropriate information regarding the connection of components and how they work in relation to each other. Intrigued by this problem, the 2nd codeign workshop focused on revealing various types of information (verbal and visual) required by the participant about a single component and how they use it to connect the components. This codesign workshop was also a way to gain insight into the participant’s process of exploring and fishing out for the required information using tools and resources available to them. The session was again split into two exercises, where the first one primed the participants by familiarising them with information needs so that in the next step they will know what information they require at a particular point in their exploration. The second activity aimed at understanding the cognitive framework used in fetching and the application of this information used to create a connected system. There were also a few physical components placed in the middle of their worksheet to elicit curiosity . A More detailed explanation of Codesign session 2 is described in Appendix 3. Figure 6.3 shows the structure and the results of codesign workshop 2.

(31)

Figure 6.3 - top (a) Worksheet displaying the structure for codesign session 2, bottom (b) A filled worksheet after codesign session 2.

(32)

Key Insights of session 2

In the first activity, one of the key insight is how the participants touched and moved the real physical components placed at the center to identify the information required. The phrases and questions collected inside the circular area summed up to questions of various aspects such as

● Basic technical information regarding individual sub components and a brief knowledge about them

● The way one would interact with the component and control it

● Questions regarding the components usage - through examples and tutorials

● The type of information it generates (analog or digital) and how it flows between these components

The set of phrases and questions that weren’t included in the circular area were mostly about the component’s domestic use, its reliability and functionality, details regarding its

manufacture and limitations. When i asked them about the cards that they didn’t pick, one of them said that

“This is background information. More like something I wouldn’t ask

because I wouldn’t understand without having the basic knowledge first”

Another participant suggested that

“I will search deeper only if I am interested and enthusiastic about the

product”

The second Part of this codesign session revealed a few insights on the way in which they explore and fetch information. For all the three participants, the first step in the process is a simple query on a search engine like google as its fast and reliable. They were inclined towards fishing out the basic technical information from trusted information sources such as Wikipedia. As most of them they are used to the ease of asking an Instructor to help them with moderately difficult questions on its usage they expected a similar form of social

interaction in the absence of a human instructor therefore they resorted to Instructions which were mostly video tutorials on how to build with the components in sequential steps. Along with the information acquired from both wikipedia and online tutorials, they were able to construct and visualise a basic birds eye view of the system, how components are connected and visual flow of information between these components that had positive effects on the rate of learning. Due to the use of keyboard and mouse this exploration still lacked rich

(33)

6.3. Discussions on the use of SPAR technology

After the codesign workshops, I had a brief discussion with the participants on the use of more immersive technologies that could possibly help in a multimodal exploration by showing them the set of images as shown in figure 6.4. This images were mainly covered the use of state of the art tools such as Lantern and Lumen as well as head mounted and handheld Augmented reality tools. Although head mounted and handheld AR was out of the scope of this project, I wanted to spark a brief debate between the two technologies. This gave them a partial glimpse into what I was trying to achieve with the different forms of SPAR. The

discussion was based on

● how these interactive technologies can enhance the search experience. ● Fit in the top down systems thinking educational approach.

● Its appropriateness in the educational domain for a multimodal learning approach.

Figure 6.4 - Examples of immersive technologies used to create a rich immersive experience

All of them agreed that this was more interesting way to learn and explore than just using traditional tools such as the computer. Although they weren’t in inclined on using a virtual reality head mounted display as compared to using a Mixed reality tool. They disliked the feeling of being “ disconnected” from the real world since VR glasses doesn’t allow one to see the real world. On the other hand they felt more comfortable with SPAR as it Gives them an overall view of the context they are embedded in. They also advocated for SPAR as it would give them the freedom to move around the table, discuss and work with others, providing a more collaborative learning experience. However they felt the need of a detailed

(34)

purpose in this project they wanted to know source of information SPAR displays over physical information.

6.4. Analysis of the workshops

Using the DIKW scheme helped to analyse the data gathered from these workshops helped in a structured manner.

● Phenomenon - _{Here the main object of study is what kind of information adults, with}

a non IT background, need to connect components and how do they explore, learn and acquire the technical knowledge

● Data - _{Here the data collected is On what basic information people need, information}

required to connect these components together and their opinion about mixed reality in the domain of education. Data is also in the form of a visual diagram depicting the task flow of exploring and fetching information on these components

● Information - _{In this section I used Kolko’s design synthesis methods such as}

concept mapping and insight combination. The concept mapping approach was done to identify the core taxonomy from the data collected that is the information needs regarding the components. This revealed various aspects of information required by the users such as basic technical details, interactivity with the components and real world examples of the components use.

The method of insight combination helped me identify patterns relevant to the core domain, that is how participants explore and fetch information using existing tools and resources. All the three participants displayed a general process of exploration which starts with the search engine. They then seek more information on sources like wikipedia and directly move on using tutorials and examples to solve the problem. By following the instructions they they learn by figuring out how to solve the problem iteratively by trial and error method.

● Knowledge - _{Combining the processed information from both concept mapping and}

insight combination method with the established theories and practices helped me perceive the bigger picture. That is what must be included (and excluded ) in the design of the experience of the educational tool. One of the design consideration is that the tool must not only provide information to the user but also guide through the exploration like a tutor. It was important to include the social qualities shared between the learner and the instructor into this tool making the interactions more linguistic. Another important consideration that sprouted out was to enhance the logical thinking required computational thinking and programming along with attributes of a systems thinker

(35)

Figure 6.5 - Sketches showing the various considerations of the physical form of the tool

6.5. Technology explorations

Exploring various solutions and the underlying technology based on practicality, tangibility, richness of interaction, sustainability & more importantly multimodality. I chose to explore SPAR along with simple electronic components. Along with this technology I used machine learning and image recognition algorithms that forms the backbone of this technology. The role of Machine learning in this project was to train the tool with specific fundamental components such as the arduino, basic sensors and actuators for it to give more specific results. Using the open source machine learning framework called Tensorflow, It was easier automate a wide spectrum of tasks. Computer as a social object in the context of this project is its ability to see, recognise and provide appropriate feedbacks and this was possible through machine learning and image recognition. My previous experience with hardware and software helped me explore the technological terrain quickly and effectively.

6.6. Concept development

Combining the results of the workshops, knowledge gained from the technological

exploration and awareness of the educational context of this research project, I proceeded with developing the context of the tool used to tutor adults with computational thinking. I start

(36)

with determining the technology first and proceed with developing the the physical form of the tool, the visual interface and the interactions possible with this tool.

Technical details

The core component used in this tool is a raspberry pi board that runs the application. To this board there is an attached camera and a mini projector that displays information as shown in figure 6.6. The application uses a machine learning framework called Tensorflow to

determine the images, objects and gestures captured by the camera. One must pre train this framework with gestures and images that will be used by the tool for an effective recognition and for displaying the best results. The compactness of the technical components reduces the overall size of the tool.

Figure 6.6 - Left (a) A raspberry pi with attached camera, Right (b) a mini projector

Form philosophy

The physical form of the tool was an important design consideration and it was dependant on factors such as the user’s position, the size and shape of the user’s table, the distance between the user and the tool and the size of the components - the projector, cameras, depth sensors and the microcontroller board. Figure 6.7 shows the 3 ways in which this tool can physically manifest itself in the environment.

The tool can be hung on the wall directly above the user’s desk, could be kept beside his desk on a side stand. Since the components are smaller in size, the tool can also be kept on the table reducing the distance between the user. The best form it can take is that of a table lamp, as its quite a classic and a ubiquitous element to any workdesk today. As shown in the figure above all the technical elements can fit into the head of the lamp and this way it makes the design less bulky and more portable. This design is very similar to the Lantern project mentioned in the section 3 of this report.

(37)

Figure 6.7 - Sketches showing the various considerations of the physical form of the tool

Designing the interface

Once the physical form was decided I moved on to designing the visual interface that will projected by the tool on to the user’s desk. Design of this visual interface was primarily driven by the need to provide accessibility to both left and right hand users. This is an important consideration that dictates how and where digital content can be displayed without the obstruction of the arm. Moreover as the objective of the tool is to provide a rich multimodal experience, I went about designing an interface that supports the use of both the hands.

The visual structure and the design of the interface elements was also important. The overall layout was of the interface had to use less interface elements to avoid clutter. Only

information relevant to the context must be displayed to improve the learning experience. Use of bigger and bolder typefaces to improve legibility. The area of tap sizes of buttons and other interactive elements must be big enough to accomodate easier and faster taps. I sketched out low fidelity prototypes initially to determine the basic flow of the application as shown in figure 6.8. A higher fidelity wireframe was made later using the tool Sketch as shown in figure 6.9.

(38)

Figure 6.8 - low fidelity sketches showing the various considerations of the physical form of the tool

(39)

Determining the interactions

To determine the interactions possible one must be aware of the affordances that arise with the use of certain technology. With image perception and machine learning, one can train the tool to identify a plethora of gestures.

I first collected all the standard set of gestures people are aware of, primarily borrowed from the ones used with prominent touch screens devices such as swipe, tap, pinch, zoom and so on. I combined this list with the gestures possible in the 3 dimensional space with tangible objects such as lifting, rotating and moving the objects around. This was I created a rich repertoire of gestures this tool can accommodate.

However, all the gestures can’t be accommodated by the tool when one starts considers the form and the interface as they their respective constraints. For example, one can’t use a “swipe” gesture as there are physical components on the table. Started off by exploring the affordances