NON VISUALS
SEBASTIAN DE CABO PORTUGAL MATERIAL EXPLORATION OF
NON VISUAL INTERACTION DESIGN
MFA MASTER THESIS 2020
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
“To understand something is not to be able to define it or describe it. Instead taking something that we think we already know and making it unknown thrills us afresh with its reality and deepens our understanding of it”
– Kenya Hara
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Forward
This project really scares me, and that’s why I had chosen to do it, I didn’t want to do something that I had done before, I wanted to try new technologies and new design methods that were completely novel for me.
This is why I chose to work with VR and AR technologies, something I have barely touched before. I wanted to really understand how to build things for these kinds of technologies, and more importantly find ways to make prototypes and lo-fi iterations without having to do massive amounts of code, or even no coding at all.
This is also why I worked with audio, something I’ve done sporadically during filmmaking but has always been my weak spot due to my heavy focus on the visual side of things. Ironic I know.
Working with visually impaired people is a user group I haven’t collaborated with yet and this also scared me, from an ethical and empathetic point of view and from a pragmatic one, for example how do I show people concepts or initial sketches if they can’t see a sketch. Finding non visual ways to communicate my ideas.
“This is not a problem driven project or a technology driven project, but an experiment driven one”
Normally the projects we’ve done in the past dealt with a specific problem or user group and in the end we could evaluate against these points of contact to see if the outcome had been successful or not. - if we had "solved" the problem at hand. But on this project I wanted to try a research through design method, a more experiential process, one where you can have many different starting points, even op- posing ones, and you develop the research and concepts through making - its use will come out after.
Even though everything about this project scares me, and I can’t imagine what the outcome will be, I feel confident that something good will be produced, I have to learn to trust the process, and myself as a designer, and that is the main reason I wanted to put myself in this situation, to prove myself I can get out of any situation successfully even if I don’t know anything about the subject at all at the beginning of it. In the future I won’t be able to choose exactly what I’ll be working on so I like to train myself to be prepared for any situation and find ways to improvise my way out. I just couldn’t predict how the world would change in the short time this thesis was being produced, and how challenging that was going to turn out to be.
This is why I chose to work on a topic that scared me most as my final project in this master’s degree.
Sebastian de Cabo Portugal - 20/01/2020
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Abstract
Design is all about visuals, or that is what I have found out during this thesis, from the pro- cess materials to the outcome our main entry point to any problem is how will we solve it visually so it’s understandable for the general user. This aspect is problematic in itself due to the fact that we, as humans, understand the world and the things around using all our senses continuously, even though we can forget as visuals are so overpowering.
There is a huge opportunity area in exploring our other senses and bringing them back to technology, and this can be seen in works in the past like Tangible Interactions [1] or Natural User Interfaces [2]. But in this moment in time, where all these new technologies like VR/AR and IoT are about to enter our lives and change them forever, this topic is more important than ever. We have already seen what happens when we turn humans into mere machines with some fingers as interactive inputs, and barely any senses to process all the information given to us. Now that these technologies are still young and malleable, we can direct the future to where we want it instead of being guided by the technology itself.
To do this we need to reimagine the design process, not reinvent the wheel, but add ex- perts which we currently leave behind and I argue are key to unlock these technologies, experts not only of the technological side of things but on the human side too, like physi- otherapists and dancers. Add also people who we never think about when we think of VR like visually impaired users, which could make these technologies inclusive since early on, insteadof as an afterthought like we usually do. Not only people, but we also need to add new materials to understand how we use our senses and explore ways that we can under- stand and explore them differently; like bodystorming and improv theatre because when things aren’t visual, how do you sketch it? A sketch turns into a video about movement.
The end result provides a wide breadth of examples of the types of innovations that can come out of using these new design materials, and to open new frontiers. From a VR game with no visuals whatsoever to an AR location based story game, to a home sized multi- modal operating system containing several different apps controlled through physical movement. The examples open up the space instead of closing into a single solution. This is just the tip of the iceberg, a hope that others will be inspired by it and continue with this journey that has just started, to guide the future into one that is more technological and at the same time more human than ever before. What we know is that VR does not equate
Visual Reality. Keywords – immersive environments, virtual reality, mixed
reality, wearable computing, virtuality continuum, internet of things
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Contents
FOREWARD -3-
ABSTRACT -4-
CONTENTS & ACKNOWLEDGMENTS -5/6- EXPECTATIONS & ASSUMPTIONS -7-
1.0 INTRODUCTION -8-
- Uniting digital and physical worlds -9- - Ethical Considerations -10- - Philosophical Considerations -11-
2.0 BACKGROUND -12-
- The world we come from/the world to come -13/14- - Focusing on visually impaired users -15- - Audiogame or be prepared to grind hard -16- - How are we adding senses to VR? Add more tech -17- - Why read a book when you have audiobooks -18-
- Inspiration -19-
- Inspiration from Microsoft Research -20- - Games, an excuse to improve the whole tech -21-
3.0 APPROACH + METHOD -22-
- Insights from background research -23- - Flip the norms - "make the known unkown" -24- - DESIGN PRINCIPLES/Perfect Timing -25-
- Research through design -26-
4.0 MATERIAL EXPLORATIONS -27- - Working in Sprints based on Design Principles -28- - How a Loop looks from the inside -29-
- My Activities -30-
INTERVIEWS
- Interview - Luca Contato -31-
- Interview - Christy Smith -32-
- Interview - Christy Smith (cont.) -33-
- Interview - M Eifler -34-
- M Eifler Studio Metaphor -35-
- M Eifler 3 levels/scales of interaction inside VR -36- - Interview - Arianna Ortelli -37/38-
- Interview - Enrique Sanchez -39-
- Interview- Pablo Gody and Nelson Sanchez -40- EXPERIMENTS
- Playing with existing tech - Amazon Alexa -41- - Playing with existing tech - VR games -42- - Playing with existing tech - VR games -43- - Playing with existing tech - Torball -44- - Experiment 1 - Spatial awareness through music -45/46- - Experiment 2 - Playing fussball blindfolded -47- - Experiment 3 - Controlling Logic Pro with moves -48- WORKSHOPS
- Workshop 1 - Music Maker with Toby & Julien -49- - Workshop 2 - Physical Game Jam/Bodystorming -50/51-
- Suddenly the world changed -52-
- Workshop 3 - Communicating through movement -53/54-
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Acknowledgments
5.0 CONCEPT EXAMPLES -55-
- Introduction to Provocation & Examples -56- - Playing games without vision - VR Audiogame -57- - Playing games without vision - AR location based -58- - Using our own bodies/movements/senses/objects as controllers/input- Home OS -59- - Using our own bodies/movements/senses/objects as controllers/input- Physicality apps -60- - Physicality apps - Music Maker through movement -61- - Physicality apps - Walk + Talk -62- - Physicality apps - Home Physical Rehabilitation -63-
6.0 ANALYSIS -64-
- Insights from explorations -65-
- How to develop a concept -66-
- Example in use: Music Maker -67-
- Design methodology insights -68-
Design values in the Concept Examples -69-
- VR Audiogame values -70-
- AR location based game values -71- - Music Maker app values -72/73- - Walk + Write app values -74- - Home rehab app values -75-
- Home OS values -76/77-
7.0 DISCUSSION -78-
- Is this better than whatwe had before? -79- - What’s changed in the design process? -80-
- Conclusions -81-
8.0 REFLECTIONS -82-
- Postface -83-
REFERENCES -84/85-
This thesis wouldn’t have been possible without the help and support from:
- My parents, for always being close even when I’m so far away - My supervising tutor Monica Lindh-Karlsson
- My thesis supervisor Stoffel Kuenen
- The best class anyone could ask for, my IXD colleagues
- The rest of UID for always being up to try one of my weird experiments - Amanda Stendahl and the Torball group in Umeå
- RISE Interactive studio Umeå - Luca Contato from Rising Pixel - Christy Smith from caniplaythat.com - M Eifler from Microsoft research - Arianna Ortelli from the Blind Console - Laura Vendrell from VRave
- Enrique Sanchez from Orbisnauta
- Pablo Godoy Physiotherapist at NHS Trust - Nelson Sanchez Physiotherapist at Fisiotec - Julian Loretz from APD
- Julien Desvaux from IDI
- Rachel Cheung from Goldsmiths University - Javier Viera aka MakerFly
- Teenage engineering for their sick products - Humble bundle for the amazing set of VR games - HTC for their Vive headset
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Assumptions Expectations
I’ll start by stating my assumptions before the start of the project after carrying out just a tiny bit of research in order to produce my brief:
- Developers don’t think this is an actual issue - What’s the point of making something in VR if the user can’t see - Is the most common answer I’ll get from developers when I explain the thesis topic, and I feel that is exactly the point why I’m trying to make.
- Visually Impaired people are so forgotten that there’s no options out there in VR/AR for them.
- Is it possible to show an idea if the user can’t see a video or a sketch, how do you explain an idea especially when it’s about 3D and about physicality?
- Is it possible to create lo-fi prototypes of games that involve moving in space with your own body and listening to sounds in your environment without coding?
- Does VR equate Visual Reality? Can we consider walking around a 3D Sound- scape or Hapticscape VR?
- There is plenty of guidelines to design for VR/AR/MR but they are all about the visuals and not much or none at all about sound, movement and physicality.
The final goal would be to end up with two or three games or experiences to be used as exam- ples, they would include many of the experiments done and tested throughout the process. It would also include a menu to choose between the sev- eral options of the games. This would be instead of making one single solution that encompasses everything as I think it would be too confusing, and it would enable me to show the many possibilities and ways this problem could be approached by others in the future.
My wish for this project would be to create some- thing no one has ever tried before, and be at the forefront of what I hope will be a more important aspect of the VR and AR community in the near fu- ture, or at least to raise awareness of these issues early on so they can be taken into account and im- proved as the rest of the technology is improved and experimented with.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
1.0 INTRODUCTION
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Uniting digital & physical worlds
This topic could be called many things, VR with no visuals, soundscapes in virtual realities, exploring other senses in the virtual realm... The main idea is that I don’t want to focus just on VR or on AR, but on the physical and the digital realities;
to more importantly, expand the ways we use our senses in these virtual worlds, explore ways we could understand them differently and thus more in depth and open it up for new opportunities that arise through new approaches or by using new design materials.
We give a higher priority to visuals in our lives than anything else, even though there are at least 4 other ways to sense our world (there are more), digitally we have been concentrating on exploiting our visual side, relegating sound to a sec- ondary position and making touch, smell or taste disappear. This is not only problematic for the general population, but it can be a real impediment for people who are visually impaired. We choose who to include and who to exclude from early on via our decision to use multimodal interactions or not.
This is why I chose to tackle this topic, VR and AR are such young technologies, unbound by all the constraints inherent in screen-based digital technologies, with a language that has been developed and matured over the past decades, that we should really try and explore all its new capabilities. Everything in VR is an experiment right now but we’re al- ready falling into our usual traps by concentrating on improving graphics to try and make the most realistic representa- tion of our world possible. This sounds a bit counterproductive, if you want the highest representation of our reality, you already have it in the real world, why make a copy in a virtual one. We are stuck in the same old paradigms because we’re using the same old design materials.
On the other hand, if the digital reality and the real one, which were once separated by screens, gets blended into a sin- gle unifying reality, where both physical and digital coexist, we should be interacting with them accordingly to the way we do things in the real world, using our full capabilities of our bodies and senses so as to not get lost or overwhelmed in them by focusing on just one sense.
This new technology can be developed differently, we can develop it to include our other senses too, from walking around and touching objects through AR or haptics in VR, to aural soundscapes and voice control, so technologies can start to feel more natural as they adapt to us and not the other way round. I see the future designers on these jobs in teams that might includemore architects than graphic designers as it will be all about the space we move in, and dancers instead of UI experts, as they will be able to guide our bodies through movements that make sense for that application.
To create these future innovative experiences, especially in games, it would be impossible to get to if we keep looking at sight as the sense with the highest grade of importance, and better visuals as our main source of inspiration.
My anchor point to start the project was blind gaming in VR, because exploring a whole technology is too vast without a clear starting point, so my research began focused on accessibility in games, especially for blind people which then evolved into how we can introduce more senses with less devices into our lives.
"Engage our imaginations
through what we hear and
feel rather than what we
see”
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Ethical Considerations
Who do we design for? Right now the user we have collectively created in our minds and for whom we work for is an incomplete version of a human being with huge eyes, tiny mouths, small ears, no nose or tongue to taste, and with no body, just one hand with a couple of fingers that can touch but not feel. This is a far cry from what we actually look like and how our bodies work and understand the world around us.
Closer to E.T than to a human being.
Even though I’ll be championing inclusive design in this project, no process or product is perfect, and everything can’t be solved in one go, but I hope this will be a first step into considering more types of people than the normative to push our technologies to places we wouldn’t be able to imagine without them. I am fully aware that some of my solutions will be inclusive for visually impaired users, but prob- ably not for people with other disabilities like motion restrictions; but as I said, due to limited time and resources I’ve decided to just focus on including this group, as well as users with full vision.
I’m also worried about creating a feeling of "using" people, going to blind associations and asking for their help might make them weary about my intentions and I’m still not fully sure how this process can be carried out, everything is still experimental. Although in our current process we are already the de- ciders on what goes into a design and what doesn’t - we don’t carry out everything the user says, but we should have a critical eye on what is important and what is not. Hopefully by always having this user group in mind, everything produced in this project will be inclusive for them no matter how much visuals it ends up having. The important thing should be that the visuals are not giving key information which if missed would make the user unable to use the product fully.
One of my objectives with this is to empower every user: to provide the visually impaired with heroes and games that represent who they are in a powerful way, having more diverse representations is quite a hot topic in the gaming and tech industry nowadays so I want to keep pushing that agenda forward;
and to provide visual users with novel experiences which will not only be enjoyable for them, but also to train their senses and let them rediscover the power of their other (sometimes forgotten) senses whilst changing perceptions about what VI people can do, referencing the title of a paper I read during this project "I can do everything but see" [3]
So what are we going to do? are we going to continue doing things the same way we’ve always done it, or are we going to embrace the change others can bring to the process like we already did with different professionals in the design teams. My hope is that this thesis will point everyone into a new direction for the benefit of all. Innovation with a reason.
E.T. the Extra-Terrestrial (1982)
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Philosophical Considerations
Working on this project gave me a lot of questions to ponder about the future we are creating and where we want to be living in a near and far away future. The first thing that comes to mind while I develop my concepts is - "Am I doing right for my users? Is this the way to move forward?" As with everything, there is no simple yes or no answer and maybe "better" is not even the right question right now to ponder. We should have the freedom to explore areas and then see what kind of solutions can fit better or be devel- oped from there, but sometimes we can’t know what we’re creating or how someone else will carry our findings forward into the future, like investigating nuclear energy to provide a better energy solution for the world and ending up creating nuclear bombs.
"These are my principles; if you don’t like them, I have others" - Groucho Marx
Is every step I make with this project taking us into a future like the one seen in The Matrix? That’s obvi- ously not my goal, but I already see how including more of our senses into these types of technologies will create more confusion about what is real and what isn’t and will suck people into the promise of un- bound possibilities and a life lived inside the machine. The more sense we include,, the harder it will be to distinguish between what’s real and what is digital and that worries me a bit. The line between what’s real or digital will blur even more, but that seems to be the direction we’re going anyways...
Berkeley argued that our only knowledge of this world is what comes through our senses, so will these technologies add or remove our sense of belonging to the planet we live in, will we care less about the environment because we have another environment to go to where everything is possible?
I’m not trying to provide a framework or even dare to propose "truths" about how these technologies should be made, there’s not enough time for it. What I wish is to provide inspiration and examples of how to do things differently, so other designers can come up with their own conclusions, and opens up a conversation on the current design approach and how we want to develop further. That’s the best part of designers and the design industry, that we are constantly looking at ourselves and our process in the same way we do any problem we encounter and we keep iterating and refining it until it gets as close to perfect as it can, and when it doesn’t fulfil our goals anymore, we change it again. This might be the next step in its chain of iterations.
The Matrix (1999)
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
2.0 BACKGROUND
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
The world we come from
We live in a visually dominant culture, so it’s normal that most of our en- tertainment, including games, are heavily focused on the visual part of things, and we have assumed that there is a hierarchy in the importance of our sense, namely sight, hearing, touching, taste and smell; but is this true for every culture?
A study on the matter revealed “that cultures which placed particular value on their specialist musical heritage were able to communicate more efficiently on describing sounds, even when non-musicians were tested. Similarly, living in a culture that produces patterned pottery made people better able to talk about shapes.” [4]
Even in Aristotle’s time, the hierarchy was sight, hearing, smell, touch and taste; so with time, smell has been demoted to the bottom of the list. So our presumption that vision and audio are more objective than the other senses, which in turn help us learn and understand better the world than our other senses, which are more crude, is wrong. [5] Which is proof to the fact that we could live in a more inclusive world without losing any value for “the people in the norm”, and that we should start focusing on our other senses, especially in terms of our modern digi- tal-led world if we don’t want to leave a huge amount of the population out of the future, especially as the world is getting progressively older, and with age comes the loss of senses.
"The visually dominant culture emphasizes fashion and style over func- tion; physical beauty over intelligence, culture and empathy; and shiny objects over Nature’s sensory cornucopia. The quality of sound, smell, taste and touch are degraded by bad acoustics, noise pollution (includ- ing audio devices(( stale indoor air and water borne sanitation( indus- trial food chain( and synthetic materials, particularly shoes and pave- ment which keep us from feeling the Earth under our feet." - Lawrence de Martin, What senses do we use most and least. [6]
If the world we live in is one of shooters and eye tracking to use our sight as a kind of desktop mouse, then the future is continuing down this line.
The biggest immediate advances in VR apart from bringing costs down;
are better field of view/screen resolution, higher PPI, eye tracking for fo- veated rendering (rendering highly what you look at instead of the sur- roundings) and better/higher refresh rates - basically everything deal- ing with improving the quality of the images seen in VR, still focused on gaming. [Youtube-Future of VR by Virtual Insider]
Why is VR so immersive? Because of the fact that you can move and look around naturally in perfect 3D "physical crouching behind cover feels completely natural because we act how we would in real life...the more you have to do with your hands, the more immersive it is...(about periph- erals) Holding on to something you can use as a real life counterpart can really add to the experience and immersion" [Youtube - Why is VR so im- mersive by Virtual Insider]
We will move beyond games into more common uses for the main- stream public, office or creative applications like the ones found nowa- days on desktop or mobile computing. The examples by Keiichi Matsuda exemplify the vision of the future most of us have relating VR at work or AR on the street - a never-ending overflow of visual stimulus which you can’t look away from and which is not only collecting all the data you can about you, but is then using it to continuously sell things back to you.
Looks like stressful times.
Everything still feels screen based, like being inside a VR environment with an infinite amount of screens and navigating menus by physically tapping digital buttons, or by waving our hands in thin air like in Minority Report (2002)
This is the kind of future I’m opposing to throughout this thesis and my proposals works on the exact opposite direction, more connectivity but also more calmness and everything controlled in a more humane way.
The world to come
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
The world we live in The world to come
Half Life Alyx by Valve Merger video by Keiichi Matsuda
HYPER-REALITY video by Keiichi Matsuda Beat Saber by Beat Games
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Focusing on Visually Impaired users
The power of the non-normative
Making products accessible means that even people beyond target users can benefit. Disabili- ties have too often stayed hidden and taboo. I be- lieve we are entering a new age where disabilities can serve as a precursor to improving the world for others. [7]
The World Health Organization (WHO) estimates that roughly 285 million people are visually im- paired worldwide, of whom 39 million are blind [8]
These numbers continue to increase as the global population ages, as right now one in three people over the age of 65 have vision-reducing eye dis- ease. [9] This implies that more and more users will encounter these barriers in the future, espe- cially when dealing with technology.
I’ve contacted Synskadades Riksförbund Väster- botten to get expertise on visual impairment and also to see if I can get in contact with some visual- ly impaired gamers to help with the concept ide- ation stages and the testing too. The other way I tried to get collaborators was through YouTubers.
Luckily there are quite a few blind YouTubers out there who create videos explaining their day to day lives and how to they deal with everything, as well as some gamer channels in Twitch where they play fighting games, Diablo 3, Nintendo games and audiogames and commenting on them just as any other streamer would, which was a great help in these initial stages to get an overview of how things are done nowadays.
Also checking Reddit like r/blind and r/audiog- ames was very useful because again people de- scribe their experiences in detail and explain what the best games for blind people are and why.
Fighting games: These are the most accessi- ble games for blind gamers due to their incredi- ble sound design, every move has its own sounds so you can memorize the combos through the sounds it creates, you can also know what the other player is doing through sounds. Gamers re- ally study the game by trying every single charac- ter and memorizing all their moves which is quite an achievement in itself.
Diablo 3: This RPG game has narrated menus so you can learn where everything is positioned, and great voice acting for every character in the game, so you can follow the story. Also if you play with friends and you leave your character alone for a while it will automatically start following the rest of the group so it’s easy to stay all together.
To move around you can hear if your character moves to your left or right and the audio changes a bit if you get stuck in a corner, there’s also audio indicators if you’re going the right way in a quest.
Still it’s very impressive to see someone playing such a complex game without being able to see it. [10]
Call of Duty: There has even been a blind player that made it to level 28 in the zombie mini game in Call of Duty Black Ops 3, this is the highest round
I’ve ever heard anyone reaching including people who have vision. “This just shows that people with disabilities can still play video games just as good as people without disabilities” [11]
I’ve even read stories about VI people playing World of Warcraft, which is a super huge and com- plicated game with loads of mechanics and sys- tems through the use of add-ons. [12]
Accessibility in games has been a hot topic in the developer community lately, as more and more companies start adding accessibility options or more levels of difficulty in order to accommo- date more people with different kinds of abilities.
There’s still a long way to go but at least now this topic is an issue developers are trying to tackle.
Companies like Microsoft have also worked on accessible hardware which was a huge develop- ment as now players with different abilities can map the controls to where it’s easier from them to reach and let them play any game available for any console.
EXIT in Braille
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Audiogames or be prepared to grind hard
Mortal Kombat 11
Diablo 3
The Blind Swordsman gameplay by caramida9
If someone is completely blind then their best option is classic games like dungeons and drag- ons where all the action happens in their imaginations, or audiogames, which are games where the only feedback you get is through audio, no visuals, mainly a black screen or some abstract colourful screen. There’s several repositories for audiogames like audiogamehub.com/games/
which has archery, memory, labyrinth or blind cricket amongst others and they’re mostly free; or audiogame.store which has more commercial games.
Audiogames play mainly with directionality, for example the Lworks shooting game Supershot, where you shoot sounds coming from the left, middle and right, and you have to shoot them us- ing left, up and right arrow before they hit the ground. It starts easy but get fast the higher the level. There’s also a game by L-works called Lockpick where you’re a bank robber trying to find the combination to open a safe before the police arrive and catch you. This game is a lot harder than Supershot, harder to understand. Lworks keep the same menu structure so once you’ve played one of their games you understand how the rest work. They always make it clear whether you’re moving or not through the appropriate sound design, like grass rustling in the Egg hunting game, where the egg make a ringing sound, getting louder if you’re near them, and the pitch is higher if it’s in front of you or lower if it’s behind as well as coming from one speaker or the other.
A game from the audiogame jam that has been highlighted by the community is EscapeBeat, where you have to escape a maze just by getting close to the walls around the game area and try to find the exit which is clearly demarcated by a specific sound. It’s a great example of spatial awareness through music, it had a great comedic voice over commenting on how well you’re doing and you even get to fight creatures by following an order of sounds like “Simon Says”. The sounds get louder the closer you get to the walls and you hear a “thump” when you hit a wall, if you’re in the top right section of the map the sounds are higher pitch and the lower sections are lower pitch so you can know the direction you’re walking through those sound cues.
Even though there is a wide variety of topics in these games they feel quite similar between each other, they’re mostly about timing and choosing sides, like pressing left, up or down to hit or avoid some object coming at you; and they’re also quite simple and short, more like experi- ments than actual games.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
How are we adding senses to VR? Add more tech
Haptic Suits and Gloves or how to run around in VR
There has been development of devices which give players the feel- ing of haptics inside a VR game through gloves like HaptX or Dexmo, which is like an exoskeleton glove around your hand that constraint your movements so when you pick something like an apple in VR, your hand stops at that point and feels like it’s also holding an apple, minus the feeling of weight.
There are also haptic suits in development like the Teslasuit which has 60 haptics points all around the body and lets you feel things like get- ting shot inside a game. It’s an interesting approach for shooters which could be expanded into other games but I feel it’s taking everything into a Ready Player One future which might not be the right path.
Finally there’s the Virtuix Omni, which is just one of the several prod- ucts in development which enable users to run around physically but statically, in order to move around in VR. It’s a bulky product which is connected to your hips to keep you in place and either has an omnidi- rectional treadmill, or a slippery surface where you’d run in your socks.
To be honest I disagree with these approaches because I feel the more separate products you need to buy to get the full experience in VR, the less mainstream it will become, it can be a good approach for special- ised VR arcades, but for the general public it lmits access and makes the technology more excluding even than before, this time in monetary terms.
On the other hand, big players like Facebook’s Oculus are going in an opposite direction by allowing hand tracking in their Oculus Quest system. Even though this approach will make everything even more intangible than with controllers, it also opens great possibilities: how could we use objects that we already own in our house to play different games, for example a banana could be a gun or it could be the joystick for an airplane.
HaptX Gloves
FEELREAL
Oculus reveal Hand Tracking update
Teslasuit
Virtuix Omni
Dexmo Gloves
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Why read a book when you have audiobooks
Some people might think that using technologies like VR and AR without visuals defeats their point, but I completely disagree. Not being able to see anything, just darkness, gives you a completely unique experience, a more focused experience. This could create a new format/type of game inside VR or AR experiences.
For example listening to audiobooks, if you look at your surroundings, even if you’re not concentrating on them makes your imagination less vivid than if you had your eyes closed, this is because our visual senses normally over- power the rest of the senses even in situations where it shouldn’t be an in- tegral part of the experience. An audiobook is a completely different expe- rience than reading a book, and some people even prefer it this way, it’s so engrained in us that listening to our parents read books is probably one of our earliest memories. I can be more immersive or interesting when differ- ent characters have different voices, and it opens the possibility of multi- tasking, although this probably reduces the concentration on the book and could be distracting.
Meditation is another powerful experience that happens with your eyes closed, as meditation is about clearing ones’ mind from any mental activity and thought and concentrating for example on the breath, it’s about resting your attention inwardly; keeping your eyes open can consequently distract you from achieving the state of meditation.
There’s even restaurants like Dans le Noir, where you have dinner blindfold- ed and in the dark, which lets the customers focus just on the tastes, smells, sounds and tactile feelings natural in food...even the waiters are blind.
Meditating at the beach Dans le Noir
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Inspiration
The company working hardest, maybe the only one who really is working on this issue is Microsoft Research who have produced most of the concepts some still in experimental mode but great inspiration nonetheless of what others are trying.
Microsoft Project Torino – Code Jumper [13]
A great example in innovation for all through design for the visually impaired is in Project Torino by Microsoft Research. Coding has stayed the same ever since it started, screen-based where you type the commands into the com- puter to then run the code; but Project Torino thought of a new way to teach coding basics to visually impaired students – Code Jumper, a physical pro- gramming language designed to be inclusive for children with all ranges of vision.
“What I like about Project Torino is that you can actually touch, physically, the program,” said Victoria, 14. Touching a program can seem inconceivable, but with code jumper kids could immediately experiment and build programs.
The way they included experts into the process, in this case the VI students, team members began meeting regularly with a small group of kids and get- ting their ideas. Based on the kids’ feedback and ideas, they switched to big- ger plastic shapes that fit easily into kids’ hands, and they created surfaces the kids could rub or squeeze in order to recognize and interact with them.
Through this approach they started seeing the technology from the kids’
perspective. For example, the kids with some vision benefited from bright, contrasting colors. They also found that kids like to work together, guiding each other’s hands, so they built the pods to be about the size of two kids’
hands. ;20; It really opened the eyes of the researchers to different per- spectives, different ways to see the world.
“These students are taking out brightly colored plastic pods, connecting them together with thick white wires and then adjusting the pod’s buttons and knobs. These physical components will be used to create computer pro- grams that can tell stories, make music and even crack jokes.”
Microsoft Canetroller [14]
Microsoft Research’s Canetroller demonstrated how rendering virtual ob- jects haptically, including simulating materials’ properties and textures, could enable users who were completely blind to successfully navigate and understand virtual scenes when paired with a novel haptic controller that mimicked the interaction of a white cane.
Canetroller provides three types of feedback: (1) physical resistance gen- erated by a wearable programmable brake mechanism that physically im- pedes the controller when the virtual cane comes in contact with a virtual object; (2) vibrotactile feedback that simulates the vibrations when a cane hits an object or touches and drags across various surfaces; and (3) spatial 3D auditory feedback simulating the sound of real-world cane interactions [15]
Microsoft Seeing VR
Microsoft Research found a great solution to solve many problem for VI us- ers with the focus on the visual side of things. Not a bad option and very use- ful for most of the population - so I’d want to try a different approach, one that completely forgets about making the visuals easier to understand.
Playing through sounds by Andreas Refsgaard [16]
Digital artist and designer Andreas Refsgaard created a program which would enable gamers to play the old school game Wolfenstein 3D (1992) just using sounds like claps to open doors, whistles to move forward
Audio in AR Space by Zach Lieberman [17]
Artist and educator Zach Lieberman uploaded a sketch created in open- frameworks which mapped audio in 3D space and this was shown through the sound waves revolved into spiky cylinders. If you moved towards the line of audio and went through it you’d hear the message backwards, and once you followed the line in the direction it was created you’d hear the saved au- dio message or sound saved in the space it was created. Mezmerizing stuff.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Inspiration from Microsoft Research
"I just felt very independent, and I liked that" said Daniel "To just do cod- ing, it was a fun experience" [13]
Project Torino - Code Jumper Canetroller
Seeing VR
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Games are the perfect ground for exploration and experimentation as it has no boundaries of what you can do with it, as long as it’s fun and excit- ing. Compared to working with utilitarian applications which are more con- strained because you have an actual problem to solve, games are so differ- ent between each other while still falling under the same category of game, there’s much more diversity in games than even in movies or music, and every year, even though it feels like everything is invented, new ways to play keep coming up.
Games are also important for another reason, they improve people’s lives even though most of the time they’re underestimated in the entertainment and design industries.
"Games do so much for me in managing my anxiety on the metro and stress about nearly getting run over and just general ability to function; I really wanted to play games. Games help me even out drastic emotions by taking away my attention for a little bit until I’m not as activated." [18]
But this is just the starting point, ideally everything that is learnt from this project would be then translated into other apps or even full operating sys- tems in VR or AR. How to guide someone through instructions, move around in the environment and attract their attention when needed are all things that games do really well. Especially due to the fact that games have in- corporated 3D environments much more prominently than apps, which are mostly 2D screens. How would the internet look if it was more inspired by gaming rather than by classic Human Computer Interaction?
Nintendo is one of the biggest players in this area, in every new game they start with gameplay first, and then they create the characters and stories around that to keep everything consistent and give an extra meaning to the gameplay. Their latest innovation apart from the Switch console was the Nintendo Labo, where they concentrated on creating more tactile control- lers through a set of cardboard pieces which you can f old into shape and put together without any glue or screws to make your games more immersive and fun, for example by creating fishing rods, a small keyboard to create mu-
sic, and even a full robot suit connected through strings so the movements the player makes get translated into the moves the robot in-game makes.
I feel this approach would be great for any kind of player but especially blind players as they would have a much more tactile feel to the controls and the games they are playing, for example while riding a bike and turning to one side or the other with the vehicle kit. This approach has also been introduced into VR/MR scenarios to give it a new tactility that would be impossible to get with the available controllers.
Games, an excuse to improve the whole technology
Nintendo Labo Variety Kit and VR Kit
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
3.0 APPROACH + METHOD
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
What I call Virtual realms is known as The virtuality continuum: which is a continuous scale ranging between the completely virtual, a virtuality, and the completely real, reality. The reality–virtuality continuum therefore encompasses all possible variations and compositions of real and virtual objects... The area between the two extremes, where both the real and the virtual are mixed, is called mixed reality [19]
Right now we are still separating them through screens where we choose when to enter the digital world or leave it, even though it’s getting harder and harder due to the addictive design of the apps we use everyday. But it will also enter our lives through our Smart Homes or Internet of Things (IoT); which is why I’ll also take it into account in this triad of technologies which could benefit from using the same modalities like sound, move- ment,etc. which I’ll be exploring.
It has been established many times throughout this report that "VISION rules everything around me" to paraphrase the Wu Tang Clan, but the problem is not only shown in the products released, but the materials and design methods used that create these products; and the visual culture in general that has gained importance throughout the decades, from audio in the radio to video with TV, and from reading posts to watching video in the internet era, flooding us with content that we can only interact with through our eyes.
Because the design materials are heavily focused on vision too, those will have to be modified to fit a non visual approach, how to brainstorm, save sketches and develop movement or sounds were some of the expected challenges.
Insights from Background Research
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
From the Background section it’s clear there is a whole research area that is being barely exploited and which could be used to great benefit for the technologies we deal with in this project. Focusing on the senses not only creates innovation, but this innovation is more inclusive, and even if later on you would add visuals, it would still be accessible enough for all.
The way this project is being approached is through finding a series of what we believe is common knowledge about VR technologies and flip it on its head - as an experiment to see what could be produced through op- posing values than the current ones. Like the quote that opened this re- port, to make the known unknown and understand it deeper due to having to question it in other terms. To see if a different future can be achieved.
The users we design for are based on an incomplete form of ourselves as humanbeings illustrated beneath. Huge eyes, small ears and mouth to talk, no nose or sense of taste, and just one or a couple of fingers to interact with most of the tech products nowadays, no body at all, just floating ghosts.
These are the key factors I’ll go against in my own Design Principles to use as a guideline and not loose track inside the huge worlds of these techno- logical advances and realities.
Not only this will be challenged but also the current user centered design methods, as there is much talk about it being user-centered but doesn’t seem to see its users as human beings with physical and other sensory capabilities more than a brain.
This new focus will have to be shown to users who like the one quoted above might find inconcievable that something can be translated from 3D into a 2D shape. So the post-its will have to go, in exchange for audio and physical sketches which would be recorded through video, and most importantly as experience prototypes one would be able to try and feel themselves, because some things have to be felt to be understood, no amount of talking would equate it.
Leaving behind:
- Visually Impaired Users
- People who think the tech is too complex for them
- People who prefer to use their bodies than just their minds - manual learners
We could now include:
- Visually Impaired Users or other sensory capabilities
- People who know how to interact with day to day objections and actions - Other types of learners that are not just visual - manual or emotional learners
Flip the norms - “make the known unknown”
“Everything in my world is 3 dimensional, everything I touch or experience, so it’s very difficult for me to imagine putting a 3D thing onto a flat piece of paper...like it doesn’t make sense to me” - quote from Tommy Edison in his Youtube video - Can Blind People Draw?
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
The design principles came from grabbing everything I disagreed with in the previous sections and the exclusive approaches that had been tried before to provide a new framework which would produce innovative work and not walk on treaded ground. These DESIGN PRINCIPLES were used to guide me through the project and inform my methodology and the experiments to come.
People think tech like VR is more mature than it actually is, defending this opinion using the Gartner Hype Cycle. [20] My approach is to bring it back to a new starting point to open it up and reimagine how things could be done in a different way while still be considered VR because it’s already getting stale. I’m proposing that this is the perfect time that we should be exploring everything about these technologies in crazyand opposite ways to see and expand what the technology really can do.
Perfect timing Design Principles
Strengthen other senses to overcome reliance on vision - Sound, touch, movement (body & space), balance, passing of time - Not just a problem for the blind community, but for everyone Make tech inclusive from its inception
- VR has affordances that uses the natural capabilities of VI users - Use less tech devices by using own objects/body as input
Unpack the technology to discover the affordances of the tech - By exploring form an expression of non-visuals
- Breaking the boundaries of what we think is VR - Thinking differently through new design methods - Designing for a rich sensory experience
vs what I’ll be focusing on
REFERENCE https://www.gartner.com/en/research/
methodologies/gartner-hype-cycle
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Research through design - trying things out
After the desktop research weeks were done, the next step was to go deep into understanding our senses in more general terms, our connections with our bodies and between movement and our mind, as well as trying things out there firsthand like the audiogames which were described previous- ly, and playing with VR and AR apps to understand the state of the current technology.
When you start designing for something other than visual, the design materials we usually use suddenly seem very limiting, or it downright doesn’t work at all, this means that we have to explore new ways of cre- ating prototypes for physicality and movement and find ways to introduce sound into the process. The importance of Experience Prototyping [21] also shows up in this case because it’s very hard to explain these types of experi- ences through talking or through drawing although storyboards and journey mapping are still useful.
These questions are relevant because the more we move away from flat screens and into 3D environments, the more we will all have to make use of these new materials for idea generation and prototyping. Especially in a project like this one where maybe you can’t even show a quick sketch to your target user because they won’t be able to see it, so what do we do when we can’t use sketches and post its? What’s the next step in the evolution of the design process?
For this project I decided to leave out some of our main senses, apart from vision, because I feel they wouldn’t be as interesting or plausible to play with in the future; for example, smell. There’s products being developed to in- clude smell in VR like FEELREAL, and in 1999 there was even a quite suc- cessful company, in terms of earning investors, called iSmell which tried to
"bring the sense of smell to your computer and the internet" but obviously flopped as soon as everyone understood they were trying to solve a non-ex- isting problem, the tech was nearly there, but the need wasn’t. Taste, tem- perature and pain are the other senses left out on purpose.
Focusing mainly on sound and touch, as well as balance, movement and the feeling of time passing. These would be easier to implement and less disruptive as a whole, imagine users wearing a mask that controls the air surrounding them to give a bit more immersion or to "smell" the internet, or electrodes in their tongue so they can taste virtual plates, not good.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
4.0 MATERIAL EXPLORATION
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Working in Sprints based on Design Principles
Working in Sprints is very common in digital product and games development, so a similar approach seemed the most reasonable, also to produce as many experiments and learning as possible to be able to input them into the next loop/sprint. The design Principles would always be used as a guideline through the fog of uncharted territory, to make sure the direction kept the values it started with and the objective it was looking for could be achieved.
The concepts came out since early in the process as concept embryos which would be fed all the new insights gathered through each loop in order to be- come fully formed examples which could be then used as an example of both the methodology and new ways to imagine VR.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
How a Loop looks from the inside
This was the experiential part of the process, 1st hand trying tech added to interviews with experts and past research would inform what experiments to conduct, these would then be analysed to get out the key findings from each and introduce these into the concept embryos, as many insights would fit in more than one concept/provocation. Everything added and learnt would be again sent into the next loop for validation and to gather further insights.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
My Activities
Interviews Experiments Workshops
Interviews are common methods in user cen- tered design, although the current way of do- ing things is to include people that are relevant to the issue at hand i.e. the users, or sociologist and physicologists who have a broader view of any issues the users could have.
My approach will not only include people from the game and tech development worlds, like game developers, MR experts and audio en- gineers; but also people currently considered outside of it, but which I feel should and will be included more prominently during the process of these immersive technologies: people con- nected to the human body and movement like physiotherapists and dancers as well as our key “extreme“ user - Visually Impaired gamers to have a holistic inclusive view since the start.
Include experts in the tech side of things and the human side of things.
These activities aren’t showed in chronological order in the report but grouped by method to make things clearer to the reader, whilst during the process it was much more chaotic and mixed, interviews happening before or after experiments and workshops could be used to validate them and then inform the next experiment/workshop for the next loop. In all honesty the interviews were as insightful as the experiments and the workshops, it really helped understand how people percieve things differently and to know what others inside this tech world thought of the provocations being proposed, if the old guard didn’t seem too convinced then I was moving in the right direction because nothing that is really innovative happens unquestioned by the previous generations.
The experiments were one of the key parts of the methodology, they started as improvisa- tions to see what directions they could take, but once it started opening some interesting areas linked to the rest of the research, they started being more focused on things which could be used in the concepts.
In general the experiments aims were to try and give me and the project an under- standing about our bodies and our senses, what it can do well or not, and how it can be tricked.
These experiments were all very lo-fi and that was on purpose to see if it was possible to mimic VR or AR experiences without hav- ing to go into Unity and code a game or app.
Just using speakers across a room or my own voice as a narrator in improv theatre style ac- tions to test and validate ideas, or to scratch them alltogether.
Workshops were also like experiments but in- volved maybe more people at the same time or just went deeper into development mode for a specific idea
They would happen in two phases, when pos- sible, where the first workshop would be more improvised just to check if the idea would work and something could come out of it; and the next workshop would be more concentrated - with a clearer objective - even though it was still all about improvisation and brainstorming.
Some workshops were moreinsightful than others due to a lack ofproper boundaries for the concepts to come out shaped in a use- ful form or because a second part couldn’t be done after we all had to go into quarantine mode. The last one was tried in Zoom and it half-worked due to the limitations of the video calling platform.
Seb a s tia n de C a b o P or tu g a l
MFA Thesis Project 2020
Interview - Luca Contato from Rising Pixel
Rising Pixel is a video game company based in Gran Canaria which I had the chance to meet in a game jam they organised a couple of years ago. The game jam they organised this year was an accessibility one and they have also done what I believe is the only Audio Game Jam done in the world. They make audiogames for commercial purposes and they are quite successful, although “success in the audiogame world means selling 5,000 units”. They have also explored many ways of controlling games like through voice or through a midi controller.
I thought they would be great people to talk to as they have plenty of exper- tise working with games specifically made for blind people and involving blind game testers into their process. He was interested in the concept buy also quite averse to it from a commercially viable point of view, which I ex- plained was not my objective in this project as I have the chance to explore something which could or could not be viable in the future but that’s exactly what I want to prove. The main points I got out of the conversation with him were:
“When we go to a games conference with our audiogames, no one is inter- ested because they say I can see, so this game is not for me/ not interest- ing”- Visuals are so engrained into what we expect from games that the lack of it is a disadvantage, and it’s true there’s a lot of people who care about the graphics most, but in general I think if it has fun gameplay then graphics come second.
“No blind person would buy a VR set”/”Why do blind people need VR if they can’t see?” – I think this is a valid point from his side as he is thinking com- mercially viable products, but I also believe he has a constrained view about what VR is and could be. Blind people might never buy a VR set because there’s nothing for them but if it was more inclusive maybe they would; it’s like a snake biting its own tail – no one makes anything inclusive, so the blind don’t buy it cause there’s nothing for them, and there’s nothing for them be- cause blind people don’t buy it... Also VR doesn’t have to be like the sets that
are currently in the market, if you can move around a space with headphones that can track the space then this is also a Virtual reality, VR doesn’t mean Visual Reality.
He also thought some of my ideas were a bit dangerous, like controlling an AR Operating System by using objects you have in your house or the location based AR game which is only audio, because it would not let them hear their surroundings which is their main sense to know what’s going on around. The solution we could find was using Bone Conduction headphones.
Blind Arena Tournament - blind multiplayer game developed by Rising Pixel
