In Your Face! – Designing Future Interaction Models for Internet of Things and Augmented Reality

(1)

LUND UNIVERSITY PO Box 117 221 00 Lund +46 46-222 00 00

In Your Face! – Designing Future Interaction Models for Internet of Things and

Augmented Reality

Alce, Günter

2018

Document Version:

Publisher's PDF, also known as Version of record Link to publication

Citation for published version (APA):

Alce, G. (2018). In Your Face! – Designing Future Interaction Models for Internet of Things and Augmented Reality. Ergonomics and Aerosol Technology, Department of Design Sciences, Lund University.

Total number of authors: 1

General rights

Unless other specific re-use rights are stated the following general rights apply:

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.

• You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal

Read more about Creative commons licenses: https://creativecommons.org/licenses/ Take down policy

If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

(2)

(3)

Lund University Faculty of Engineering Department of Design Sciences Ergonomics and Aerosol Technology

ISBN 978-91-7753-728-1 ₉789177

537281

Printed by Media-T

ryck, Lund 2018 NORDIC SW

AN ECOLABEL 3041 0903

More and more devices are being connected to the Internet but how should we discover and directly interact with all of them in an intuitive and comfortable manner? Up until now, smartphones have shown potential for managing the Internet of Things (IoT) environments, but we cannot rely on that technology. Wearables, on the other hand, are becoming more mature and are available in many different form factors. It is speculated that combining wearables with augmented reality (AR), which has the ability to merge the real world with the virtual, is more suitable.

The aim of the research presented in this thesis was to develop and explore three tools that can be used for prototyping AR and IoT interaction and to introduce four interaction models for controlling IoT devices.

GÜNTER ALCE started to work at Ericsson as a software engineer after his Master of Electrical Engineering in 2000. After having worked with low-level software, he moved on to Sony Ericsson working with user interfaces. Next, he started to work at the technology office at Sony as Senior En-gineer. Eventually, Günter looked for new challenges and decided to do his Ph.D. at the Department of Design sciences at Lund University. His research is focused on Augmented Reality interaction for the glass form-factor.

(4)

(5)

(6)

(7)

2UJDQL]DWLRQ /81'81,9(56,7< 'RFXPHQWQDPH '2&725$/',66(57$7,21 'HSDUWPHQWRI'HVLJQ6FLHQFHV (UJRQRPLFVDQG$HURVRO7HFKQRORJ\ 6ZHGHQ 'DWHRILVVXH $XWKRU *QWHU$OFH 6SRQVRULQJRUJDQL]DWLRQ 7LWOHDQGVXEWLWOH ,Q<RXU)DFH±'HVLJQLQJ)XWXUH,QWHUDFWLRQ0RGHOVIRU,QWHUQHWRI7KLQJVDQG$XJPHQWHG5HDOLW\ $EVWUDFW ,WLVHVWLPDWHGWKDWWKHQXPEHURIGHYLFHVFRQQHFWHGWRWKH,QWHUQHWZLOOEHELOOLRQE\+RZVKRXOGDQRW VRWHFKVDYY\HQGXVHUEHDEOHWRGLVFRYHUDQGGLUHFWO\LQWHUDFWZLWKDP\ULDGRIFRQQHFWHGWKLQJVLQDQLQWXLWLYH DQGFRPIRUWDEOHPDQQHU"8SXQWLOQRZVPDUWSKRQHVKDYHVKRZQSRWHQWLDOIRUPDQDJLQJWKH,QWHUQHWRI7KLQJV ,R7HQYLURQPHQWVEXWZHFDQQRWUHO\RQWKDWWHFKQRORJ\:HDUDEOHWHFKQRORJ\GHYLFHVDUHPDWXULQJDQGDUH DYDLODEOHLQPDQ\GLIIHUHQWIRUPIDFWRUVLQFOXGLQJKHDGZRUQGLVSOD\V+:'VVPDUWZDWFKHVDQGZULVWEDQGV 7KH\HQDEOHDFFHVVWRLQIRUPDWLRQDWDJODQFH7KH\DUHLQWHQGHGWRDOZD\VEHµµRQ¶¶WRDOZD\VEHDFWLQJDQGWR DOZD\VEHVHQVLQJWKHVXUURXQGLQJHQYLURQPHQWLQRUGHUWRRIIHUDEHWWHULQWHUIDFHWRWKHUHDOZRUOG$WHFKQRORJ\ VXLWDEOHIRUWKHVHNLQGRIXVHULQWHUIDFHVLVDXJPHQWHGUHDOLW\$5GXHWRLWVDELOLW\WRPHUJHWKHUHDOZLWKWKH YLUWXDO+RZHYHUSURWRW\SLQJ$5XVHULQWHUIDFHVWRGLVFRYHUDQGFRQWUROFRQQHFWHGWKLQJVFDQEHGLIILFXOWDQG FRVWO\EHFDXVHLWLQYROYHVDQXPEHURIGLIIHUHQWGHYLFHVDQGV\VWHPVZLWKYDU\LQJOHYHOVRIWHFKQRORJLFDO UHDGLQHVV 7KHDLPRIWKHUHVHDUFKSUHVHQWHGLQWKLVWKHVLVZDVWRGHYHORSDQGH[SORUHWKUHHWRROVWKDWFDQEHXVHGIRU SURWRW\SLQJ$5DQG,R7LQWHUDFWLRQDQGWRLQWURGXFHIRXULQWHUDFWLRQPRGHOVIRUFRQWUROOLQJ,R7GHYLFHV2QHRI WKHWRROVLVEDVHGRQUHDOZRUOG:L]DUGRI2]:2=SURWRW\SLQJPHWKRGZKLFKOHWDKXPDQRSHUDWH XQGHYHORSHGFRPSRQHQWVRIDWHFKQLFDOV\VWHPDQGWKHRWKHUWZRDUHEXLOWRQYLUWXDOUHDOLW\95EDVHG SURWRW\SLQJIRUDQ,R7HQYLURQPHQW7KHLQWHUDFWLRQPRGHOVZHUHGHYHORSHGIRUGLIIHUHQWIRUPIDFWRUV2QHLV EDVHGRQDVPDUWZDWFKIRUPIDFWRUDQGDQLQWHUDFWLRQPRGHOFDOOHGUbiCompassDQGWKUHHDUHEDVHGRQ+:' IRUPIDFWRUDQGLQWHUDFWLRQPRGHOVFDOOHGFloating IconsWorld in MiniatureDQGFloating MenuUHVSHFWLYHO\ 2YHUDOOWKHUHVHDUFKSUHVHQWHGLQWKLVWKHVLVIRXQGWKHWKUHHSURWRW\SLQJWRROV±:R]$5G,9$5DQG958EL±WR EHXVHIXOIRUSURWRW\SLQJ$5DQG,R7LQWHUDFWLRQ2QHLPSRUWDQWWDNHDZD\IRURUJDQL]DWLRQVWKDWGHYHORS,R7 V\VWHPVRUVHUYLFHVLVWRXVH95WRVLPXODWHGLIIHUHQWVFHQDULRVDQGLQWHUDFWLRQV7KHWZR95EDVHGSURWRW\SLQJ WRROVDUHVXLWDEOHIRUVLPXODWLRQVRIPRUHFRPSOH[VFHQDULRVVLQFHUHJLVWUDWLRQDQGWUDFNLQJFDQEHHDVLO\ VLPXODWHGZKLOH:R]$5GLVVXLWDEOHIRUSURWRW\SLQJVLPSOH$5XVHULQWHUIDFHV 2YHUDOOWKHLQWHUDFWLRQPRGHOVSUHVHQWHGXWLOL]HWZRIRUPIDFWRUV±VPDUWZDWFKDQG+:'±ERWKRIZKLFKGLG ZHOOGXULQJWKHH[SHULPHQWV7KH\ERWKIRFXVRQWKUHHDVSHFWVGLVFRYHULQJFRQQHFWHGGHYLFHVVHOHFWLQJDQG FRQWUROOLQJFRQQHFWHGGHYLFHVDQGWKDWWKHXVHUQRWQHHGLQJWRVWDUWDQDSSOLFDWLRQ$QH[DPSOHRIWKHODWHULV WKDWWKHXVHULQWHUIDFHVKRXOGMXVWDSSHDUZKHQDSHUVRQHQWHUVDVPDUWRIILFH .H\ZRUGV ,QWHUQHWRI7KLQJV$XJPHQWHG5HDOLW\:HDUDEOHV9LUWXDO5HDOLW\3URWRW\SLQJ8VHULQWHUIDFH &ODVVLILFDWLRQV\VWHPDQGRULQGH[WHUPVLIDQ\ 6XSSOHPHQWDU\ELEOLRJUDSKLFDOLQIRUPDWLRQ /DQJXDJH(QJOLVK ,661DQGNH\WLWOH ,6%1 5HFLSLHQW¶VQRWHV 1XPEHURISDJHV 3ULFH 6HFXULW\FODVVLILFDWLRQ

(8)

(9)

&RYHUSKRWRDQGGHVLJQE\*QWHU$OFH &RS\ULJKW*QWHU$OFH )DFXOW\RI(QJLQHHULQJ 'HSDUWPHQWRI'HVLJQ6FLHQFHV (UJRQRPLFVDQG$HURVRO7HFKQRORJ\ ,6%13ULQWHG 3GI 3ULQWHGLQ6ZHGHQE\0HGLD7U\FN/XQG8QLYHUVLW\ /XQG 12 5' ,&6:$1 (&2/$_% (_/

(10)

(11)

$FNQRZOHGJHPHQW $EVWUDFW 6DPPDQIDWWQLQJ /LVWRILQFOXGHGSDSHUV 3DSHU:R]$5G$:L]DUGRI2]0HWKRGIRU:HDUDEOH$XJPHQWHG 5HDOLW\,QWHUDFWLRQ±$3LORW6WXG\ 3DSHU$3URWRW\SLQJ0HWKRGWR6LPXODWH:HDUDEOH$XJPHQWHG5HDOLW\ ,QWHUDFWLRQLQD9LUWXDO(QYLURQPHQW±$3LORW6WXG\ 3DSHU8VLQJ95DV3URWRW\SLQJ7RROIRU,R7,QWHUDFWLRQLQD6PDUW+RPH (QYLURQPHQW 3DSHU8EL&RPSDVV$Q,R7,QWHUDFWLRQ&RQFHSW 3DSHU$5DVD8VHU,QWHUIDFHIRU7KH,QWHUQHWRI7KLQJV±&RPSDULQJ 7KUHH,QWHUDFWLRQ0RGHOV 2WKHUSXEOLFDWLRQVE\WKHUHVSRQGHQW ,QWURGXFWLRQ 5HVHDUFKREMHFWLYHV 1DYLJDWLQJWKHHPHUJLQJWHFKQRORJ\ODQGVFDSH 7KHRUHWLFDORYHUYLHZ ,QWHUQHWRI7KLQJV $XJPHQWHG5HDOLW\ $5WHFKQRORJ\ $SSOLFDWLRQDUHDVIRU$5 :HDUDEOHWHFKQRORJ\ 2YHUYLHZ +HDGZRUQGLVSOD\ :DWFK 9LUWXDOUHDOLW\ 95WHFKQRORJ\

(12)

,QWHUDFWLRQPRGHOV 8EL&RPSDVV )ORDWLQJ,FRQV :RUOGLQ0LQLDWXUH )ORDWLQJ0HQX 0HWKRGVXVHGIRUGDWDJDWKHULQJ 0HWKRGV 4XHVWLRQQDLUHV ,QWHUYLHZV &RQFXUUHQWWKLQNDORXG 2EVHUYDWLRQ 6WDWLVWLFV 3DUWLFLSDQWV 3DSHU6XPPDULHV 3DSHU:R]$5G$:L]DUGRI2]0HWKRGIRU:HDUDEOH$XJPHQWHG 5HDOLW\,QWHUDFWLRQ±$3LORW6WXG\ 3DSHU$3URWRW\SLQJ0HWKRGWR6LPXODWH:HDUDEOH$XJPHQWHG5HDOLW\ ,QWHUDFWLRQLQD9LUWXDO(QYLURQPHQW±$3LORW6WXG\ 3DSHU8VLQJ95DV3URWRW\SLQJ7RROIRU,R7,QWHUDFWLRQLQD6PDUW+RPH (QYLURQPHQW 3DSHU8EL&RPSDVV$Q,R7,QWHUDFWLRQ&RQFHSW 3DSHU$5DVD8VHU,QWHUIDFHIRU7KH,QWHUQHWRI7KLQJV±&RPSDULQJ 7KUHH,QWHUDFWLRQ0RGHOV 'LVFXVVLRQ 3URWRW\SLQJPHWKRGV $ELOLW\WRGHVLJQDQGH[SORUHXVHULQWHUIDFHV )LGHOLW\OHYHORIWKHWRROVDQGIRUZKRP" ,QWHUDFWLRQPRGHOV +RZFORVHWRUHDOSURGXFWVDUHZH" )RUZKRPDUHWKHLQWHUDFWLRQPRGHOVLQWHQGHG" 0HWKRGRORJLFDOLVVXHV

(13)

)XWXUHSRVVLELOLWLHV +RZIDUFDQZHWDNHWKHSURWRW\SLQJWRROV" +RZZLOOZHLQWHUDFWLQWKHIXWXUHZLWK,R7GHYLFHV" $UHWKHUHDQ\SUREOHPVWRFRQVLGHU" &RQFOXVLRQV )XUWKHUUHVHDUFK 5HOHYDQFHWRVRFLHW\ 5HIHUHQFHV $SSHQGLFHV3DSHUV

(14)

(15)

(16)

(17)

(18)

(19)

(20)

(21)

(22)

(23)

(24)

(25)

(26)

(27)

(28)

(29)

(30)

(31)

(32)

(33)

x x

(34)

(35)

(36)

(37)

(38)

(39)

(40)

(41)

(42)

(43)

(44)

(45)

x x

(46)

(47)

(48)

(49)

(50)

(51)

(52)

(53)

(54)

(55)

(56)

(57)

(58)

(59)

(60)

(61)

(62)

(63)

(64)

(65)

(66)

VSDFHRI$5DQG,R7 LQWHUDFWLRQXVLQJDYLUWXDO HQYLURQPHQW 'DWDORJJLQJ 9LGHR REVHUYDWLRQ WHFKQRORJ\QHHGVWR LPSURYHERWKZLWKUHJDUGV WRLQWUXVLYHQHVVDQG SUHFLVLRQ 3DSHU ,QWURGXFHDQGHYDOXDWH WKH95WHFKQRORJ\EDVHG RQURRPVFDOHWUDFNLQJWR SURWRW\SH,R7LQWHUDFWLRQ 7KHWRROLVFDOOHG958EL &DQ95EHXVHG WRH[SORUH,R7 LQWHUDFWLRQ" 6WUXFWXUHG LQWHUYLHZ 1$6$7/; ZLWKRXWZHLJKWLQJ 68634 7KLQNDORXG 9LGHR REVHUYDWLRQ 95KDVDSRWHQWLDOWR EHFRPHDXVHIXO SURWRW\SLQJWRROIRU,R7 LQWHUDFWLRQ 3DSHU 7RLQWURGXFH8EL&RPSDVV FRQFHSWDQGWRFRPSDUHD IXQFWLRQDOVPDUWZDWFK SURWRW\SHRILWZLWKD FRPPHUFLDOPRELOH DSSOLFDWLRQZKHQXVLQJDQ ,R7VROXWLRQ +RZGRHVWKH 8EL&RPSDVV FRQFHSWZRUN FRPSDUHGZLWK WUDGLWLRQDO PRELOH DSSOLFDWLRQV" 6HPLVWUXFWXUHG LQWHUYLHZ 1$6$7/;ZLWK LQGLYLGXDOZHLJKWLQJ 686 9LGHR REVHUYDWLRQ 8EL&RPSDVVZRUNV FRPSHOOLQJO\IRUDFFHVVLQJ LQIRUPDWLRQDWDJODQFH DQGIRUVLPSOHLQWHUDFWLRQ 3DSHU 7RLQWURGXFHDQGFRPSDUH WKUHHEDVLFLQWHUDFWLRQ PRGHOVIRU+:'EDVHG $5ZLWKDIRFXVRQWKH GLVFRYHU\DQGVHOHFWLRQRI ,R7GHYLFHV :KDWDUHWKH SURVDQGFRQVRI WKUHHEDVLF PRGHOVIRU,R7 LQWHUDFWLRQXVLQJ JODVVHVEDVHG $5" 6HPLVWUXFWXUHG LQWHUYLHZ 1$6$7/;ZLWK LQGLYLGXDOZHLJKWLQJ 2ZQ TXHVWLRQQDLUH 9LGHR REVHUYDWLRQ :LWKIHZGHYLFHVWR FRQWUROWKHPRGHOVGLGQRW FDXVHVLJQLILFDQWO\ GLIIHUHQWLQWHUDFWLRQV +RZHYHUZLWKPDQ\ GHYLFHVWRHQJDJHZLWK WKH:,0PRGHOVWRRGRXW DVHVSHFLDOO\GLIILFXOWDQG WLPHFRQVXPLQJ

(67)

(68)

(69)

(70)

(71)

(72)

(73)

(74)

(75)

(76)

(77)

(78)

(79)

(80)

(81)

(82)

(83)

(84)

(85)

(86)

(87)

(88)

(89)

(90)

(91)

(92)

(93)

(94)

(95)