Report
Teaser, summary, work performed and final results
Periodic Reporting for period 3 - SenseX (Sensory Experiences for Interactive Technologies)
Teaser
In the 20th century the industrial demands for a controllable way to describe colours initiated intense research on the descriptions of colours (substantially contributing to advances in computer graphics and image processing, photography and cinematography). Similarly, the...
Summary
In the 20th century the industrial demands for a controllable way to describe colours initiated intense research on the descriptions of colours (substantially contributing to advances in computer graphics and image processing, photography and cinematography). Similarly, the 21st century now demands an in-depth investigation of touch, taste, and smell as sensory interaction modality for interactive systems.
Despite the fact that interactive technologies have permeated our environment (e.g., mobile, ubiquitous computing) and have become an essential part of our everyday life (e.g., work, leisure, education, health, etc.), the senses we call upon to interact with technology are still very limited relying mostly on visual and auditory senses. Given the immediacy of touch and the ubiquity of taste and smell and their importance to health, safety, leisure and work, and a persons emotional wellbeing, future multi-sensory experiences with interactive technologies can have a major impact on society and consumer markets creating entirely new product, technology, and service opportunities. More importantly, multi-sensory experience research can make a step-change in our understanding of the human senses as interaction modalities and revolutionize existing interaction paradigms within Human-Computer Interaction (HCI).
The grand challenge and vision of this project is to gain a rich and integrated understanding on touch, taste, and smell experiences for interactive technologies. We aim to achieve this ambitious grand vision by 1) creating a ‘sensory interaction framework’ on the bases of a systematic empirical investigation of touch, taste, and smell experiences, 2) integrating the generated understanding on the three senses into meaningful and efficient experiential cross-sensory gamuts and interaction principles, and 3) demonstrating the added value of the created experiential understanding on touch, taste, and smell – aka the experiential gamuts – through their integration into the development of multi-sensory systems verifying the short-, mid- and long-term societal and scientific impact (short-term: multi-sensory media experiences; mid-term: interaction concepts for partially sensory impaired people; long-term: multi-sensory interaction approach for life beyond Earth).
This research will pioneer novel interaction concepts for interactive technologies in relation to essential components of multisensory experiences. This project will transform existing interaction paradigm in Human-Computer Interaction (HCI) and likewise impact other disciplines such as sensory and cognitive sciences by delivering ground-breaking new insights on the experiential dimensions underlying neurological processes and human perception.
Work performed
\"The Sussex Computer Human Interaction (SCHI \'sky\') Lab research lies in the area of Human-Computer Interaction (HCI), an area in which research on multisensory experiences makes a difference on how we design and interact with technology in the future. The interdisciplinary team explores tactile, gustatory, and olfactory experiences as novel interaction modalities.
Key publications around Touch:
*Marianna Obrist, Sriram Subramanian, Elia Gatti, Benjamin Long, and Thomas Carter. 2015. Emotions Mediated Through Mid-Air Haptics. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI \'15). ACM, New York, NY, USA, 2053-2062. DOI: https://doi.org/10.1145/2702123.2702361
*Dario Pittera, Marianna Obrist, and Ali Israr. 2017. Hand-to-hand: an intermanual illusion of movement. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 2017). ACM, New York, NY, USA, 73-81. DOI: https://doi.org/10.1145/3136755.3136777
*Damien Ablart, Carlos Velasco, and Marianna Obrist. 2017. Integrating Mid-Air Haptics into Movie Experiences. In Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video (TVX \'17). ACM, New York, NY, USA, 77-84. DOI: https://doi.org/10.1145/3077548.3077551
*Maggioni, E., Agostinelli, E, and Obrist, M. (2017). Measuring the Added Value of Haptic Feedback. In proceedings of the International Conference on Quality of Multimedia Experience (QoMEX 2017), May 31 – June 2, 2017 in Erfurt, Germany. DOI: 10.1109/QoMEX.2017.7965670
*Giles Hamilton-Fletcher, Marianna Obrist, Phil Watten, Michele Mengucci, and Jamie Ward. 2016. \"\"I Always Wanted to See the Night Sky\"\": Blind User Preferences for Sensory Substitution Devices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI \'16). ACM, New York, NY, USA, 2162-2174. DOI: https://doi.org/10.1145/2858036.2858241
*Vi, T.C., Ablart, D., Gatti, E., Velasco, C., and Obrist, M. (2017). Not just seeing, but also feeling Art: Mid-air Haptic Experiences integrated in a Multisensory Art Exhibition. In International Journal of Human-Computer Studies, Volume 108, 2017, 1-14. DOI: https://doi.org/10.1016/j.ijhcs.2017.06.004
Key publications around Taste:
*Chi Thanh Vi, Asier Marzo, Damien Ablart, Gianluca Memoli, Sriram Subramanian, Bruce Drinkwater, and Marianna Obrist. 2017. TastyFloats: A Contactless Food Delivery System. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS \'17). ACM, New York, NY, USA, 161-170. DOI: https://doi.org/10.1145/3132272.3134123
*Chi Thanh Vi, Damien Ablart, Daniel Arthur, and Marianna Obrist. 2017. Gustatory interface: the challenges of ‘how’ to stimulate the sense of taste. In Proceedings of the 2nd ACM SIGCHI International Workshop on Multisensory Approaches to Human-Food Interaction (MHFI 2017). ACM, New York, NY, USA, 29-33. DOI: https://doi.org/10.1145/3141788.3141794
*Spence, C., Obrist, M., Velasco, C., and Ranasinghe, N. (2017). Digitizing the chemical senses: Possibilities & pitfalls. In International Journal of Human-Computer Studies, Volume 107, 62-74. http://www.sciencedirect.com/science/article/pii/S1071581917300885
Key publications around Smell:
*Dmitrijs Dmitrenko, Emanuela Maggioni, and Marianna Obrist. 2017. OSpace: Towards a Systematic Exploration of Olfactory Interaction Spaces. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS \'17). ACM, New York, NY, USA, 171-180. DOI: https://doi.org/10.1145/3132272.3134121
*Dmitrijs Dmitrenko, Emanuela Maggioni, Chi Thanh Vi, and Marianna Obrist. 2017. What Did I Sniff?: Mapping Scents Onto Driving-Related Messages. In Proceedings of the 9th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI \'17). ACM, New York, NY, USA, 154-163. DOI: https://doi.org/10.1145/3122986.3122998
*Dmitrijs Dmitrenko, Chi Thanh Vi, an\"
Final results
The first achievements in the SenseX project are linked to the creation of a sensory interaction framework based on a systematic empirical investigation of touch, taste, and smell experiences (see publications above). Based on those sensory-specific investigations we now started investigating cross-sensory experiences that will enable us for the first time to establish an integrated understanding on the three senses alongside audio-visual stimuli. We will progress towards a design framework that will guide the meaningful and efficient integration of sensory stimuli into interactive systems and demonstrate their added value in three areas of interest from a societal and scientific perspective covering different time-scales: short-term: multi-sensory media experiences in the realm of games, television, and online video production; mid-term: novel interaction approaches for people with sensory impairments advancing research efforts in the development of sensory substitute devices; long-term: multi-sensory research for preparing humanity for life beyond Earth (colonization on Mars and long-term space flights).
Website & more info
More info: http://multi-sensory.info/.