Report

Teaser, summary, work performed and final results

Periodic Reporting for period 2 - MAMEM (Multimedia Authoring and Management using your Eyes and Mind)

Teaser

Traditionally, human computer interaction has been grounded on the principle of a healthy neuromuscular system allowing access to conventional interface channels like mouse, and keyboard. However, the used of these interfaces is limited by the pre-requisite of a healthy...

Summary

Traditionally, human computer interaction has been grounded on the principle of a healthy neuromuscular system allowing access to conventional interface channels like mouse, and keyboard. However, the used of these interfaces is limited by the pre-requisite of a healthy neuromuscular system. It has been only recently that devices recording accurate information from eye movements and brain electrical signals has given a new perspective on the control channels that can be utilized for interacting with a computer.

The necessity of using these control channels has been motivated in the context of assisting people with disabilities. Loss of voluntary muscular control but with preserved intellectual functions is a common symptom of neuromuscular conditions (i.e. muscular dystrophy, multiple sclerosis, Parkinson disease, spinal cord injury) leading to functional deterioration and poor quality of life. Among a variety of functional deficits the most affected people may also lose their ability to operate software tools and, as a result, become marginalized and unable to keep up with the rest of the society in a digitized world.

MAMEM’s overarching goal is to integrate people with disabilities back into society by endowing them with the critical skill of managing and authoring multimedia content using novel and more natural interface channels. These channels will be controlled by eye-movements and mental commands, significantly increasing the potential for communication and exchange in leisure (e.g. social networks) and non-leisure context (e.g. workplace).

Work performed

In the direction of capturing and recording eye-movements brain electric signals and bio-measurements, MAMEM has investigated two types of configurations: a) “heavy-weight” consisting of high-cost and high-accuracy devices, and b) “light-weight” consisting of low-cost and lower accuracy devices. On the software side we have adopted LabStreamingLayer to serve as the middleware of our system and developed the necessary wrappers for the utilized devices. Moreover, considerable work was performed on ensuring the synchronization of the independent streams going through LabStreamingLayer and a number of best practices have been developed for ensuring the reliable acquisition of eye-gaze, EEG and GSR data in non-laboratory environments.

MAMEM has also worked in the direction of translating the acquired signals into meaningful control. More specifically, we have developed the GazeTheWeb framework as the means of communication between the user and the computer using an eye-tracker, and we have also studied the different techniques for understanding brain signals like Steady State Visual Evoked Potentials (SSVEP), Senso-motoric Rythms (SMR) and Error-related potentials (ErrPs). Our work in these directions has resulted in a significant number of high-quality publications, while a number of novel prototypes for multi-modal interaction were developed including an Error-aware keyboard based on ErrPs and MM-Tetris (i.e. a hands-free version of Tetris based on eye-movements and sensory-motor rhythms). Finally, GazeTheWeb has evolved into a mature framework for performing web-browsing using your eyes and constitute the main product derived from MAMEM.

In terms of integration we have integrated everything related to sensor data acquisition and processing under a single library working as an extension of GazeTheWeb. In this way, MAMEM succeeded in delivering a system where the process of signal acquisition and processing is completely transparent to the end-user. In addition, the system was also equipped with a step-by-step installer and a logging mechanism, allowing us to have the system tested in the home environment of our end-users for a period of one month.

Next to the work performed on signal acquisition and processing, MAMEM has also contributed in the theory and practice of persuasive design for user engagement. More specifically, following an extensive literature review and the profiling of our end-users into a set of personas, a lab study has been conducted showing the cases where the use of persuasive design principles is actually effective in making the user interfaces more engaging. In addition, the use of persuasive design has been also employed in the context of a dashboard aiming to increase the social inclusion of its users by logging and quantifying the participants’ digital social inclusion behavior.

In the direction of assessing the degree of success in bringing disabled people back into society we have decided to use pre and post MAMEM usage data so as to detect shifts in social inclusion. More specifically, qualitative data were collected through questionnaires and the participants’ spontaneous feedback, while quantitative data were collected through: a) a logging mechanism recording actions like sites visited, clicks performed, and b) a social media monitoring mechanism collecting activities in online social networks.

Finally, concerning the dissemination and exploitation of our results it has been a common decision among the consortium to place GazeTheWeb at the core of our dissemination and exploitation activities since it has surfaced as the most concrete outcome of MAMEM. Following this decision we have produced a well-elaborated exploitation plan for GazeTheWeb including a) a SWOT analysis for GazeTheWeb and a detailed description of the competitive landscape, b) a deep analysis of the market identifying key players, needs and structure, c) a suggestion for a marketing activity plan with all maj

Final results

In the context of MAMEM we have generated 44 (29 Conferences, 13 Journals and 2 Technical Reports) publications that are categorized under the general topics of: a) Computer use habits of patients with motor disabilities, b) persuasion strategies and persuasive interface design, c) eye-tracking related interfaces and applications, and d) EEG-related methodologies and tools.

In terms of societal impact, MAMEM has successfully completed an extensive set of trials that took place in the home environment of its end-users, allowing them to use the system for their own every-day activities either for leisure (i.e. watching YouTube videos, posting in facebook, etc) and non-leisure (i.e. sending e-mails) purposes. The feedback that we have collected both from the logging mechanism about the use of various different applications, but even more importantly, the spontaneous feedback received from our end-users through WhatsApp messages, e-mails and oral feedback, has been particularly encouraging about the potential impact that such a system could bring to the life of our end-users.

Finally, it terms of socio-economic impact the development of a GazeTheWeb as a reliable and cost-effective tool for browsing the web with your eyes can be considered as a major contribution in the constantly evolving market of assistive devices, where eye control is gradually making its way into the most widespread operating systems (i.e. Windows 10).

Website & more info

More info: http://mamem.eu.