#	Pagina
attuale pagina	/open-h2020/projects/206993/results.html

Report

Teaser, summary, work performed and final results

Periodic Reporting for period 2 - OPTINT (OPTINT: Optimization-based Design of Interactive Technologies)

Teaser

Summary

As digital technology moves further away from the desktop setting, it is becoming increasingly clear that conventional interfaces (i.e., mice and keyboards) are no longer adequate means for interaction and that the traditional computing paradigm will be replaced or complemented by new forms of interaction with digital information. While the exact nature of future interfaces is not known, we already do know that future devices will be mobile, will be equipped with ample computational power and will utilize several sensing capabilities. Furthermore, we will interact with diverse set of devices and interfaces such as wearable computers, head-worn displays. Finally, today everybody uses computing devices and the expectation is that these are usable without any prior training or deeper understanding of the technical aspects.

Contemporary user interface (UI) design techniques have been developed for the era of the PC, but designing for these emerging interaction paradigms is a much more diverse and difficult problem and requires reasoning about i) choice of sensing technology ii) sensor placement and configuration iii) and finally reasoning about appropriate input recognition algorithms. These aspects are intertwined with usability in an inseparable fashion.

The main goal of OPTINT is to lay the foundations for the design and implementation of 21st century interactive technologies. In order to achieve this goal the ERC sponsored team works on computational approaches to model human behavior, algorithms to interpret this data and to extract semantic meaning from the low-level observations and to predict the state of the user into the future. This state includes observable aspects such as the location and movements of the user but also unobservable aspects such cognitive load and attention. With such models in place the team then develops algorithms for the computational design of interactive technologies. Such tools may then help a designer to build user interface technologies while jointly considering low-level sensing technology and usability concerns. Leveraging optimization-based algorithms, it will allow for a faster and more efficient exploration of the design space, leading to better designs of user interfaces and to reduced cost of the design process.

Work performed

The ERC funded team has made significant progress on the two main fronts of the project.

First, we have developed a number of data-driven models of human behavior. Human activity is inherently complex, non-linear and over long timescales becomes a stochastic process. However, many applications require predictive models of human activity, for example to design better user interfaces one needs to reason about human action sequences, or in human-robot interaction the agent needs to plan with respect to the current and future state of the human in order to minimize latency. Our lab is at the forefront of data-driven user modelling efforts reaching from algorithms to predict 3D motion data into the future, to inferring cognitive load from pupil dilatation to inferring visual attention from images alone.

Second, we have developed several computational UI adaptation techniques that leverage the inferred user state and acquired models to adjust the type, timing and location of information that is displayed. This work is foundational to an emerging area in computational system and user interface design as it changes the way we think about and ultimately design user interfaces. For example, in a recent paper to appear at the International Symposium on User Interface Software and Technology (ACM UIST) we demonstrate how an estimate of cognitive load can be leveraged to continuously optimize what information is shown to the user, where it shown and how much screen real-estate is dedicated to the element in Mixed Reality applications. In the context of the OPTINT project we are developing several related algorithms and combine them with traditional User Interface methods in order to give designers better tools to build more usable interactive technologies.

Final results

The team has made significant contributions to the state of the art in a variety of important sub-areas including data driven user modelling, human motion prediction, gaze and attention estimation and has leveraged these models in optimization-based design tools. The work has lead to several top-tier publications at the premier venues for computer vision (CVPR, ICCV), machine learning (ICLR) and human computer interaction (CHI UIST). The team will continue its outstanding work on both data driven user modelling and optimization-based design tools and will continue to unify the findings in order to provide a coherent framework for the optimization based design of interactive technologies.