Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - MIM (Enhancing Motion Interaction through Music Performance)

Teaser

As Information Communications Technology (ICT) is now spreading outside of workspaces towards our everyday life, new elements of human life such as culture, emotion or experience must be embraced in technology for Human-Computer Interaction (HCI). Part of this new objective in...

Summary

As Information Communications Technology (ICT) is now spreading outside of workspaces towards our everyday life, new elements of human life such as culture, emotion or experience must be embraced in technology for Human-Computer Interaction (HCI). Part of this new objective in HCI is to build technologies that are “closer”, or more “natural”, to human, promoting the use of body movement in HCI. Exploring this challenge has been the cornerstone of digital arts and especially performing arts involving digital media. For example, music technologists have leveraged on motion sensing technology to explore new types of interaction between the body and digital medias. Nevertheless, there is a general agreement that these new interactive systems remain less satisfying in terms of expressive control compared to traditional instruments, and, importantly, do not allow to acquire skills through them.

The general objective of the project MIM is to investigate motor learning mechanisms in instrumental music practice in order to enhance movement computational models in movement-based musical interaction systems. The project MIM addresses three specific scientific objectives:
- Objective 1: Study the role of movement task variation in music skill acquisition,
- Objective 2: Study the nature of movement variability in the development of motor learning,
- Objective 3: Build musical interactive systems based on models of movement variability.

The importance of the project is two-fold. First enriching our knowledge on motor skill acquisition in complex human activity such as music performance might trigger new ways of addressing practice in performing arts, sports and rehabilitation. Second conceiving music technology facilitating motor skill learning would impact positively creative industries.

Work performed

The project started by an exhaustive literature review on sensorimotor learning and music performance.

A first experimental study has been conducted showing that the motor task variability has a dissociable effect on timing and motor skill acquisition in piano sequence learning. The results were published in PLoS ONE:
- Caramiaux, B., Bevilacqua, F., Wanderley, M., and Palmer, C. Dissociable effects of practice variability on learning motor and timing skills. PLOS ONE. 13(3), 1-18. 2018.

A second experimental study has been initiated (pilot study has been run) investigating the individual-specificity of motor variability in piano sequence learning and the role of cognitive and biomechanical processes on motor variability. A conference proceedings showing preliminary results has been published in MOCO:
- Caramiaux, B., Bevilacqua, F., Palmer, C., and Wanderley, M. Individuality in Piano Performance Depends on Skill Learning. Proceedings of the 4th International Conference on Movement Computing. 2017

Design methodology to transfer this knowledge to the realisation of movement-based music systems have been explored. A collaborative paper reports on design highlights:
- Höök, K., Caramiaux, B., Erkut, C., Forlizzi, J., Hajinejad, N., et al. Embracing First-Person Perspectives in Soma-Based Design. Informatics. 5(1), 8. 2018

Final results

While often considered as noise stemming from the motor or the sensory system, motor variability has recently been shown to have an important role in sensorimotor learning. Underlying processes vary the motor output in order to learn faster and better certain changes in environment dynamics (motor adaptation) or a new control policy (skill acquisition). This new perspective of motor variability has gained interest since the seminal Nature article in 2014. In music performance, variability has been barely explored. Few studies have shown that motor variability can be used to highlight chunking. However, motor variability as a parameter for skill acquisition in music remains unexplored.

Our first experimental study brings a new contribution to the state of the art in music performance and sensorimotor learning showing the impact of motor variability on motor and timing skill learning.

The impact is important for music pedagogy and the design of innovative skill-facilitating interactive designs. Such impact has recently been discussed during the Piano & gesture capture workshop at GREAM, at University of Strasbourg, and during a workshop dedicated to the topic at the Conference NIME (New Interfaces for Musical Expression) in Copenhagen.

Website & more info

More info: http://mim.ircam.fr.