Report

Teaser, summary, work performed and final results

Periodic Reporting for period 2 - DE-ENIGMA (DE-ENIGMA: Multi-Modal Human-Robot Interaction for Teaching and Expanding Social Imagination in Autistic Children)

Teaser

DE-ENIGMA aims to revolutionise autism therapy by offering adaptive robot-mediated therapy. DE-ENIGMA wants to realize robust, context-sensitive, multimodal and naturalistic human-robot interaction (HRI) aimed at enhancing the social imagination skills of children on the...

Summary

DE-ENIGMA aims to revolutionise autism therapy by offering adaptive robot-mediated therapy. DE-ENIGMA wants to realize robust, context-sensitive, multimodal and naturalistic human-robot interaction (HRI) aimed at enhancing the social imagination skills of children on the autism spectrum.

In the first period of DE-ENIGMA, 128 autistic children were randomly assigned to participate in robot-assisted or adult-led teaching conditions of a 6-step emotion-training programme. More than 154 hours of raw audio-visual data was collected and has been annotated and will serve for machine analysis of facial, bodily, vocal and verbal behaviours.

Through 2 design iterations and subsequent usability studies, we have explored several game activities aimed at teaching autistic children about facial features and emotional facial expressions. The partners have investigated appropriate methods for children to interact with the game and with the robot.

The DE-ENIGMA team have since moved away from pursuing intervention and teaching goals. We are now re-purposing these prototype materials, which have already been demonstrated as engaging and understandable for the target participant group, to test claims prevalent in the autism and robotics literature about whether and how robot behaviour positively impacts children’s interactions.

Complementing these efforts, the DE-ENIGMA team have been developing state-of-the-art technical systems, based on Artificial Intelligence, to automatically detect and reason about the emotional state of the children during the therapy and their level of interest in the activity at hand. The technical system incorporates vocal, facial and body posture cues with deep learning architectures to create robust and generalisable models across the target population. Furthermore, the DE-ENIGMA team will utilise the same technologies to create a novel reporting tool for therapist which will log a wide range of non-verbal behaviour observed during the therapy.

The members of the DE-ENIGMA consortium are extremely experienced and have complementary expertise, allowing optimal user studies from a technological as well as an autism therapeutic perspective. The involvement of stakeholders such as Autism-Europe, Autism Serbia and robotics SME IDMind will allow market and general awareness for future uptake of the project\'s outcomes.

DE-ENIGMA is the first project of its kind that will allow a wide scientific audience access to valuable behavioural data, extend social signal processing capability with this challenging user group and provide a robust test of robot effectiveness in teaching socio-emotional skills to autistic children.

Work performed

The DE-ENIGMA team have annotated the recordings of 50 (25 British and 25 Serbian) participants in terms of continuous emotion stated, measured in valence (positivity / negativity of the emotion) and arousal (intensity of the emotion). The annotation results were then aligned using dynamic time warping for development of the machine learning method.

Online distribution of the dataset has been shown to be impractical due to the large size of the corpus (~7 TB). Instead, data distribution is now facilitated by delivering of physical storage
medium and is stored at more than one partner to help with data storage and distribution.

The following Prototypes have been delivered since the beginning of the DE-ENIGMA project:

• Prototype 0: the first version of the technical set-up was designed to support data collection.

• Prototype 1: the first technical integration was completed and the first interactive game has was developed. It was evaluated with additional children on the autism spectrum to further reduce the risk of triggering noise sensitivities.

• Prototype 2: This is a robot that responded to simple automatically observed situations. We had envisioned a multimodal recognizer, able to interpret and react appropriately to stress, affect, and interest shown by autistic children. However, this has proven a particularly challenging task for the tech partners; we speculate this is due to the challenge of working with children with autism, who by definition display very different types of behaviours and emotions. A tablet was added so that the child had an extra means of communicating with the robot and going through the therapeutic exercises. Some autonomous robot behaviours were integrated based on the messages generated by observed situations.

• Prototype 3 we originally had envisioned to make a final iteration to the previous prototype and test that on a larger scale with autistic children. We now would like to approach that differently: We have identified a need to more fundamentally explore the notion that robots help children with autism \'because they are predictable\'. This assumption is commonplace throughout human robot interaction and autism literature, but to the best of the DE-ENIGMA team’s knowledge, has never been empirically proven. In order to contribute fundamentally to robot interaction with autistic children we want to develop a third prototype, 3A, to carry out an experiment with autistic children at UK locations to test that hypothesis, we believe it will highly impact research if we are successful.

• In order to demonstrate the integrated technical capacity of the robot, we will build a 4th prototype, 3B, where there is multimodal interpretation/reasoning and where we provide teachers with insights the robot has into the children\'s progress through automated detection. We will user test this capability with expert teachers. This prototype will demonstrate the way the robot and the exercise adapt to the children\'s responses and will also provide the teacher with feedback on for instance micro-expressions, arousal, engagement, or performance of the children.

Our Dissemination partners maintain the project website and other related online dissemination. We continuously update the website content with news items, biographies of involved staff, publications and DE-ENIGMA was invited to present its work to European leaders at the Tallinn Digital Summit.

Final results

The DE-ENIGMA dataset has been collected and is now being digitally tagged for features such as:
● facial mapping coordinates (smiles and frowns and other facial expressions can be recognised);
● snippets of speech and vocal noises (the software can judge whether the child or the robot or the therapist is speaking, a log different vocal cues the child can produce, for example if they laughter, cry or shout and estimate the arousal level of the child, for example);
● different body postures and the angle and rotation of the child’s head (the software can estimate whether the child is still paying attention to the robot or not interested anymore)

The DE-ENIGMA team integrated these modules into the prototype used in the usability study.

The DE-ENIGMA team also developed the model for rapport / interest recognition based on the British data. The valence and arousal recogniser was trained and tested on both British and Serbian cultures and achieved similar performance in terms of accuracy.

This dataset will allow the wider scientific community to research the behaviour of children on the autism spectrum to improve current recognition software which will lead to better automatic recognition of physical features in a neurodiverse population. This should eventually lead to improved therapeutic and educational solutions for neurodiverse children.

Website & more info

More info: http://de-enigma.eu/.