Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - Sound of Vision (Natural sense of vision through acoustics and haptics)

Teaser

Sound of Vision (vision restoration through sound and haptics) will design, implement and validate an original non-invasive hardware and software system to assist visually impaired persons in understanding the environment and to navigate. The Sound of Vision solution will do...

Summary

Sound of Vision (vision restoration through sound and haptics) will design, implement and validate an original non-invasive hardware and software system to assist visually impaired persons in understanding the environment and to navigate. The Sound of Vision solution will do this by creating and conveying auditory and haptic representations of the surrounding environment. Those representations will be created, updated and delivered to a blind person continuously and in real time.

The overall objective of the project is to develop a system that will help visually impaired persons both in perceiving the environment and in independently moving in indoor or outdoor areas, without the need for predefined tags/sensors located in the surroundings.

The process proposed by Sound of Vision consists of a series of repetitive steps. The first step generates a 3D model of the surrounding environment from the video captured in real time by the Sound of Vision device camera. In the second step, the objects from the 3D model are transformed into 3D sound and haptic sources. In the last step, the 3D sound and haptic sources are combined and conveyed to the user through specialized wearable hardware and algorithms.

During the project, three versions of functional prototypes will be developed, each of which will be extensively tested and validated. Each new version of the prototype will be improved based on the experience with the last prototype – i.e. using test results and feedback provided by visually impaired persons, training specialists and neurologists.

The end result – the Sound of Vision system – will be sold as a hardware and software solution and will include training courses to help visually impaired persons use the system. The consortium has the necessary competencies to design and develop the proposed system, test the prototype, establish training protocols and commercialize the system as an intuitive and accessible final product.

Sound of Vision is a concept that goes beyond the state of the art of visual sensory substitution systems and has the potential to become an affordable commercial product that will actually help visually impaired persons. This system can have an impressive social impact, improving the lifestyle of visually impaired persons as well as their dependence on their families and friends.

Work performed

The first work package (WP1) was dedicated to the preparation of the development of the Sound of Vision system. The first consortium workshop took place in Barcelona in February 2015. At the workshop, the partners discussed and established specific tasks, such as management, inter-communication and reporting tools and procedures. The first draft of the User Requirements Document (URD) was presented to the Consortium (by UPB) and discussed by the partners.

During WP1, the consortium investigated and analysed the latest concepts and results of research relevant to Sound of Vision. A further refinement of the system requirements was performed using the results of questionnaires and interviews with end users. This work resulted in the URD. Beginning with the URD, the consortium established the design goals for the Sound of Vision system, including all the aspects related to performance, dependability, costs, usability and modularity for distributed development. Guided by these, the design work progressed iteratively and produced a high-level guide and blueprint for the implementation of the system, described in the Architectural Design Document (ADD).

A comprehensive cost-performance analysis of existing devices and equipment that could be used to implement the SOV system was carried out during Months 3–4 of the project, with contributions from all partners. The analysis resulted in a list of equipment to be purchased and evaluated by the consortium in work package 2 (WP2).
Dissemination activities were also initiated within the project. The online presence of the project was established, i.e. website and social media (Twitter and Facebook). The studies performed by the consortium in WP1 were described in scientific survey articles, which were submitted to relevant journals and conferences.

WP2 was dedicated to experimenting with and evaluating technical alternatives for the Sound of Vision system. Teams consisting of consortium members with relevant expertise were formed, focusing on the following technical fields related to the development of the SOV system: 3D image acquisition, 3D image processing, audio models, specialized prototyped headphones, haptics and virtual training/testing environments. Furthermore, the following preliminary tests and experiments with end users were performed: headphone testing and comparison, sound model testing, outdoor mobility study, EEG and physiological signal studies.

As the project pays exceptional attention to user training aspects, the design concept of the virtual training environment was also defined to a good level of detail during WP2.
Standard testing procedures for testing and evaluating the Sound of Vision prototypes were devised in WP2. These procedures included the paths and places required for the tests. The development of the standard testing procedures was an important goal of WP2, as it was crucial to the processing and interpretation of the test results of the Sound of Vision system.
The blueprint for the implementation of the Sound of Vision solution was designed and written in a Detailed Design Document (DDD). The purpose of the DDD is to guide the prototyping work in work packages WP3–WP5. The DDD was based on the URD, ADD and the results of all the WP2 experiments. However, the DDD can be modified and improved as the project advances and as the partners learn more from testing and using the prototypes.

The dissemination of concepts and results follows a general communication plan, which was established in WP1. The partners have submitted and published many conference and journal papers. All the papers are listed on the project\'s website. Furthermore, the online social media has also been updated regularly with news about the partners\' activities within the project.
The main focus in the 3rd work package (WP3) was the development of the first prototype of the Sound of Vision system. Considering the complexity of the project and its highly innovative nature, an ag

Final results

The huge number of persons affected by visual impairment has been impelling researchers worldwide to develop information systems to assist these people in accomplishing different activities. Attempts to replace vision using the haptic and auditory senses indicate the significance and potential of these approaches. There are systems that help visually impaired persons use computers and public transportation or to navigate in known environments (for example, museums or university areas covered with sensors or tags). However, very few systems provide a high degree of independence, allowing visually impaired persons to significantly achieve mobility and integrate in the active life. In the EU and throughout the world, many researchers believe that it is actually possible to develop such solutions, backed by technological advances in computing, sensors and neurosciences. Still, the difficulty of the task, which requires much advanced niche expertise, as well as the lack of synergies, has prevented the creation of powerful enough, easy-to-use and affordable solutions.

The overall concept and design of the Sound of Vision system are tailored to overcome the limitations of previous approaches. The development of the Sound of Vision system is based on the following key concepts, which render it an overall progress beyond the state of the art in the design of assistive technologies for the blind:
- The Multi- and inter-disciplinary approach allows us to incorporate the latest technological advances in computing, sensors and neuro and behavioural science.
- Emphasis on heavily exploiting end users\' feedback: The end users are directly involved in the system design and testing phases. Their involvement is achieved through direct participation, as consortium members, of two organizations for the visually impaired, caretakers of visually impaired people in the teams of several partners, as well as large volunteer participation. This has allowed the consortium to carry out a large survey with visually impaired persons in four European countries. The conclusions of this study have been incorporated in the specifications of user requirements (D1.1), the system design (D1.3 and D2.9) as well as the definition of the testing and training procedures (D2.6, D2.8). Moreover, all these specifications have been devised and reviewed with the participation of end users and caretaker members of the consortium. Testing with end users is extensively exploited to improve the design and implementation of the incremental Sound of Vision prototypes. Testing scenarios are carefully designed by neuro and behavioural scientists, together with technical specialists, to fully exploit user feedback.
- Rich data acquisition: The Sound of Vision system incorporates multiple environment sensing technologies (stereo, structured light, inertial) to compute a full 3D reconstruction of the environment in many possible situations (indoor, outdoor, various lighting conditions, etc.).
- Rich, natural-like perception: The Sound of Vision system provides a continuous real-time reconstruction and rendering of the environment. The use of two output modalities, audio and haptic, allows the system to provide a natural perception of the environment, similar to the real visual sense. New sonification and haptic encoding models to deliver the environment information are being developed and tested.
- Focus on training: Sound of Vision regards training as the unique key for users to unlock the full potential of the device and achieve full proficiency. Gamified virtual training is the most important component, providing high availability, cost effectiveness, motivation and safety for our end users. The training mode of the device will provide a collection of serious games based on virtual environments of gradually increasing complexity. The design of these serious games was a highly innovative and creative activity, and its results – described in detail in D2.5 – represent a clea

Website & more info

More info: http://www.soundofvision.net.