Report

Teaser, summary, work performed and final results

Periodic Reporting for period 1 - VRTogether (An end-to-end system for the production and delivery of photorealistic social immersive virtual reality experiences)

Teaser

The widespread adoption of smartphones, tablets, laptops and, more recently, head mounted displays, has transformed the classical television or cinema consumer experience, which was essentially social, into an individual experience. Broadband streaming, and new distribution...

Summary

The widespread adoption of smartphones, tablets, laptops and, more recently, head mounted displays, has transformed the classical television or cinema consumer experience, which was essentially social, into an individual experience. Broadband streaming, and new distribution models, have also changed the consumption habits: content is increasingly consumed on-demand, and live events - music, sports, live TV shows - are one of the rare cases where people still gather together to watch TV. The arrival of head-mounted displays (HMDs) to the consumer market is likely to introduce further isolation: people wearing Virtual Reality (VR) goggles can sometimes feel like being in a different place, and do not see or hear their physical surroundings. However, isolation is not a necessary consequence of new media formats: people can feel like being in another place with others.

The grand promise of Virtual Reality is that of a medium which makes you feel like being in a place where you are not, where there is an unfolding plot within which you can take part, and freely navigate within it, as well as interact openly with any element in it, including with virtual characters. In this project, our aim is to radically improve the experience by innovating in how media formats are used (i.e., how audio, video and graphics are captured, delivered and rendered at users’ homes) demonstrating a significant improvement of the feeling of being there together and the photorealistic quality of the content.

VR-Together addresses 5 specific objectives:

OBJ1. Develop and integrate new media formats that deliver high quality photo-realistic content and create a strong feeling of co-presence in coherently integrated experiences.
OBJ2. Adapt the existing production pipeline to capture and encode multiple media formats and integrate them with state-of-the-art post-production tools.
OBJ3. Re-Design the distribution chain so such innovative content format can be orchestrated and delivered in a scalable manner.
OBJ4. Develop appropriate Quality of Experience (QoE) metrics and evaluation methods to quantify the quality of these new social VR experiences.
OBJ5. Maximize the impact of VR-Together can have on content creators, producers, distributors, tooling companies, service providers and the general audience.

Work performed

During the first year of the project, the VR-Together has designed and produced a short VR experience aligned with the objectives of the first pilot with the idea that it will constitute the first chapter of a longer story that will be further developed in future pilots. VR-Together has started to explore the insertion of user representations based in time varying meshes, point clouds and 2D video (2D with removed background, as chroma-key, and 2D+ with color and depth data) and the rendering of different scene typologies where the displayed content results from the rendering of heterogeneous formats (3D, stereo and mono video billboards TVMs and Point Clouds).

At technical level, the work focused on the end-to-end readiness of 3 user distribution pipelines reflecting the 3 user capture formats: time varying meshes (TVM), point clouds (PC) and 2D. This is each of the format was integrated into an investigated a different distribution chain: TVMs are distributed via a message queues (RabbitMQ), PCs are distributed via low latency MPEG-DASH (Dynamic Adaptive Streaming over HTTP) and 2D user representations are distributed via WebRTC. Many challenges had to be overcome in the integration in order to achieve low latency and stable performance in the 3 pipelines. Furthermore the results of the first year give in-depth knowledge of the advantages and disadvantages of the 3 distribution approaches. In addition to that, the project has created a new protocol and set of metrics, and the associated data analysis toolset, for evaluating social VR with end-users. The impact of this result may go beyond the project, since it can become the de-facto standardised manner for evaluating a new genre of experiences: social VR.

Final results

During the first year no significant progress beyond the state of the art has been achieved since efforts have been mainly directed to integrate existing components and try to achieve a first stable version for different content pipelines. Despite of that, VR-Together has generated a relevant impact in different standardisation groups since the project is addressing multiple challenges that are currently on the table of these groups.

Website & more info

More info: http://vrtogether.eu/.