Delivering uninterrupted 3D immersive content, such as sports games and live concerts, to many viewers is a highly ambitious goal. It involves sending very large volumes of high-quality multimedia data to a wide range of user devices. The difficulties are multiplied when viewers are both fixed and mobile, as well as watching content on different types of devices. The challenge is also exacerbated when users wish to have a shared experience while watching the same content remotely at the same time.
The three-year ROMEO Integrating Project (Remote Collaborative Real-Time Multimedia Experience over the Future Internet) came to an end last year. The project, which was co-funded by the European Commission's ICT FP7 Networked Media research program, has addressed these challenges through compression and hybrid synchronized 3D multi-view media delivery. With high-bandwidth services of the future in mind, the project has developed a fully functional and demonstrable platform for 3D multi-view video and associated spatial audio. This uses content-aware delivery and intelligent adaptation methods to provide an immersive experience for remote users, including those on the move. This has been achieved by combining Digital Video Broadcast - Second Generation Terrestrial technology (DVB-T2) with a content-aware peer-to-peer distribution system that operates over wired and wireless broadband links, including Wi-Fi and LTE. The platform has also enabled multiple users to access and share media content with adaptable service quality, as well as experience and talk about the media they are seeing and hearing at the same time. All of these features have collectively contributed to realizing an enhanced collaborative and immersive 3D media delivery and consumption platform - the ROMEO platform.
The immersive viewing experience has been guaranteed in such a way that the main 3D stereoscopic content (e.g., one viewing angle) is delivered via DVB-T2 to all user terminals, while simultaneously streaming the other viewpoints using the peer-to-peer distribution system. The received multi-stream content is then appropriately synchronized to provide a seamless multi-view presentation. The peer-to-peer delivery has been ensured thanks to scalable and multiple-description coding of the content against problems like network congestion, as well as through the use of ROMEO's in-house-designed multicast multi-tree structure for distributing additional views with enhanced robustness.
In addition to the visual immersion provided by 3D multi-view content, the perceived high-quality spatial audio has proved to be a key component in building up the overall immersive 3D media experience. A real-time audiovisual communication link has also been deployed in the ROMEO platform, which enables users to become both creators and consumers of 3D multi-view video and spatial audio, enjoyed remotely yet jointly.
Maintaining the spatial alignment of the audio and video scene throughout the 3D media experience is essential. It has also been demonstrated that both audiovisual components of the media should be aligned accurately with their respective views from different angles, while at the same time those views formed by different streams of audiovisual media received from hybrid delivery means (i.e., from DVB-T2 and peer-to-peer transmission) should also be synchronized. Equally, all users accessing the 3D media need to be synchronized with one another for the joint experience.
The ROMEO platform guarantees the stereoscopic 3D media service at all times over the DVB-T2 network. It also deploys scalable coding and a content-aware, network-adaptive peer-to-peer media distribution scheme over the internet. This ensures an uninterrupted media streaming service to connected viewers in synchronicity with the DVB-T2 broadcast, even under unfavorable network conditions (e.g., network congestion and temporary packet losses). Since the compressed media representation has been made scalable (several layers of the compressed 3D media can be discarded in a controlled manner), re-buffering the stream after temporary drops in the network state is no longer required. This is of particular benefit to mobile users with limited resources. In addition, the platform ensures seamless service continuity for mobile and portable users with dynamically changing network coverage status (e.g., relay over to LTE when the user moves outside the coverage of a Wi-Fi hotspot without interruption to services).
A visual attention model is also used to identify the perceptually important visual regions in the 3D video and deliver them with higher visual quality under poor network conditions, while a quality-of-experience model is used to perform perceptually optimized bandwidth adaptations, which is of particular importance for serving resource-limited mobile users.
Finally, the ROMEO platform has supported virtualization of home user equipment. This eliminates the need for set-up work to be carried out in users' homes, which reduces costs for network operators and makes life much easier for lay users.
Find out more about the ROMEO project consortium here.
