Google launched 3D holographic calling black technology, thousands of miles can be “face to face” chat with friends and relatives.

At Google I/O last May, Google CEO Sundar Pichai unveiled Project Starline, a 3D holographic video calling project that could be implemented without needing a headset.

Talking to a “real” person via Project Starline

Project Starline is one of Google’s highly secretive internal experimental projects. According to Google’s researchers, the project team started to do related technical research six years ago.

Project Starline recently invited media and enterprise partners, including The Verge, to the lab for a hands-on experience.

The Verge editor Jay Peters exclaimed after the experience, which was the most realistic virtual reality interaction he had ever experienced, that it “felt like science fiction come to life.”

Human communication has been constantly upgraded in line with the development of information technology. From the ancient telegraph to the telephone and then to video calls, human information transmission has been upgraded, and the timeliness and content of information transmission have also been improved.

In many science fiction movies, science fiction creators believe that hologram calls are the next generation of communication that meets this definition.

In other words, after humans have succeeded in bringing familiar faces around, the next step is to bring the “person” to the scene.

Thus, Project Starline was born.

In the press interview, Andrew Nartker, director of product management for Project Starline, said that their key breakthroughs in communication technology would allow users to become closer and more connected to people worldwide.

Jay Peters says Project Starline is like a magic window that connects the two rooms. The Google researcher sitting across from him looks almost as tall and good-looking as a real person. When he hands over an apple, he can even feel the spatial distance through the shadows on the screen that resemble the real world.

It was like a face-to-face meeting at the same table, and the level of realism almost made people forget that the two test rooms were far apart.

The secret of 3D video – How Project Starline works?

Sitting in front of Project Starline, nothing is more striking than the large 65-inch screen, which is not your ordinary home TV. Still, an 8K/60Hz 3D light field display has twice the pixel density of VR devices like the Oculus Quest for a more detailed stereoscopic display.

This light field display must capture information about the object’s depth to show a stereoscopic effect. The researchers placed multiple binocular stereos and infrared cameras above and below the screen so that the depth information from multiple angles could be combined to restore a sense of spatial distance to the figure.

Multiple monochrome tracking cameras were installed on the sides and top of the screen for real-time 3D positioning of the character’s facial features so that the character can be accurately modeled as it moves.

When this captured position and picture information is transmitted to the receiving end, the system renders three depth streams based on the position of the viewer’s eyes, mixed with the grain stream to render a realistic “virtual person.” (3D video)

When the viewer’s eye position changes, the position of the image will also follow the eye movement to make a subtle position deflection, a spatial realism that traditional 2D images cannot provide.

In addition to breaking the “next dimensional wall” visually, Project Starline has also put a lot of effort into the auditory aspect.

On the input side, the microphone can work with the face-tracking data to form a dynamic beam with directionality for noise reduction and anti-interference. On the output side, the tracking system locates the audience’s ears. It simulates the spatial audio, making the voice sound as if it were coming from the mouth on the screen.

Imagine when you want to hold an important video conference with colleagues and partners; participants no longer use computer vision, eyes no longer lost outside the camera, but can make natural eye contact with you, even use some body language to assist expression, such a participant experience will be a qualitative leap.

But at this stage, Project Starline still has a long way to go before it can be used in practice.

• Lauren Goode, the editor of Wired, said after experiencing Project Starline that while the visual effects of the holographic video were great most of the time when the characters moved too much, the original three-dimensional image would briefly turn back to flat, making the experience less than perfect.
• Google researchers also mentioned that when dealing with complex shapes (such as fluffy hair and optical glasses) or moving objects, there are often false “artifacts.” Their team is currently working to improve the accuracy of image processing.
• In addition to technical obstacles, the cost of deploying a complete set of Project Starline is quite high.

In an interview with The Verge, Andrew Nartkerct said frankly that Project Starline is an exciting new technology and a groundbreaking exploration of space interaction. However, it is still hardly a “product” at this point.

The sky is the limit when there is a friend in the sea.

So, what is Project Starline trying to solve with all this effort? The answer from the project team is simple: to make you feel like you’re with a real person.

In the past year, teleconferencing software, represented by Zoom, has been widely used in businesses and schools, where people are separated for various reasons and have to work together through video software. Thus the human connection is diluted.

In a report on the impact of video conferencing on productivity, Microsoft mentioned that people could be more easily distracted in video conferences than in real-world communication, a natural coping response to high-stress video conferencing.

Project Starline, which provides a real-life conversation experience, seems to eliminate this sense of stress that gets in the way of communication.

Although Project Starline is filled with all kinds of camera equipment, when it’s working, all this cold technology is left behind by the participant sitting in front of it. You can communicate with a near-real “person.”

Google researchers found that participants using Project Starline had more eye contact and body language than traditional video communication and could recall 28% more content at the end of the communication. This is the beauty of natural communication.

For now, Project Starline is only being tested on a small scale within Google for employees. Google says it will be tested with enterprise partners, including Salesforce, WeWork, and T-Mobile, to drive deeper connections between people starting this year.

Will Google’s Project Starline become popular in the future?

It’s hard to answer this question for such a technologically advanced project definitively. 

Project Starline may succeed as a real product, or high costs may fail it, or it may even become part of Google’s future AR and VR technologies.

But no matter what, this future technology that echoes people’s real needs deserves more attention from the technology circle.

Overall, people are used to the separation, and nothing is more precious than seeing in person face to face.