NVIDIA’s Live Streaming Software Can Digitally Fake Eye Contact
NVIDIA Broadcast, the company’s tool for live streaming and video conferencing, can now digitally alter a subject’s eyes in real-time to make it seem as though they are looking directly into the camera.
One issue with shooting live-streamed video at home for professional use — and it even crops up in simple video calls — is that many people are uncomfortable staring directly into the camera instead of either on a script or at other people on screen. While doing so makes the user uncomfortable, not doing so breaks engagement with audiences.
To help solve this problem, NVIDIA has been developing a type of “deepfake” technology that can artificially adjust a subject’s eyes to make them appear as though they are making contact with the camera. Called Eye Contact, it improves audience engagement with a subject by simulating eye contact.
“The new Eye Contact effect moves the eyes of the speaker to simulate eye contact with the camera — achieved by estimating and aligning gaze.,” NVIDIA explains. “The eyes retain their natural color and blinks, and there’s even a disconnect feature in case you look too far away, to transition smoothly between simulated and real eyes.”
While only being introduced into Broadcast this week, the technology has been under development at NVIDIA for the last last year under the name Maxine. Maxine is a set of GPU-accelerated artificial intelligence (AI) SDKs and cloud-native microservices that are designed to enhance audio, video, and augmented reality effects in real-time.
“Eye Contact is ideal for content creators seeking to record themselves while reading their notes or a script, or avoiding having to stare directly at the camera.” NVIDIA continues. “Video conference presenters will look at the eyes of the person they’re speaking with, improving engagement with their audiences.”
The benefits of AI-corrected eye contact were shown in a demonstration video for Maxine early last year:
While it is deploying into NVIDIA Broadcast, Eye Contact is still in beta and NVIDIA says that it is looking to work with the community of users to improve it. Since there are so many different shades of eye colors and lighting combinations, it will need to work on the AI over time to make sure it is reliable across the board.
