Meta’s Photorealistic Avatars Can Be Generated with Just an iPhone
Scanning a photorealistic avatar for virtual reality (VR) once took more than 100 cameras to pull off, but now researchers at Meta have revealed it can be done with just an iPhone.
Facebook first announced “Codec Avatars” in 2019, which are a class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D and therefore virtual reality (VR). The avatars are almost indistinguishable from video and the company is hoping that the lifelike avatars will be important in the future, especially considering its metaverse ambitions.
Meta had been using a specialized rig to generate the Codec Avatars called MUGSY, which contained 171 high-resolution cameras, as seen below. But as reported by Upload, it recently had a breakthrough and has been able to bypass MUGSY by using a scan from a smartphone with a front-facing depth sensor, such as an iPhone that has FaceID.
To capture the Codec Avatar, the phone must be panned across a neutral face, then again while copying a series of 65 facial expressions. The researchers say that the scanning process takes three and a half minutes on average — though users will have to wait six hours for a machine with four high-end GPUs to render the avatar in full detail. It’s likely that if this product is made available to the general public, the rendering would happen on a cloud GPU and not on the user’s device.
The iPhone scanning system utilizes the Universal Prior Model (UPM) “hypernetwork,” a neural network that helps to generate the person-specific Codec Avatar. This UPM hypernetwork is trained by scanning the faces of 255 diverse individuals using an advanced capture rig, similar to MUGSY but with 90 cameras.
Other researchers have already demonstrated avatar generation from a smartphone scan, but Meta claims the quality of its results is state of the art. However, the system is limited to only scanning the head of a person and it’s unable to interpret glasses or long hair.
Codec Avatars Still Some Time Away from Release
Meta Avatars currently have a basic cartoon art style, so this type of high-fidelity still appears to be a long way away. On that note, Meta’s avatars featured on products such as the Oculus Quest 2 have actually decreased in realism over time to better accommodate complex environments in apps like Horizon Worlds.
Back in February, Mark Zuckerberg said in an interview with Lex Fridman that in the future VR users may use an “expressionist” avatar in casual games and a “realistic” avatar in work meetings.
Facebook has declined to say when Codec Avatars may be available to general users but the company is confident it is getting closer.
Image credits: Photos via Authentic Volumetric Avatars from a Photo Scan, Meta Reality Labs
