Sony Prototypes a Motion Capture Rig That Controls a 3D Virtual Camera
Sony has showcased a type of collaborative virtual effects system that uses Unreal Engine 5 and a motion capture “camera” rig that instead of using a real camera, it controls a virtual camera that exists in a 3D space.
The project, spotted by Sony Alpha Rumors, takes disciplines that normally work in stages one after the other and combines them into a fully collaborative environment where they instead work in conjunction with each other simultaneously.
At Sony Playstation’s motion capture studio, a team of virtual effects specialists from Pixomondo — who has worked on major past projects like Martin Scorsese’s Academy Award-winning Hugo, HBO’s Game of Thrones and House of the Dragon, and Amazon’s The Boys — were able to direct and capture motion capture footage live on a stage instead of waiting to receive the footage after the fact.
Normally, virtual effects specialists will receive motion capture footage from a team and use that to create final footage, but in this case, the specialists were able to actually work directly with the motion capture actors to create custom scenes live in the virtual space. This was made possible thanks to Sony Playstation’s large motion capture space that uses traditional cameras to build out a whole space virtually.
From there, the Pixomodo team was able to use an unusual shooting system that replaces a real camera with a set of motion capture balls that the computer recognizes as a camera’s position in a 3D space. That information is sent to a monitor on the capture rig and allows the operator to move the camera around that 3D space in the same way that a traditional camera operator navigates reality.
That 3D space is created in Unreal Engine 5, which can accurately recreate entire environments — including people — and the lighting in them. That kind of flexibility gives directors the ability to choose the time of day and the exact action of each item in a set piece and tailor it perfectly to a vision as well as give them absolutely perfect reproducibility.
Combined with the unusual physical camera rig, the director of a production is able to direct a real-life actor to get a performance out of a digital character that would typically be difficult or outright impossible to do if he was relegated to just standing over an artist’s shoulder, which is the traditional method.
The prototype system is best explained in the video from Sony above and the parts that focus on motion capture take place in the first four minutes.
Image credits: Sony
