Runway’s New Model Can Transform Existing Video Footage Using Just a Prompt

Runway has launched a new AI video model that can manipulate existing footage with text prompts — rather than generative synthetic videos from nothing.

The impressive hype video below that Runway put out for its new model — which is called Aleph — may pique the interest of some filmmakers. For example, a shot of someone standing by a window can be iterated on by typing “Show me a different angle of this video” as a prompt, and the AI model will use the existing video as the basis of another shot of the same scene.

It is unlikely that this will work perfectly every time, but if it can do anything like the introductory video, then it will certainly have its use.

As well as generating new camera angles, Runway Aleph can also generate the next shot in a sequence from a source video. There is also a ‘Style Transfer’ which can transform any video with a desired aesthetic, according to Runway.

The weather, location, season, and time of day can also be altered. For example, a video clip of a park during a beautiful sunny day can be transformed so that it is raining instead.

Elements, like fireworks, trees, extra characters, and crowds can be added to existing footage. “Simply describe what you want or provide a reference image, and watch it appear naturally in your video with proper lighting, shadows, and perspective,” Runway writes in a blog post.

Aleph can also remove or change objects in a scene. That includes people or people’s reflections. Objects can be transformed, too. Runway gives the example of a video clip of a car driving down the road, changed so it is a horse and carriage instead.

If you want a person’s appearance changed so they are older or younger, that can also be done on Aleph. And if footage was shot during the wrong time of day for a project, that can be altered too by turning night into day or vice versa.

Runway has made no secret of its hopes to disrupt Hollywood with its products and it is easy to see filmmakers might be tempted to use Aleph rather than reshoot a scene that might contain a mistake. Aleph is available now to Runway Enterprise accounts.