AI Text to Video Generators Are Coming, and They Are Impressive

Artificially intelligent (AI) text-to-image generators have been lighting up the airways so far this year but now a company has teased an AI text-to-video generator.

Video editing software company Runway put out an awesome promotional video where it teases creating any scene that you can think of but as a video.

The video starts with an operator typing in “Import city street.” A street scene is created and then tweaked with the input “Make it look more cinematic.”

Perhaps most impressively, is when a lamppost is removed from the moving scene by using a type of spot healing tool.

The video then switches to “Generate a lush garden.” Once entered, a number of different scenes are spawned for the user’s perusal that includes many different styles.

After choosing a scene, the user can choose an animation option such as scale in overtime, as well as create text to fade in and out.

Next, a video of a photographer appears and “Green screen character” is inputted into the prompt box. This separates the subject from the background, allowing the operator to blur it and then make it black-and-white.

Runway has been a video editor for some time, but this new AI offering appears to be totally novel in the film-cutting space. The software does already have AI features, such as the ability to mask objects in a video.

AI Video Generators Face Difficulties

According to Arts Technica, one of the issues for text-to-video programs is the training set. There are established picture training models for AI text-to-images programs, such as LAION-5B.

As of today, the training models for AI video are primitive and there are also issues because of video’s high computational demands. However, this time last year AI-image generators were in a similar state so it is not a stretch to imagine that AI-video generators may come on leaps and bounds in the coming months.

Runway is a browser-based app and is available now for a monthly fee. But the advertised text-to-video feature is still currently in an early access phase and to test it out users must sign up for the waitlist on Runway’s website.