Fillmmaker Creates Trippy Music Festival Video With Midjourney
A filmmaker employed the artificial intelligence (AI) image generator Midjourney to make a surreal video edit of a music festival.
Although Midjourney is not known for creating video, Rufus Blackwell coupled the generative AI tool with motion tracking techniques and other AI tools to create a unique film.
“While AI art and video are exploding right now, AI video generation is lagging in terms of the quality and consistency achieved in still imagery,” Blackwell tells PetaPixel.
“The intricate nature of generating moving images poses far greater challenges than still life. My goal was to fuse the exceptional image quality of Midjourney with the dynamic nature of videos, leveraging my VFX tool set.”
Creating a Video With Midjourney
Blackwell extracted key frames from the footage he shot of a boutique beach festival in Vietnam.
“These frames were then fed into Midjourney, prompting it to generate an entirely new scene,” he explains.
“For instance, I could provide an aerial wide shot of the festival at night as a reference and use a prompt like the following: “Imagine a futuristic neon-lit alien cityscape, set in a dystopian future, where colossal skyscrapers dominate a post-apocalyptic wasteland.”
Blackwell says the key ability of AI image generators like Midjourney is their ability to “match color grading, lighting conditions, and overall energy of the original scene.”
Once he had the new AI image, Blackwell carefully painted it back into the scene using motion-tracking techniques to ensure proper alignment with the original video.
“Motion tracking captures the movement and parallax of the video sequence, integrating the new scene seamlessly with the correct motion dynamics,” he says.
“The motion tracking can create some weird warpy artifacts, but the results are generally interesting.”
However, it’s not as simple as the above. People in the frame add another layer of complexity as it disrupts the motion tracking.
To remedy this, Blackwell utilized another AI too, Runway ML, to remove the people and create a clean tracking plate.
“Afterwards, the characters can be isolated using Runway ML to create an alpha channel for the character, allowing for their placement against the new background,” he explains.
“It is astonishing how rapidly AI workflows have seamlessly integrated into my creative process.”
Blackwell also used Runway ML to artificially swap the clothes of the people in the video, leading to some groovy outfit changes.
He also used the Vid to Vid tool on Runway ML to achieve a more “motion graphics feel,” by exporting a five-second segment he could apply a photo prompt to which generates a version of the video in a similar style.
“The resulting video could then be easily layered on top using blending modes like Add/Screen,” Blackwell says.
“The trick with these is to use an image prompt that would create a highly stylized output so it overlays adding to the original scene.”
Blackwell want to use AI tools to “elevate and enhance” his final video output. The VFX artist, drone pilot, and adventure filmmaker is based in Saigon.
More of his work can be found on his website, YouTube, Facebook, Vimeo, and Instagram.
Image credits: Images courtesy of Rufus Blackwell.
