PetaPixel

Game Changing Algorithm Turns First Person Videos Into Incredibly Smooth Hyperlapses

Researchers at Microsoft just changed the POV video game — there’s no doubt about that. Using a newly developed algorithm, they transform long, boring first person videos shot with helmet cams into super smooth hyperlapses that look like they were shot with a steadicam.

The quality of the results really defy description. What begins as a long and tedious first person video of, say, someone rock climbing, turns into a steadicam-smooth roller coaster ride up a mountain… and it’s all software based.

Here is a technical explanation of what the algorithm is actually doing in order to obtain these results:

Basically, the software uses the video to recreate the camera’s path on a 3D depth map, picking a smooth path through that map, and reconstructing what the camera on this path would see using multiple frames to reconstruct each frame.

Doing this, they’re able to get much smoother results than standard stabilization techniques that deal with a single frame at a time. A marginally smooth, nauseating video turns into something not just watchable but downright impressive.

They even remove any sections where the camera stays still for too long, so the movement of the camera can stay smooth and continuous.

Screen Shot 2014-08-11 at 8.52.49 AM

The final results aren’t always perfect, you can see some issues that arise from combining several frames that don’t always match as well as they should, fixing for distortion and white balance. But it’s leaps and bounds beyond what the ‘naive’ time-lapse that uses every 10th frame can do.

If you haven’t already, definitely watch the video at the top to see the technique in action and hear a brief explanation, check out the second video if you want to get into the technical details, and if you’re really intrigued, you can read the full paper by clicking here.

And get excited, because while no specific timeline has been laid out, creators Johannes Kopf, Michael Cohen and Richard Szeliski say they are working hard to put the tech into a Windows app.


Thanks for sending in the tip, Lloyd!


 
  • admiralofbirds

    I know I’m not the only one who finds the result of this technique unwatchable for more than a few seconds. While the motion is somewhat smoothed, there aren’t enough frames for the algorithm to work with to smoothly map them to the calculated 3D space, resulting in many visible areas where we see the new images being “painted” in. This is most evident in the mountain-climbing bits.
    I don’t understand why the smoothing technique was applied before the frame rate change. It should be the other way around, it seems to me.

  • Savatar

    Everyone’s a critic!
    Your options are:
    1. Go ahead and do a better job
    2. Watch the 10x time lapse
    3. Stop complaining

  • XTheHacker2000X

    (i think) Those tests were shoot in 1080p resolution in 24fps and the algorithm only looked at one frame every ten, it’s just a convenience thing because ideally you want to shoot the best resolution at the best framerate, look at every picture and not one every ten, then it looks nice, but you need an enormous processing power, it would take days to complete on a normal machine.

    With the GoPro Hero 4 we will probably be able to shoot 1080p 120fps, at that point it you took the time to let the algorithm look at every frame (72000 frames in 10 minutes of video), then it would be perfect

    Obviously you are stretching images so it wuold be even better shooting at 2.7k or better and render in 1080

  • slyman

    still 3fps of the original at 10x, i’d rather see like 100x

  • Justin Meade

    I like that the first comment was a technical response. If there aren’t enough frames, then I wonder what recording at 60 or 120 FPS would do for this. It would give them a higher sample set to draw from, and the frame rate change would obviously need to be adjusted for the same speed output. Also processing time would jack up as a multiplier.
    As for the smoothing, they dont actually do a frame rate change, so I’m not sure what you’re getting at. the smoothing is done on the calculated camera path.

  • Tobias W.

    Is anybody else curious how your average, pov porn clip would look like using this technology? (If I made you laugh, donate a dollar to the Red Cross please. Thanks.)

  • loder74

    Very impressive.

  • http://simeonpilgrim.com/ Simeon Pilgrim

    Wow, that was super impressive. I would be super keen to have that to summarize long trips, without the crazy shakes.

  • kassim

    Just like in the movie. Well done Microsoft!

  • Cinekpol

    OMG where do I throw my money?!

  • Cinekpol

    It’s made by Microsoft, so some people feel obligated to complain. They would complain even if Microsoft would invent a cure for AIDS.

  • Bolkey

    Don’t take the first cure. At least wait for SP3 when most bugs have been fixed..

  • mzungu

    Wow…Cool, but hate to think what kind of computing power this will need… Definitely not on my PC, that;s for sure.

  • kassim

    For a cure for AIDS, people will kill for the beta version.

  • kassim

    Homemade porn? Because it would not makes any difference in studio-shot footage.

  • Chang

    Time for an upgrade :-)

  • Pagan_God

    This would make for a perfect demo for their Azure cloud computing solution. Prep the file on your PC, upload, come back later and download.