SmoothCam Weirdness

[Paris Remillard]

Hey, I was just looking at this SmoothCam clip from a blogwad link on HDforindies.

http://postproduction.digitalmedianet.com/articles/viewarticle.jsp?id=162747

Anyone else notice the weirdness in the edges of the frame? It looks similar to the stretchy stuff in Crossing the Line. Just wondering if this is a common side effect of SmoothCam. Also, if trying to smooth out the airplane shots could be a cause of that strangeness in Crossing.

Sorry, the clip is about halfway down the page on the link.

 

[Joel Kaye]

(Note: the original source material was HDV 1080i50 with CineFrame 25; however, the clip I dropped it onto is 1080p24—23.08 fps—rendered in pjp 75.) It took around 13 hours and 20 minutes (13:20:00) to analyze the clip, and less than 4 minutes to render on a 3GHz Mac Pro Quad with 8GB of RAM.

I see what you're saying. I think smoothcam must stretch the edges as part of how it fills the black space. Can you set it to enlarge instead? In this case there is so little headroom for the guy that might not really work either.

But 13 hours for 1 minute? OMG - I guess steadicam isn't going out of business. I was kinda hoping smoothing handheld 4K would be a great way to speed production and get a lot of dynamic shots fast. It may yet prove to be as there are many smoothing tools out there. And hey, 90 minutes x 13 hours... that's only 40ish days. Render farm time.

 

[mdo]

And hey, 90 minutes x 13 hours... that's only 40ish days.

At 1920x1080.

 

[Mark Pugh]

you have to get out of HDV - convert to prores and times drop dramatically

 

[Anders Holck]

What you are seeing is distortion from the wideangle lens.
The smoorthcam algorith pans in and out of the outer edge, and therefore the lensdistortion moves in and out of the frame.

Smoothcam works most of the time, but there are no settings to tweek if it tracks the wrong object. Also sloppy object edges can be very obvious at times.
But for simple shots it generally works great.

If Apple would just make it analyze the actual used clip, instead of the full mediafile, a lot less time would be wasted.

 

[Greg Voevodsky]

If you are getting motion blur from vibration... smoothcam is not going to remove it since the pixels it is referencing are blurred. I've had shots in high wind that shake the camera... and what you see with the smooth cam is that the camera is smooth but the content will go in and out of focus (actually motion blur). If it is not there (ie blurred) it will appeared blurred in the stabilized shot! Nothing like shooting it right the first time. There are times where this trick will work... like on Apple's demo of a moving and blurred shock on a motorcyle that you expect to be blurred... However, on a locked off shot say, a beach shot with the objects, hit by high wind... it will all go soft when stablized.

 

[Cail Young]

Smoothcam works most of the time, but there are no settings to tweek if it tracks the wrong object. Also sloppy object edges can be very obvious at times.
But for simple shots it generally works great.

Smoothcam doesn't track objects, it averages the entire frame's motion to determine the camera's movement in 2D (i.e. pan/tilt, zoom, rotate, and perspective change). If you can afford Shake, the smoothcam node offers a lot more control.

 

[Anders Holck]

Smoothcam uses motion vector analysis to track objects/features within the frame, (Apple calls it optical flow) and use that information to extract the camera data. While technically it will be at the pixel level.
The algorithm can indeed get fooled by multiple objects, and locking onto that instead of the background/camera motion.

 

[Hans von Sonntag]

Smoothcam doesn't track objects, it averages the entire frame's motion to determine the camera's movement in 2D (i.e. pan/tilt, zoom, rotate, and perspective change). If you can afford Shake, the smoothcam node offers a lot more control.

I second this. Shake is affordable and a great composition tool. Never use HDV for postproduction. Use ProRes HQ which is in most cases a great option. Export only the part of the media which shall be stabilised. Put it into Shake and analyse it there. Use "intersection" and "smooth". In the end put a Resize node and enter the size-values of the used media. eg 1920 x1080 for HD. You'll be able to finish on G5 Quad a 10-15 second shot within an hour.

Smoothcam cannot de-blur motion blur nor is it a replacement of a steadycam system, but it can stabilize helicopter shots or shots from a power boat nicely.

Hans

 

[Cail Young]

The algorithm can indeed get fooled by multiple objects, and locking onto that instead of the background/camera motion.

I should have been clearer. I'm aware that it only tracks a certain number of points but what I was getting at is that the intent is to find global movement not local. If enough points are moving in the same direction counter to camera movement then yes, it will get confused.

 

[GlennChan]

Is it just me or should you generally try the normal Stabilize/1-point motion tracker first? It's much faster.

 

[Gavin Greenwalt]

Normal 1-point tracks usually introduce some high frequency jitter. I recommend at least a 2 point averaged track.

 

[Kiwi Red]

Blurred Pixels

Blurred Pixels

If you are getting motion blur from vibration... smoothcam is not going to remove it since the pixels it is referencing are blurred.

So by shooting at a higher shutter speed you will end up with sharper pixels - or less blur. Then I'd imagine that Smoothcam will do it's work nicely...?

 

[Gavin Greenwalt]

So by shooting at a higher shutter speed you will end up with sharper pixels - or less blur. Then I'd imagine that Smoothcam will do it's work nicely...?

Or you can try using something like the deblur node from Furnace and cross your fingers.

 

[Stokestack]

If you want to focus SmoothCam on particular parts of the image, you can decide what the most reliable moving features are in the scene, track them in Shake with traditional tracking points, and then apply the tracked motion paths to simple dots against a black background and let SmoothCam analyze that. Then pipe the original footage through the SmoothCam.

This'll give you the "smoothed" (not locked-down) look of SmoothCam, as opposed to the regular Stabilize node. It's also more automated than filtering out the high-frequency movement of the tracking points with an expression and smoothing it that way.

Unfortunately, this technique can become tedious when the background is approaching or receding rapidly, and it's the only part of the image with static-enough details to track; there aren't any tracking points that remain viable for very long.

 

[ChristopherKenworthy]

If Apple would just make it analyze the actual used clip, instead of the full mediafile, a lot less time would be wasted.

First time I used SmoothCam I was astonished to find it was analyzing the entire media-file - especially as some of mine were 20 minutes long. Crazy. The workaround was to export the part of the clip I was using, lossless, and re-import, but we shouldn't have to perform that sort of workaround.

I find I get the best results with smoothcam if I'm able to crop out a good 10-15% of the edges. Of course, this reframes your shot and is a resolution problem with most media, but if you're shooting Red 4K and know you have a bumpy shot ahead, you can frame with this in mind, lost those big edges and finish in 2K. Again, it's a workaround, not a place to live.

 

[Stokestack]

First time I used SmoothCam I was astonished to find it was analyzing the entire media-file - especially as some of mine were 20 minutes long. Crazy.

Definitely not convenient and could be improved. However, if you think it through, it's not that surprising for a first release. During analysis, SmoothCam creates a large file containing motion-vector information for the clip. If you wanted to limit this calculation to just the section that was used in your timeline, you'd also want a couple of seconds around that section analyzed for handles. And what if you were cutting in and out of the clip with multiple events throughout your timeline? You'd have a motion-vector file made of discontiguous data for only the sections you've used in your timeline (and their handles).

Then, what if you trimmed your timeline event while analysis was taking place? What if you extended one of the timeline events beyond the handles that were analyzed? That would have to launch another analysis of the newly exposed area and then insertion of the results into the motion-vector file.

All of this would be the proper way to do it, but as you can see, it's a non-trivial programming exercise. The first-stab, thorough solution would clearly be to analyze the whole file at once and provide only for the detection of wholesale invalidations (like replacing the file that was analyzed).

Thus, we're left with the workaround of chopping out the stuff we want to retime from the larger file.

 

[ChristopherKenworthy]

All of this would be the proper way to do it, but as you can see, it's a non-trivial programming exercise. The first-stab, thorough solution would clearly be to analyze the whole file at once and provide only for the detection of wholesale invalidations (like replacing the file that was analyzed).

Thus, we're left with the workaround of chopping out the stuff we want to retime from the larger file.

I agree with all that you say, except that it appears non-trivial because you could write a workaround in two minutes. ANALYZE TIMELINE ONLY + MEDIA HANDLES BUT IF CLIP-LENGTH CHANGES THEN REANALYZE CLIP + MEDIA HANDLES. It's been twenty years since I did any programming, but surely a shell like this could be put around the difficult-to-change-code underneath.

 

[Jeremy Teman]

Smooth Cam clips

Smooth Cam clips

The distortion is most definitely from the use of a wide lens. A lot of the moving shots in the video below were corrected with Smooth Cam. Notice the edges in the shot with the ramps towards the end-- we used a 28mm lens. The shot moving on the road has a few seconds that turned out quite nice considering the original shot, which I believe was a 35mm lens.

A good thing to consider if planning on shooting for a "Smooth Cam" shot is using a higher shutter speed as said earlier.

http://youtube.com/watch?v=KxOgUSBTBs4

 

[Stokestack]

I agree with all that you say, except that it appears non-trivial because you could write a workaround in two minutes. ANALYZE TIMELINE ONLY + MEDIA HANDLES BUT IF CLIP-LENGTH CHANGES THEN REANALYZE CLIP + MEDIA HANDLES. It's been twenty years since I did any programming, but surely a shell like this could be put around the difficult-to-change-code underneath.

Remember that the clip can be used in many events in the timeline, and the vector file must contain sections for each of those events. If any of those events are extended, the relevant section of the vector file must be invalidated, regenerated, and replaced in the file.

Again, it's not incredibly complicated bookkeeping... it's just additional programming and (more importantly, when it comes to release deadlines) testing.