matchmoving a 5K 120 fps shot - rolling shutter issues?

[gkocov]

Hi everybody,

I'm doing camera tracking on a Red Epic 5K 120 fps shot and I'm getting some tracking issues that seem to be rolling shutter related. If I solve for rolling shutter in the matchmove application, I get a read/reset time value that is about 8 ms long which is very close to 8.33 ms i.e. the length of a single frame in an 120 fps footage. That basically means that at 120 fps in a 5K image the last line of a certain frame is recorded almost at the same time as the first line of the next frame ending up with significant rolling shutter distortion in the footage. Since I don't have a lot of experience with high frame rate Red footage, I wanted to check with people here if this sounds correct and it's to be expected?

Thanks in advance,
Goran

 

[Craig Parkes]

What was the shutter speed used on the camera? Sounds like they set it to a 360 degree shutter when shooting.

 

[Ken Wilson]

When I track I try to use the highest shutter speed possible to avoid blur and have very sharp images. What shutter speed you use? Not fps.

 

[gkocov]

I'm not exactly sure what the shutter speed was set to since we don't have access to the raw R3D file (we are working with a DPX image sequence made from it, which unfortunately has almost none of the metadata present in the raw file), but I think that the shutter speed doesn't have any effect on the rolling shutter distortion. I haven't checked this on Red, but from my experience with several other cameras that use a CMOS sensor, I'm pretty sure this is correct.

In any case, I did some research this past several days and what I suspected in my original post did turn out to be true. Because the read/reset time for a given resolution is constant, the higher the frame rate (i.e. the shorter the time between two frames) the more pronounced the rolling shutter distortion will be. The read/reset time is one of the main reasons why you can't go over a certain frame rate for a given resolution.

Thanks,
Goran

 

[Björn Benckert]

If you have problem tracking a shot it's usually one of the following...

1. Extreme camera movement.
2. A difficult lens, normally anamorphic's and non rectilinear lenses are difficult to track.
3. You do not have your physical camera setup right, i.e wrong fov or backplane info.
4. The shot has great focus pulls or zooms.

My advices are

1. Do not rely on any kind of auto track, that is for the demo guys to use on those special clips, a normal real life situation clip needs manual tracking with a good volume on the tracked cloud.
2. Make sure to use the correct lens correction, a lot of times it difficult to see on the clip that needs to be tracked then look for other clips shot with the same lens setup and see if you can find straight lines or such to get the setting right.
3. Make sure your back plane is right a lot of time people shoot epic FF 5k and by mistake stretch it to 16:9 wich basically is like putting a stick into your camera solve wheel. Such little aspect fuckup will make even a good track impossible to solve nicely. if shot FF 5k then make sure you got the little letterbox in 16:9 (or cropped sides) but better to have the little letterbox so that you know for sure that you have the full frame to track. And make sure your backplane is set to 27.65mm in width. (the height is 14.58mm) and again make sure you got the right aspect / 1:1 on your pixels.
4. If you done all of the above correct then you should be able to get around even the most deadly focus pulls and still get a usable track... If somebody has been playing with the zoom aswell, well then you really are in a world of pain... look through the tracking apps tutorial, but to make a long story short... it takes some mad skills to be able to 3D track a moving camera that has an unknown zoom going at the same time, I have people here that can do it / have done it many times but honestly it's usually shit hard.

If the above does not help upload the clip and I think if not me than somebody else on this forum can help. We track Epic material day in day out and the rolling shutter has pretty much never been a real issue and at hundred frames you got a 1/4 the shutter speed of real time so even if you shoot 360 deg shutter you still are twice as good as 25p... but again, if you put the camera on a milk shaker or such then yes there is problems. normal hand held stuff in 100 fps is no problem, camera on car with no gyro and bad vibrations can be just as bad as the milkshaker... just saying.

 

[Mike Harrington]

actually with the backplane...especially with syntheyes... calculate the recorded resolution (not rendered) by 5.4 microns for the proper backplane size....in syntheyes it is not so much the aspect ratio as the physical backplane size in order to jive with lens data
so for example 5120 x 2170 would be 27.648mm x 11.718 mm ect

each resolution will have a different backplane size as the sensor gets cropped....

it's not critical if you allow syntheyes to assume the focal length....but if you force syntheyes to use a set focal length then you really have to give it correct backplane size, otherwise it will force the error into the 3d solve.

as far as rolling shutter and shutter speed...you are correct as they have no correlation...shutter speed just minimizes blur

after that I just echo everything Bjorn says....and I have tracked a lot of epic and Red one shots handheld at 24p and the rolling shutter is not much of an issue

 

[gkocov]

Thank you very much for the comments and for the suggestions. I really appreciate that you took time and effort to reply to my posts.

Please don't take this the wrong way, since I'm new to this forum, but it's not the tracking workflow that I needed actual help with. I have worked as a lead matchmove artist on more then 20 blockbusters (some of which are Riddick, White House Down, GI Joe 2 and Dark Shadows), so I'm not really a beginner in matchmoving (Björn feel free to ask Rasmus Pultz about me if he's at Syndicate at the moment).

I absolutely agree with a lot of what you said - manual tracking is absolutely essential for good matchmoves, accurate lens distortion calculation as well (don't ignore higher order lens distortion, it's present in a lot of lenses, especially zooms and wide primes and if you have an anamorphic shot, give 3DEqualizer a try, it has the best lens distortion calculation engine by far), variable focal length calculation is often necessary even for primes when there is a focus pull, etc.

That being said, I'm also absolutely certain that the rolling shutter distortion increases with the frame rate (I explained why in my previous posts). The reason why we can often get away with ignoring rolling shutter in high fps shots is that the relative motion is often small even for a camera that moves and pans fast. But often doesn't mean always. We did quite a lot of tests this past several days, checking both Red and Alexa shots on how much we could improve the accuracy of the track with rolling shutter calculation on and the results were surprising. Even 24 fps shots that had a camera that moves slightly faster were noticeably improved with rolling shutter calculation (the solve error dropped by around 10%-15%). Shots that were filmed between 60 and 120 fps had a solve error drop of around 25%-50%.

The main workflow issue in regards to rolling shutter and VFX is not the matchmove, but 3D and comp. SynthEyes and 3DEqualizer make it possible to calculate the rolling shutter during the matchmove and/or you can use a plugin like The Foundry's Rolling Shutter to remove the rolling shutter from the plates (although this is far from artifact free because of the optical flow issues), but if you want to render the CG elements with rolling shutter or for ex. to use motion vectors for adding rolling shutter to the CG elements in comp, you are basically left with almost no options other then creating your own tools (one notable exception is LightWave which has a renderer that supports rolling shutter).

I apologize for the longer post.

Thanks,
Goran Kocov