Digital image stabilization

[kenneth gruyaert]

One possible use of high framerates that i havent seen mentioned yet is to do what could be called digital image stabilization, it is possible to shoot long exposures in multiple frames and combine them by tracking individual surfaces instead of simply overlaying the entire frames. Since RED is marketing their camera's as DSMC's it would also be great to introduce this feature for stills, I have seen the sensor can run at over 300fps so it would be possible for example to shoot 16 exposures with 1/320th shutter at 320fps for a 4-stop digitally stabilized image with 1/20th second exposure. I know the processor cant handle this at full resolution but the sensor can, so it would depend on wether or not its feasible to send the data to a dram buffer, if so it could easily be big enough for much longer exposures.
A possible use for movies would be to shoot for 24fps 180degree shutter by recording at 48 360 to capture twice the amount of light with the same motion blur.
I would like to know what you guys think about this and if something like this is being done already and also if RED is considering such a feature for the EPIC or future camera's

 

[paulherrin]

not exactly sure i'm catching everything you're saying...

digital image stabilization, as opposed to optical image stabilization is a process of stabilizing shaky footage, but rather than using gyros like lenses or other rigs offer, it is done in post-production by using tracking software and algorithms that scan the image information.

are you talking about frame averaging? that could always be done, much in the same way it's already done.

there's really no way i know of to turn a frame (or multiple frames) that's been induced with blur into a sharp image, the information just isn't there... it's been blurred. the only option would be to paint it manually by hand, which you could do anyways. or just use a tripod.

and 24fps w/180 degrees is the same exposure/amount of light as 48fps w/ 360 degrees... but it would be twice the data/frames...

 

[kenneth gruyaert]

not exactly sure i'm catching everything you're saying...

digital image stabilization, as opposed to optical image stabilization is a process of stabilizing shaky footage, but rather than using gyros like lenses or other rigs offer, it is done in post-production by using tracking software and algorithms that scan the image information.

I remember now i read something about that when looking with google for digital image stabilization so sorry for the confusion it is not what i meant. Though the software analysis would be similar but for a different purpose.

are you talking about frame averaging? that could always be done, much in the same way it's already done.

I think you mean just adding them up? That's what i meant when i said simply overlaying the frames so no that's not it either.

there's really no way i know of to turn a frame (or multiple frames) that's been induced with blur into a sharp image, the information just isn't there... it's been blurred.

That is the whole reason to capture a certain exposure time in multiple frames instead of one: to separate the information so it is not blurred together which means the information actually is there.
Think of it this way: the same scene shot with an object moving across the sensor blurring its image might as well have been shot with the lens tracking the object sending light from the same point on the object continuously to the same point on the sensor resulting in a sharp image which obviously means the data is there. When you are not tracking the object the same information is being captured by the lens and sent to the sensor with the only difference being that light from different moments during the exposure is being sent to different pixels, all you have to do is capture the different moments separately to prevent wrong light being added up together which enables it to be added up correctly digitally either by the camera or in post processing. This is potentially far superior to optical stabilization since you can track multiple objects at different speeds and in different directions simultaneously meaning you can have both the moving object and the background in sharp motion 'focus' instead of having to pick just one of them.

and 24fps w/180 degrees is the same exposure/amount of light as 48fps w/ 360 degrees... but it would be twice the data/frames...

Yes you would have twice the amount of light (data) captured enabling better low light quality equivalent to using a larger aperture or longer shutter but without the reduced depth of field (if a larger aperture is even available) or increased motion blur.

 

[Fabricio Morato]

Hi Kenneth,

I'm also not quite following...

do you want something like still photography "focus stacking" but for motion? or to act like an object motion blur remover?

what do you mean by "the lens tracking the object sending light from the same point on the object continuously to the same point on the sensor resulting in a sharp image"?

 

[kenneth gruyaert]

Hi Kenneth,

I'm also not quite following...

do you want something like still photography "focus stacking" but for motion? or to act like an object motion blur remover?

what do you mean by "the lens tracking the object sending light from the same point on the object continuously to the same point on the sensor resulting in a sharp image"?

Im not familiar with focus stacking but from the name i would think it builds an increased or full depth of field from multiple captures? That would capture other information in multiple frames rather than more of the same information. I think the focusing equivalent of what i am talking about would be capturing the 4D lightfield and focusing after capture in software allowing you to focus on everything simultaneously which would be the ultimate camera in my mind but is unfortunately not something we can hope to get anytime soon. :sad:

By lens tracking i simply meant following the object so it is not moving in the captured video, i was trying to show that the same light you are capturing then is also captured when you do not track the object and pointing out in detail what the difference is between the two scenarios to show that the lens movement is actually in theory irrelevant.

 

[Eki Halkka]

One possible use of high framerates that i havent seen mentioned yet is to do what could be called digital image stabilization, it is possible to shoot long exposures in multiple frames and combine them by tracking individual surfaces instead of simply overlaying the entire frames. Since RED is marketing their camera's as DSMC's it would also be great to introduce this feature for stills, I have seen the sensor can run at over 300fps so it would be possible for example to shoot 16 exposures with 1/320th shutter at 320fps for a 4-stop digitally stabilized image with 1/20th second exposure. I know the processor cant handle this at full resolution but the sensor can, so it would depend on wether or not its feasible to send the data to a dram buffer, if so it could easily be big enough for much longer exposures.
A possible use for movies would be to shoot for 24fps 180degree shutter by recording at 48 360 to capture twice the amount of light with the same motion blur.
I would like to know what you guys think about this and if something like this is being done already and also if RED is considering such a feature for the EPIC or future camera's

This is a good idea. To a lesser degree, it can be done already, simply by shooting high framerates and doing all the processing manually in post. It's also similar to an idea i had a while ago, before Red introduced HDRx. I called my method "Timeslice subframes", and the main application was to get more dynamic range / lower noise by averaging multiple exposures shot at high speed. Here's the beef:

http://reduser.net/forum/showthread.php?24780#7

 

[kenneth gruyaert]

To a lesser degree, it can be done already, simply by shooting high framerates and doing all the processing manually in post.

120fps would be very restrictive though and it would be a lot of work but it would be interesting to see a test of it. Maybe on a cropped sensor at 300fps? I would like to see a comparison between a photo taken in the various ways: one simply at a long exposure time, another captured with many subframes with the same total exposure time for stabilization and a third picture could simply show a single subframe exposure. This would allow a comparison of the low light quality between capturing with subframes and a single long exposure to see if the sensor is capable of capturing an exposure in slices with a negligible loss of quality.

It's also similar to an idea i had a while ago, before Red introduced HDRx. I called my method "Timeslice subframes", and the main application was to get more dynamic range / lower noise by averaging multiple exposures shot at high speed. Here's the beef:

http://reduser.net/forum/showthread.php?24780#7

I had also thought about that possibility to allow capturing more light without clipping, it doesnt capture more light in dark areas compared to a single long exposure though. But since you dont need to capture more light in the bright areas its actually not necessary to split the exposure in many (possibly hundreds) different frames to be able to accumulate all the light in bright areas so it turns out that simply doing a double exposure mix of a long and short exposure like HDRx is enough for hdr, the only thing that probably needs to improve for this is that the sensor should not black out after the short exposure to be able to capture all available light for the low light areas. Ofcource when we eventually have camera's capable of recording at thousands of frames per second this will automatically give enough headroom in the bright areas as a side effect to eliminate the need for a feature like HDRx.

 

[kenneth gruyaert]

to act like an object motion blur remover?

I dont think it should be thought of as a motionblur remover since it doesnt capture the blur to begin with. The real purpose is to be able to capture the amount of light of a long exposure while keeping the negligible or at least much smaller motionblur of a short exposure.

 

[paulherrin]

it sounds like you can already test all that out roughly in photoshop. if it doesn't work there, it's not gonna work. it's really hard to understand what you're trying to explain... check the terminology you're using to describe this.

 

[kenneth gruyaert]

it sounds like you can already test all that out roughly in photoshop. if it doesn't work there, it's not gonna work.

It depends on wether the sensor is capable of capturing the same amount of light split in slices that when accumulated has the same quality as a single exposure. This is equivalent to the question if a sensor is capable of capturing more smaller pixels with the same quality. In other words, with current sensor technology is it possible to make a sensor with 40mp that when the image is downsampled to 10mp produces the same quality per pixel as the same size sensor made with 10mp? Assuming the pixels can cover the same percentage of their area on the sensor. If the answer is yes or at least very close which i think it is then it is also possible to do the same with frames split into multiple exposures.

it's really hard to understand what you're trying to explain... check the terminology you're using to describe this.

I know im not good with words but i did the best i can sorry ... I actually do think i explained it in enough detail to convey the idea.
Which terminology are you unclear about?

 

[Eki Halkka]

it sounds like you can already test all that out roughly in photoshop. if it doesn't work there, it's not gonna work. it's really hard to understand what you're trying to explain... check the terminology you're using to describe this.

I've kind of tested the methods described in that writing of mine, and they indeed seem to work. I didn't have proper footage for the tests though - just regular 25 fps stuff. What i'd need for a true test would be the same scene (a sec or two would be enough) shot at 25 fps and 1/50 shutter, plus 100 fps and 1/100 shutter.

I don't own a Red, and so far every time i've been on a Red shoot we've either been too busy, or i've simply forgotten...