"HDR MODE" - Possible on Red?

[Andrew M.]

We did few experiments with two cameras side by side, locked in sync.
We posted some early results of this experiment here on this forum.
Well, for some shots in wide open spaces the 3D effect was negligible.
For anything filmed from less than 150’ the spatial shift was not acceptable.
We couldn’t combine two frames with enough of edge accuracy.
One camera was running +2 stops the other -2 stops.

I am still thinking that two subsequent frames with the different shutter speed but double the frame rate for some less dynamic shots would be acceptable and strobeing effect would be negligible.

 

[Joe G.]

"I am still thinking that two subsequent frames with the different shutter speed but double the frame rate for some less dynamic shots would be acceptable and strobeing effect would be negligible."

That was my original pitch. The first frame takes the entire 360 degrees, and the next frame takes the continuation of time, for another 90 degrees or so. The motion is continuous across the two frames with no jump.

In a lot of situations that would be perfectly fine.

 

[David Mullen ASC]

I didn't understand what you were describing, but anytime you combine two sequential shots into one shot, then a moving object occupies two different positions in each element you are combining -- there's no way to say that the motion would be continuous since it basically can't be -- they are separate exposures, which is why you are getting the HDR effect. Saying that with minimal motion, you can get away with it ignores the fact that movies have all sorts of motions in it. You can think your actor is hardly moving but then they pick up a plate quickly or suddenly stand-up.

The only way it's going to work for motion pictures, other than static shots of static subjects, is a technique that involves simultaneous exposures, not sequential exposures.

 

[Eki Halkka]

I didn't understand what you were describing, but anytime you combine two sequential shots into one shot, then a moving object occupies two different positions in each element you are combining -- there's no way to say that the motion would be continuous since it basically can't be

You forget that digital cameras can have a 360 degree shutter. The motion continues without breaks (read-reset may cause a slight break, but that should be unnoticeable).

I actually wrote down a pretty detailed proposition of the technique last year - i posted it somewhere here, but can't find the thread right now, so here's my essay again:



Timeslice subframes: SNR / DR improvement with firmware and software modification for temporal supersampling

By Eki Halkka, 17. Nov. 2008 (Edited 20. Nov, 23. Nov)




Intro

The idea is to use "exposure timeslice subframes" to store more information for each frame, using temporal supersampling - a bit like adding more cores to a processor instead of making it clock faster, if you may.

Dynamic range can already be expanded with temporal supersampling using current tools - by shooting at higher frame rate and combining the frames in post. Simply averaging two frames together reduces noise in half, or in other words adds 6dB to signal-noise ratio (SNR). Adding the frames together adds one stop of dynamic range (DR).

Unfortunately, this is a rather complicated procedure in practice right now.

What i suggest, is to make temporal supersampling an integrated part of RED work flow. Ideally, this should be done with the raw linear sensor data, before debayer.




Example case

For the sake of simplicity, i'll use a target of 25 fps and 1/50 shutter (nice round numbers) on the following example of a practical implementation:

The camera sensor is set to run at 100 fps, 1/100 (360 degree) shutter speed. The exposure is set properly for this, not the targeted 25 fps, 1/50 shutter speed, thus preventing overexposure by one stop.

Out of these originally captured 100 fps frames, or exposure "timeslices" as i like to call them, each group of four frames would be treated as “subframes” of a final 25 fps frame.

Captured frames 1 and 2 of each group would be saved, but with a flag that describes them as partial frame 1a and 1b. Captured frames 3 and 4 would be discarded.

The result would be a 50 fps stream, but the extra frame rate would be used to store twice more of intra-frame precision, not more temporal information.

If possible, the unused subframes (3 and 4) would actually never be captured at all – the sensor would have a bit of time to cool down, maybe it could help to implement this at higher target frame rates.

Supersampled subframes give a lot of new flexibility in post production:

A) If subframes 1a and 1b are averaged together, the result is an 1/50 shutter speed frame which has 6dB lower noise, but the same dynamic range.

B) If subframes 1a and 1b are added together, we end up with an 1/50 shutter speed frame that has the same noise level, but one stop higher dynamic range – it's essentially a 13 bit accumulation of the original two 12 bit images, reaching sensor saturation one stop later. This will naturally require the post production software to be able to hold values that “go over 100%”, floating point style.

C) Any blend of the above is also possible - they're two sides of the same phenomenon.

D) One bonus possibility is to simply discard one of the two sub frames - user could modify shutter speed in post between 1/50 and 1/100 - at the cost of losing the improved SNR / DR.

E) If all four captured 100 fps timeslices were saved as subframes instead of just the two in the above example, user could have also 1/33 and 1/25 shutter speeds at his/her disposal, trading in more motion blur and capture storage space for additional DR and SNR in the resulting image, making the blur-noise trade-in decision in post rather than on the field.

F) Timeslice subframes might also open up other interesting possibilities, i.e. isolating moving areas of the frame by difference matting etc.

G) As a bonus, this method would work as an "electronic ND filter", the aperture could be opened by a stop without overexposing.

Summing it up

+ At least 6dB less noise (12 dB with four subframes)
+ At least one f-stop better dynamic range at original noise level (two f-stops with four subframes)
+ Ability to control shutter speed in post (i.e. 1/100, 1/50, 1/33, 1/25)
+ Capture proper motion blur for video and faster shutter speed for stills simultaneously.
+ Can also be used as an "electronic ND filter"
+ Should help to minimize compression artifacts by averaging them over frames
+ Should be doable with current hardware, with minor changes in firm- and software.
+ The technique scales up as technology evolves

- With Red One, the technique probably only works in windowed modes
- The technique only works at relatively slow target frame rates
- The shutter speed variation is limited
- The storage requirements increase
- The post processing burden increases
- The technique improves the dynamic range "from the top up" by preventing overexposure - there's no low light sensitivity gain.​

Doable with current technology

Red one can shoot 2K up to 120 fps, which is enough for the above 6dB / one f-stop improvement with frame rates up to 30 fps. Depending on what the frame rate bottleneck is (sensor, the electronics, redcode, other..), it might be possible to use three or more subframes instead of two in a "burst mode", for an additional 3dB / half stop improvement, especially as the sensor has time to cool down between bursts.

Using this method would require adding a new timeslice/subframe recording mode to the camera's firmware, with the flags i described above. As there's probably not enough processing power in the camera to do proper frame blending etc., the real time preview should be modified to account for this method - possibly simply by adjusting display gain if needed.

The post production software would also need to be modified to read the flags and combine frames from the new recording mode, in the ways i described above. If individual frames cannot be flagged in redcode, it would probably be enough to simply to have "footage uses two subframes" flag in the clip's meta data.

But that's it - that's all that's needed to make this work, as far as i can see.




But wait, there's more

In the future cameras, faster sub frame rates and even bigger improvements could be obtained. If RED ever comes up with a 1000 fps high speed camera, the high temporal precision could be used to store many, many additional stops as timeslice subframes - the camera would in fact classify as a true high dynamic range motion camera.

Another highly desirable option would be to use time slices with varying shutter speeds - the same way as HDRI is currently captured with still cameras. I assume this would require much bigger modifications though - but the potential benefits would also be big. In addition to capturing HDRI, the same footage could be used for video with a (relatively) slow shutter speed partial frame and for stills with (relatively) high shutter speed partial frame - the trade off is differences in exposure, (still frame would be underexposed), but the benefit of usable, sharp stills of moving subjects might be worth it.

 

[David Mullen ASC]

I'm not sure why adding up frames all shot at the same exposure would expand the dynamic range captured. The RAW file of each frame in this case would have the same information in it, so how does adding it up increase that range? Unless all you really mean is that doing this allows you to underexpose one-stop but then reduce the noise in the shadows by adding the exposure from the other frames.

Still, all of this for one more stop of DR, at the cost of losing a stop of speed? So your 320 ASA Red now is a 160 ASA camera effectively?

I can understand the notion of two shots with no shutter lapse being treated as a continuous exposure, but this precludes the notion of shooting those two shots at different exposures, which some people are suggesting, because as the object moved from A to B in that combined frame, it would get darker or brighter halfway there

Also, I don't think many people want to drop to the lower resolution of 2K RAW on the Red unless absolutely necessary. And sticking to 4K at 100 fps is quite a lot of data to manage in post.

 

[Eki Halkka]

I'm not sure why adding up frames all shot at the same exposure would expand the dynamic range captured.

The sensor is just a counter, keeping track of how many photons hit any given photosite on the sensor. It's like a bucket that starts to fill, and at some point it's full - the sensor is fully saturated, clipping to white.

A dark part of an example image may have the bucket 10% full, while the bright part is 90% full, and highlights clip to pure white at 100% (they would have gone to 150%):

..........
..........................................................................................
....................................................................................................|clipping


If you take two frames instead, using half the shutter speed, you end up with two 5% buckets in the dark parts, two 45% buckets in the bright part and two 75% buckets in the highlights:

.....
.....
.............................................
.............................................
...........................................................................________________|no clipping
...........................................................................________________|no clipping

When you add these two frames together, you get the same 10% in shadows, the same 90% in the bright parts, but the highlights don't clip at 100%, they go to 150% (the clipping would happen at 200%):

..........
..........................................................................................
......................................................................................................................................................____________________________|no clipping

In other words, you have one extra stop of dynamic range.

Still, all of this for one more stop of DR, at the cost of losing a stop of speed? So your 320 ASA Red now is a 160 ASA camera effectively?

No. It would still be a 320 ASA camera, but you would have the highlights clipping one stop later.

The combined shutter speed for i.e. 25 fps target would be the normal 180 degree 1/50 (1/48 for 24 fps). It would just be added together from two 1/100 (1/96) shutter speed frames.

Even if dynamic range is not a problem, an added benefit would be less noise, in situations that have enough light: as long as you can get proper exposure at 1/100, you would have two properly exposed frames instead of one. Averaging these two together also averages the noise to half - this could be very useful i.e. for greenscreen shoots, especially in tungsten light.

Also, I don't think many people want to drop to the lower resolution of 2K RAW on the Red unless absolutely necessary. And sticking to 4K at 100 fps is quite a lot of data to manage in post.

In my suggestion, two frames from each group of four would be discarded, resulting in 50 fps (48 fps) stream. It's twice the data rate, but there are cases where that extra stop of DR would come to critical use and that would be acceptable.

And of course, the new cameras are very soon available, with the ability to do higher data rates at full resolution.

The method i suggest can be put to use with the current camera, simply by shooting at four times the target speed, 360 degree shutter, and combining the images in post. It requires a few extra post steps but it seemed to work fine when i (briefly) tested it.

 

[Noah Yuan-Vogel]

I posted a thread with a similar idea a while ago. So I agree that this would be a great feature to have or at least for RED to test in terms of increasing dynamic range/reducing noise and easily allowing variable framerates and shutter speeds in post. I think perhaps a 3k native camera would be better suited for this, and it might also be worth implementing a lower bit-rate codec to make this option more attractive.

Another idea that may (or may not) be more feasible than creating HDR from two frames that do not match temporally or from an additional chip installed with a beamsplitter might be to just replace the Bayer and olpf filters. Seems possible that one could, make a new filter pattern that would put a green and a green+nd filter next to each other and red and red+nd filter next to each other, etc. or next to each block of grgb (or whatever it is) put a grgb with ND filters over them. in either case I'm sure both would require a stronger OLPF to avoid increased aliasing. These pixels could be combined in camera (no debayer, just simple adding of pixels, but higher bit depth codec might be required) or done in post. The benefit of doing it in camera is that you might be able to get away with combining two 12bit pixels and recording to 16bit values, which would save space, since there would be half as many pixels to store and only 33% more bit depth needed to store the HDR info adequately. Also, itd probably be much easier on the codec to store half the pixels at higher bit depth than a full-raster image with a checker pattern all over it. Of course the light and dark frames could be stored separately in the codec as well and combined during decoding. I'd gladly sacrifice some sharpness and half of the 4k sensor's pixel information to add 4stops of dynamic range. But then I'm also someone who doesnt feel that strong a need for shooting 12mp video as 6mp is more than plenty for me.

Anyway just came up with that idea 5min ago so any thoughts?

 

[Eki Halkka]

Anyway just came up with that idea 5min ago so any thoughts?

I recall reading that Panavision had something a bit like that under developement a few years ago. To me it sounds like a good idea, i see no reason why it wouldn't work.

If the two pixels were read at 12 bits, and combined in-camera, it would only require 13 bits to save all the data (each new bit doubles storable info). Oh, BTW, i don't know for sure, but i've always assumed that also the raw sensor data is actually saved to file for each color separately (as a big green chunk, blue chunk, red chunk), not as a checkerboard (RGGB, RGGB, RGGB, RGGB), then reconstructed when playing back. That should be more efficient for compression.

 

[J. Eric Camp]

The only ways a single chip will deliver HDR is:

Pixel binning of varying sensitivity pixels.

or

if the controller board could pull multiple frames out of one exposure.
ie separate images are complied simultaneously from different exposure times.
So at 24fps
Frame 1 starts.
Frame 1a is pulled at 1/192
Frame 1b is pulled at 1/96
Frame 1c is pulled at 1/48
Frame 1d is the full 1/24

However you would still have motion artifacts with this, albeit less than a high frame rate combination. Also this would actually give you DR gains versus over clocking the sensor for no extra sensitivity.

As David says. They only truly acceptable way for motion to gain HDR would be through simultaneous exposures (and exposure times). This can be combined on one sensor or if you use an existing 3d rig, or make a new device similar to a 3ccd camera with a beam splitter but with 3 fully contained CMOS bayers.

 

[Andrew M.]

Properly done downers from 4K to 2K theoretically should give you one stop extra.
Also proper combining of 4 frames in to one should give you another stop extra.
Here temporal movement between frames may help or may not help to average down the background noise.

However I am not sure how ground noise level will combine after going through A/D converter.

Remember, we are not combining analogue values from the sensor, we are processing it as a digital zeros and ones in post and as such you will need not 12 bit A/Ds but 14 bits A/D to start with and the noise level of 4 buckets combined digitally may not be adding much of dynamic range here.
Nothing will replace just 4 times bigger surface of the pixel.

The best will be to try it out.

Andrew

 

[Joe G.]

There's a lot being said here. I believe the initial post was a call to experiment with it. We don't know what's "acceptable" until we give it a try.

What if I just want a nice landscape establishing shot with a fairy tale HDR look?

Any of these methods would probably be better than no HDR option.

Also, if I'm only interested in pulling more highlights, then there could be a blending mode that only takes the whitest areas and ignores the rest. It should be possible to just use the second exposure to capture some blown out highlights, and blend them back into something that looks better than if it wasn't an option.

It's worth tinkering.

 

[Eki Halkka]

The only ways a single chip will deliver HDR is:

Well, that of course depends on the definition of HDR. Your variable exposure example would add five stops of dynamic range. I assume that's something that could be done with better sensor technology in the future (the dynamic range of video sensors has probably improved by more than that amount in the last decades).

Or, in other words, a 1960's video camera using your technique would still probably have a smaller dynamic range than Red One.

Which one is HDR in that case?

Anyway, i.e. the idea of using ND filters in half of the photosites should be able to yield similar improvements.

Oh, BTW, As far as i understand, you don't need to have the exposures bracketed in one stop increments, simply doing i.e. 1/24 and 1/192 should be enough to retrieve similar information: this scenario should work pretty well with the DR of Red One, as the usable exposure range of both still overlaps.

 

[Eki Halkka]

Properly done downers from 4K to 2K theoretically should give you one stop extra.

It pretty much does, also in practice, if you compare the two images in pixel per pixel level. The overall appearance doesn't change though.

Edit: actually, that should give two stops.

Also proper combining of 4 frames in to one should give you another stop extra.
Here temporal movement between frames may help or may not help to average down the background noise.

Yep, but make that two stops. Two frames is enough for one stop. Temporal supersampling works fine, it's actually one much used method on noise-reduction software.

Remember, we are not combining analogue values from the sensor, we are processing it as a digital zeros and ones in post and as such you will need not 12 bit A/Ds but 14 bits A/D to start with and the noise level of 4 buckets combined digitally may not be adding much of dynamic range here.

As long as we're talking aout Red One, 12 bits is supposed to be enough to capture the whole range of the sensor with decent precision. 14 bit A/D wouldn't make a difference if these statements are true.

Nothing will replace just 4 times bigger surface of the pixel.

Theoretically, it shouldn't make any difference if you add together the photons gathered from four small photosites, or one big one, as long as the combined area is the same. You will end up with the same value in both cases. In practice, there probably is a small difference, but not THAT big.

The best will be to try it out.

I did some tests earlier with a small pocket digi camera (Canon Ixus IS 80) and a DSLR (Nikon D90), using the same focal length, ISO and exposure in both, and the same sensor area (in other words, i cropped the DSLR image to match the framing of the pocket cam).

I then downscaled the pocket camera's image to match the cropped DSLR's pixel size.

The resulting images were pretty similar, the DSLR did look better, but the difference might very well be rather due to overall better quality in the DSLR than the difference in photosite size.

Edit: i found it, here's my test with images and all....

http://hv20.com/showpost.php?p=212753&postcount=28

 

[Alexander Smith]

Has the potential for a 3 CCD prism block consisting of 3 RGB CCD's rated at different ISO's been discussed? You would have one for shadow, one for midtone, and one for highlights; each one with independently adjustable ISO's. If it were 2/3" we already have a whole bunch of awesome lenses available that were designed for 3 CCD prism blocks.

 

[J. Eric Camp]

Yes, you are right you have no need for some of the exposures in my example to accomplish the standard way of doing HDR. Usually it's a two to three stop spread between two to three images. That is to say... usually.


The assumption that adding five stops is accomplishable in the future removes it from the category of HDR would also negate adding 12 stops. Just as you said, CCD's from 1970 versus our current photosites should be all the example necessary.

That being said 15 stops is not really HDR.

 

[Eki Halkka]

That being said 15 stops is not really HDR.

Exactly... or, Red One already is HDRI as is. Depends on the definition. My point was just that the "High" part in the term isn't actually too clearly defined.

A big lot of the HDRI images out there, or ones people refer to as HDRI, don't actually have that wide a dynamic range. They could have been just as well processed from a single RAW exposure.

 

[Noah Yuan-Vogel]

I think that any system that uses shutter speed to control exposure for HDR is not likely to be worthwhile for RED (or any system that captures moving pictures) given likelihood of edge artifacts for objects in motion.

As for what is HDR and what is not, I have done some work with HDRI but not much so I just did a few quick tests on my 5D Mark II using only two images and tried using a 6stop bracket and a 8stop bracket. One thing that struck me in particular is that these images rendered the scene with even less contrast than my eyes do looking at the same scene. Conventionally, there is something about the tone mapping in HDRI that creates a color saturation and rendering that is unlike any single-exposure capture system. I think these are an important aspect of HDR. These might not always be good attributes because they can make images seem quite surreal and distracting (although beautiful), but i imagine one could grade this out. More information is better, no?

Most SLRs only support +-2ev bracketing so that is commonly used, and i have no doubt more samples allows for easier working with images, but my quick test seems to tell me that it is quite possible to extend dynamic range very considerably with only two exposures with a big 6-8stop exposure difference. I also think that because the camera is a very specific system it would not be very hard for RED to use an ND1.8 or higher on half RED's photosites, and program their decoder to handle combining the ND and non-ND pixel data. I dont imagine tone mapping 20stops of scene contrast to be displayed correctly on 8bit displays is that simple (especially for skin tones?) but i bet if they can pull it off it could look pretty amazing.

@eki: I hardly think that 13bits is enough for RAW HDR given the fact that CMOS/CCD respond to light linearly. 13bits storing RAW pixel data linearly is only enough for about 13 stops of dynamic range. One must also consider that if you are shooting video in HDR and capturing 18-20 stops of dynamic range, you are likely to often come across scenes that dont have nearly that much contrast, so youd have to increase contrast in post, throwing away some bits. You CAN store 20stops in 12bits but it takes some processing and then if you shoot a scene with 14stops of contrast, youll end up with about 8bits of information.

 

[Eki Halkka]

I think that any system that uses shutter speed to control exposure for HDR is not likely to be worthwhile for RED (or any system that captures moving pictures) given likelihood of edge artifacts for objects in motion.

I wouldn't be so sure such artifacts would be visible, as long as there's no breaks between exposures (360 degree shutter).

Combining those together should yield the same result as just using single exposure with the combined exposure length - except for extreme highlights and shadows. Those would have differing amount of motion blur - it's hard to say what it would look like, without actually testing it.

In the method i'm suggesting (multiple exposures which all share the same shutter speed), there shoudln't be any motion blur artifacts whatsoever, the result should look just like a single exposure with the combined shutter speed.

As for what is HDR and what is not, I have done some work with HDRI but not much so I just did a few quick tests on my 5D Mark II using only two images and tried using a 6stop bracket and a 8stop bracket. One thing that struck me in particular is that these images rendered the scene with even less contrast than my eyes do looking at the same scene.

In most cases, what getting a pleasing image in a high contrast situation would need most would be a few extra low contrast stops in highlights, gently rolling off to white. The shadows and midtones usually look good with pretty neutral contrast.

Conventionally, there is something about the tone mapping in HDRI that creates a color saturation and rendering that is unlike any single-exposure capture system. I think these are an important aspect of HDR.

I don't ;-)

That's just one look, and IMO pretty unnatural as such. HDRI is just an image which holds a large dynamic range, it can be converted to an image that can actually be viewed on a computer screen or whatever in many different ways. My own preference is to treat HDRI just like a normal RAW file - it's just a source with as much information as possible available for regular color correction tools.

These might not always be good attributes because they can make images seem quite surreal and distracting (although beautiful), but i imagine one could grade this out. More information is better, no?

That tone mapping thing is, as said, just one way to use HDRI. The reason the images look surreal, is that when creating a viewable, standard DR image, the software tries to maintain local contrast by using differing color correction for varying parts of the image.

As said, in my opinion it's a vanity look, which is not too useful in general purpose photography - usable for a dream sequence, maybe, but for whole movie, not really...

I also think that because the camera is a very specific system it would not be very hard for RED to use an ND1.8 or higher on half RED's photosites, and program their decoder to handle combining the ND and non-ND pixel data. I dont imagine tone mapping 20stops of scene contrast to be displayed correctly on 8bit displays is that simple (especially for skin tones?) but i bet if they can pull it off it could look pretty amazing.

As said, if i was to color correct such footage, usually most of those 20 stops would be just the highlights, rolling very gently off towards white.

@eki: I hardly think that 13bits is enough for RAW HDR given the fact that CMOS/CCD respond to light linearly. 13bits storing RAW pixel data linearly is only enough for about 13 stops of dynamic range.

Nope.

13 bits is enough to store two 12 bit exposures that are added together - it doesn't matter what the actual content of those exposures is (how many stops you bracket).

Both 12 bit exposures can have 4096 different brightness values - add them together and you have 8192 brightness values, which conveniently is exactly what one can store in 13 bits.

As the longer exposure will clip at 100% saturation, the resulting image is not linear, but that should be possible to compensate when processing the image.

 

[Nick Brian Walters]

I for one can tell you that 48fps definitely generates some oddities when combined for HDR...when I first made this video I actually said that it seemed ok, but I realize I was just trying to fool myself:
http://www.youtube.com/watch?v=4TApJxQGsXo

Here's some examples of another HDR videography experiment I've done:
http://www.youtube.com/watch?v=5PHcitPfNBk

I'm still experimenting as I can with HDR, and am looking at trying to use a simple beam splitter that would usually be used for side by side 35mm 3d photography to do some HDR.....basically it'll split my sensor in two and make two images that are generally the same on half of each side of the sensor...then I'll just record with the whole rig sideways and apply some ND to one section, take out the images in post and recombine for an SD image.

In the future I'd like to try using two cameras either with a plain jane 2 way mirror setup, or a two way mirror setup that looks in on a Redrock M2 adapter.

No matter what, keep up the good work guys, and I'll continue following this thread.

 

[J. Eric Camp]

I don't quite understand how are you suggesting that one would add these two almost identical photos together for luminance gains?



Also...
Then if the object is moving, say it's a white dog running across a black back ground. The dog is blown out by one stop using conventional frame rates and exposure.

In the second frame the dog will occupy a different space and the edge pixels will not line up to be matched together for luminance gains. This includes any edge. The collar, the eyes any color or luminance shift.

Again I say (from my perspective and with respect for yours) the only proper way to extend the DR would be simultaneous exposures from binned photosites or split beams.