Red hdr?

[kraemer]

I am just curious why I havent seen any mfg seriously chase HDR digital cinematography. Given enough bandwidth between the sensor and the frame buffer, it should be theoretically possible to capture tens of thousands of frames per second out of the sensor. With this in mind wouldn't it be possible to capture +1EV and -1EV of every frame for an HDR composition, either to storage or real time composition in camera?

Just a thought.....

 

[Noah Kadner]

Search the boards and Graeme will chime in on why this simply isn't possible with current sensor technology. Remember HDR photography in still cameras is possible because it's stills that can be made up of adjoining frames merged together. At 24fps there's no extra frames to use. Although I see no reason why someday a camera won't be able to shoot 3 times as many frames at +/- EV settings and merge back automatically somehow compensating for the differences in motion. Lots of processing power/faster chips will be needed.

Noah

 

[kraemer]

Well that is why I am asking. The bandwidth between the sensor and frame buffer is the limiting factor. I cant imagine it being too slow to get away with HDR when its theoretical maximum refresh could be in the tens of thousands per second. If it can do 60p, then at 24p its got bandwidth to spare....

 

[Chris Newman]

I have read the other discussions, but this one is close to something that could actually work. For my example, consider a sensor that has fast enough A/D's, etc. to do 192 fps (8 x 24 fps). To create the effect of 24 fps with 1/48 sec (180 deg) shutter speed, it could shoot at 192 fps (and 360 deg), add up 4 adjacent frames, store the result, throw out the next 4 frames, etc. The 4 frames it keeps are shot at 1/192, so it will take 2 stops more light to over-expose them, but when you add them up, the noise averages (assuming the camera has sufficient bit-depth), so the noise level will be the same as if it was shot at 1/48 sec.

Net result: 2 stops better dynamic range.

This is actually a feature that could be implemented in post-production software, but would require 8 times as much production storage space. But in this case you would have the added benefit of a slow-motion option if you need it, or you could choose your shutter angle (in 45 deg increments) in post.

Are Graeme or any third-party software developers listening and have any opinions or questions about this?

Chris

 

[Jeff Brue]

look at the phantoms EDR mode if you're interested. The big downside to any of this is the need for motion vector based combination which can still result in weird artifacts. As far as HDR look to companies that are integrating black silicon. Anything else is going to get expensive very fast.

Also one other note...for locked off shots think about integrating multiple frame exposures in the DI. For extended DR also there a lot of tricks in regards to motion control that are possible.

 

[Chris Newman]

Phantom's EDR feature:

EDR - Extreme Dynamic Range Exposure, or "dual-slope integration," within a single frame, is a very useful feature when the subject of interest contains very bright areas as well as darker areas and both must be exposed properly throughout the cine recording. The EDR function permits the user to expose for darker areas in the subject via the sample rate and exposure time settings, then set a separate and independent pixel level, EDR Exposure, for all the portions of the image that might become over exposed. The option is automatically enabled when any value, other than zero, is entered in the EDR Exposure field.

This seems to allow you to use different exposure settings for different parts of the image. This and some of their other features seem to be designed for research applications. For filmmaking, I suppose you could have a post-production filter to change the exposure back, but as you say, it seems like shots would rarely be static enough to allow this to work very well.

 

[Eki Halkka]

I have read the other discussions, but this one is close to something that could actually work. For my example, consider a sensor that has fast enough A/D's, etc. to do 192 fps (8 x 24 fps). To create the effect of 24 fps with 1/48 sec (180 deg) shutter speed, it could shoot at 192 fps (and 360 deg), add up 4 adjacent frames, store the result, throw out the next 4 frames, etc. The 4 frames it keeps are shot at 1/192, so it will take 2 stops more light to over-expose them, but when you add them up, the noise averages (assuming the camera has sufficient bit-depth), so the noise level will be the same as if it was shot at 1/48 sec.

Bingo!!




I had the exact same idea - tried to suggest it directly to RED, not publicly here, as i thought it might actually have been patent worthy. Unfortunately, they do not accept research/development ideas from outsiders at this point. Anyway, as the cat is already out the bag and thus public domain, and Phantom also seems to have something similar already... here's my take on the same issue (long post):




Timeslice subframes: SNR / DR improvement with firmware and software modification for temporal supersampling

By Eki Halkka, 17. Nov. 2008 (Edited 20. Nov, 23. Nov)




Intro

The idea is to use "exposure timeslice subframes" to store more information for each frame, using temporal supersampling - a bit like adding more cores to a processor instead of making it clock faster, if you may.

Dynamic range can already be expanded with temporal supersampling using current tools - by shooting at higher frame rate and combining the frames in post. Simply averaging two frames together reduces noise in half, or in other words adds 6dB to signal-noise ratio (SNR). Adding the frames together adds one stop of dynamic range (DR).

Unfortunately, this is a rather complicated procedure in practice right now.

What i suggest, is to make temporal supersampling an integrated part of RED work flow. Ideally, this should be done with the raw linear sensor data, before debayer.




Example case

For the sake of simplicity, i'll use a target of 25 fps and 1/50 shutter (nice round numbers) on the following example of a practical implementation:

The camera sensor is set to run at 100 fps, 1/100 (360 degree) shutter speed. The exposure is set properly for this, not the targeted 25 fps, 1/50 shutter speed, thus preventing overexposure by one stop.

Out of these originally captured 100 fps frames, or exposure "timeslices" as i like to call them, each group of four frames would be treated as “subframes” of a final 25 fps frame.

Captured frames 1 and 2 of each group would be saved, but with a flag that describes them as partial frame 1a and 1b. Captured frames 3 and 4 would be discarded.

The result would be a 50 fps stream, but the extra frame rate would be used to store twice more of intra-frame precision, not more temporal information.

If possible, the unused subframes (3 and 4) would actually never be captured at all – the sensor would have a bit of time to cool down, maybe it could help to implement this at higher target frame rates.

Supersampled subframes give a lot of new flexibility in post production:

A) If subframes 1a and 1b are averaged together, the result is an 1/50 shutter speed frame which has 6dB lower noise, but the same dynamic range.

B) If subframes 1a and 1b are added together, we end up with an 1/50 shutter speed frame that has the same noise level, but one stop higher dynamic range – it's essentially a 13 bit accumulation of the original two 12 bit images, reaching sensor saturation one stop later. This will naturally require the post production software to be able to hold values that “go over 100%”, floating point style.

C) Any blend of the above is also possible - they're two sides of the same phenomenon.

D) One bonus possibility is to simply discard one of the two sub frames - user could modify shutter speed in post between 1/50 and 1/100 - at the cost of losing the improved SNR / DR.

E) If all four captured 100 fps timeslices were saved as subframes instead of just the two in the above example, user could have also 1/33 and 1/25 shutter speeds at his/her disposal, trading in more motion blur and capture storage space for additional DR and SNR in the resulting image, making the blur-noise trade-in decision in post rather than on the field.

F) Timeslice subframes might also open up other interesting possibilities, i.e. isolating moving areas of the frame by difference matting etc.

G) As a bonus, this method would work as an "electronic ND filter", the aperture could be opened by a stop without overexposing.

Summing it up

+ At least 6dB less noise (12 dB with four subframes)
+ At least one f-stop better dynamic range at original noise level (two f-stops with four subframes)
+ Ability to control shutter speed in post (i.e. 1/100, 1/50, 1/33, 1/25)
+ Capture proper motion blur for video and faster shutter speed for stills simultaneously.
+ Can also be used as an "electronic ND filter"
+ Should help to minimize compression artifacts by averaging them over frames
+ Should be doable with current hardware, with minor changes in firm- and software.
+ The technique scales up as technology evolves

- With Red One, the technique probably only works in windowed modes
- The technique only works at relatively slow target frame rates
- The shutter speed variation is limited
- The storage requirements increase
- The post processing burden increases
- The technique improves the dynamic range "from the top up" by preventing overexposure - there's no low light sensitivity gain.​

Doable with current technology

Red one can shoot 2K up to 120 fps, which is enough for the above 6dB / one f-stop improvement with frame rates up to 30 fps. Depending on what the frame rate bottleneck is (sensor, the electronics, redcode, other..), it might be possible to use three or more subframes instead of two in a "burst mode", for an additional 3dB / half stop improvement, especially as the sensor has time to cool down between bursts.

Using this method would require adding a new timeslice/subframe recording mode to the camera's firmware, with the flags i described above. As there's probably not enough processing power in the camera to do proper frame blending etc., the real time preview should be modified to account for this method - possibly simply by adjusting display gain if needed.

The post production software would also need to be modified to read the flags and combine frames from the new recording mode, in the ways i described above. If individual frames cannot be flagged in redcode, it would probably be enough to simply to have "footage uses two subframes" flag in the clip's meta data.

But that's it - that's all that's needed to make this work, as far as i can see.




But wait, there's more

In the future cameras, faster sub frame rates and even bigger improvements could be obtained. If RED ever comes up with a 1000 fps high speed camera, the high temporal precision could be used to store many, many additional stops as timeslice subframes - the camera would in fact classify as a true high dynamic range motion camera.

Another highly desirable option would be to use time slices with varying shutter speeds - the same way as HDRI is currently captured with still cameras. I assume this would require much bigger modifications though - but the potential benefits would also be big. In addition to capturing HDRI, the same footage could be used for video with a (relatively) slow shutter speed partial frame and for stills with (relatively) high shutter speed partial frame - the trade off is differences in exposure, (still frame would be underexposed), but the benefit of usable, sharp stills of moving subjects might be worth it.

 

[Alexander Smith]

What about a 3CCD type arrangement of sensors but instead of separating the image into red, green, and blue it has 3 sensors of varying sensitivity.

Or maybe, something like this: http://www.imaging-resource.com/NEWS/1043202460.html

 

[Tom Lowe]

Why not simply compose your HDR scenes off the single 24p RAW files? You could render out three frames - one stop under, one stop properly exposed, one stop over, then merge them?

I don't care for HDR, though. The software just aint there, IMO. Even single frames don't look good to me, unless the HDR is extremely subtle. Once you start applying those algorithms to moving images, things get even worse, from what I have seen.

Better just to master grad NDs, or shoot in great light! :weight_lift:

 

[Eki Halkka]

Why not simply compose your HDR scenes off the single 24p RAW files? You could render out three frames - one stop under, one stop properly exposed, one stop over, then merge them?

Because that wouldn't add any new dynamic range. It would be the same as just adjusting the curves, only with a lot more work ;-)

 

[Eki Halkka]

What about a 3CCD type arrangement of sensors but instead of separating the image into red, green, and blue it has 3 sensors of varying sensitivity.

Or maybe, something like this: http://www.imaging-resource.com/NEWS/1043202460.html

Cool stuff - the difference naturally being that these are hardware based advances, which will take time to produce. Supersampling on the other hand works with current technology.

 

[Uli Plank]

Sorry, I have to contradict. HDR algorithms like tone-mapping are changing contrast locally, not globally. It does not look the same as an adapted curve for the whole frame.

That said, it's true that tone mapping can look awful with moving images, but it depends a lot on image content. I got some nice landscapes by exporting 16 bit TIFF sequences from RA und batch processing them in Photomatix. You need experience (I've done HDR photography for more than 4 years) and hav to tweak the tone mapping values carefully.
The audience declared the footage as "very painterly" and didn't even notice the treatment…
So, yes, a single RED can do HDR!

 

[Eki Halkka]

Sorry, I have to contradict. HDR algorithms like tone-mapping are changing contrast locally, not globally. It does not look the same as an adapted curve for the whole frame.

No, HDR images are just what the name suggests. High Dynamic Range images. HDR doesn't imply how the images get used for standard range output if at all (HDRI:s are often used in computer graphics as is). SOME algorithms do change local contrast when converting HDRI to standard images, but that's a deliberate choice made in post production.

Adjusting local contrast is not required (or usually even desirable) for the conversion - changing global contrast usually yields more natural results in my opinion.

Anyway that's a matter related to decoding HDRI to SDRI, not to acquiring the images.

Adjusting the raw image does not add any new dynamic range to footage. That requires something extra to be done when shooting. Applying local contrast to RAW footage doesn't add any new dynamic range either, but it may make the existing dynamic range more visible.

 

[Chris Newman]

Or maybe, something like this: http://www.imaging-resource.com/NEWS/1043202460.html

While they are at it, they can make the blue pixels bigger than the red and green.

What about a 3CCD type arrangement of sensors but instead of separating the image into red, green, and blue it has 3 sensors of varying sensitivity.

I don't care for HDR, though.

This thread is starting to repeat the previous HDR thread. The 3CCD thing was brought up, except you only need two (with one beam splitter that sends most of the light to one of two identical sensors).

Tom, the point is dynamic range is widely considered the only advantage that film still has over digital. The output doesn't have to look like stereotypical HDR pictures. You can grade it to use the dynamic range however you want.