Exposing for the RAW format

[Philip Powell]

Raw and saturation question

Raw and saturation question

Is it true, as was stated earlier on the thread that the Raw format (and the Red by implication) reallly records low saturation?

As I thought I understood from Grame, it only appears to be low sat. and that it's actually recording a wide range of saturation levels.

Because I would think, if not, if you were boosting low sat levels in post, it would make it very noisy.

 

[David Mullen ASC]

Most or all digital cameras shoot with a somewhat muted color level until it is boosted in post or by the color processor in the camera, as an HD camera does when recording a REC709 image. The REC709 color specs involve boosting some of the color channels, such as red, so that it looks fairly saturated on an HD monitor. Turn REC709 off in an HD camera and you'll get a more muted image, color-wise. But the color range is not being limited just because it looks more pastel. In fact, the REC709 color space, though it looks more saturated, involves throwing away some color information to fit within the color space that a monitor can handle.

 

[I Bloom]

Is it true, as was stated earlier on the thread that the Raw format (and the Red by implication) reallly records low saturation?

As I thought I understood from Grame, it only appears to be low sat. and that it's actually recording a wide range of saturation levels.

Because I would think, if not, if you were boosting low sat levels in post, it would make it very noisy.

Well in RGB, saturation is determined by the ratio values between red green and blue. Low contrast images are by nature low saturation. So simply by adding contrast to a RED image, you are adding saturation overall. You don't neccessarily need to increase the actual saturation value in REDcine. Though you might, and noise isn't a problem when adding a small amount of saturation.

IBloom

 

[Bill Anderson]

Sir, if you shot a subject that had low color saturation and you wished to boost that in post, then you might take a little quality hit (negligible even). But if the subject was vivid, then all you're basically doing is re constituting that value from the RAW data.
And I think these are two fairly different things. I've never had a RAW equivalent that failed to meet the saturation level of the subject. I think I've had more problems with over saturation and luminosity extremes than less. Of course much of this depends on to what lengths I go to reconstitute a value- but let me just say, there's no better way than RAW. (Yet)

When you shoot for a file that is not RAW, then its info is converted
to a fairly limited set of values as far as allowing you to reinterpret the color and luminosity, etc.. Just like a transparency, everything is baked in so that any bold re-interpretations will come at a fairly high price as far as quality is considered.
The RAW file resembles a film neg in that when you hold the neg up for viewing it really doesn't resemble the scene at all. But once an appropriate paper and developer (RAW conversion method) have been deployed, you will see all the saturation and luminosity that you've so carefully planned for. Not only that but what you see will still be quite open to interpretation- (within limits).
When representing an image: Tiffs are color swatches- RAW is the cans of paint. More or less.

 

[Bill Anderson]

Of course, there's no free lunch and all of the conversion and workspaces are to some degree destructive, fitting square pegs into round holes. However, once we've carefully sifted through our square peg pile and found the best fit:RAW, there's little point in bitching and theorising any farther. It's time to start hammering. At least until someone like Graeme Nattress invents another peg, or hole.

 

[Gregor Hagey]

Red's latitude and effective ISO

Red's latitude and effective ISO

Shooting RAW is new to me and it opens up a lot of creative and technical opportunities. I shot some stouffer step wedges at different T Stops and processed them at different ISOs (in Red Alert). My conclusion is that processing the .r3d in Red Alert at 320 ISO with -.40 exposure gives you the most latitude. You can apply a slight inverted 'S' curve to pull more out of the shadows and highlights, but the resulting image is very low contrast which is great for exporting for colour grading, but not a good look for editorial. When applying a slight inverted ‘S’ curve by eye I can read 24 steps, but by a waveform monitor 27 steps are definitely visible which equals 9 stops of dynamic range.

Upon analysing the steps you will notice that they don’t change on a logarithmic scale. The steps should be in 1/3 stop increments, but at the knee the first two steps are expanded and cover an entire stop each and at the toe the steps are compressed and cover only 1/10 stop each. Just like other video cameras there is plenty of information in the shadows, but it’s important to protect your highlights. When I read the Stouffer on a waveform I see there are actually only 10 steps (3 1/3 stops) above 50% and a whopping 17 steps (5 2/3 stops) below 50%. For the purpose of capturing the most dynamic range placing middle grey at 30-35 (instead of 50-55) on your zebras is the best way to go. When I set the camera to 320 ISO (archaically called ASA in the menu) and expose for 30-35% zebras on the viewfinder and then check the T Stop against my incident meter I actually end up with 320 ISO. It seems Red has already built this concept into the ISO settings on the camera. If I want to expose middle grey for 50-55% zebras I need to overexpose by 2 stops. This results in a cleaner image, but you risk clipping the highlights and reducing your overall dynamic range.

You can view the files here:

http://www.dghagey.com/Director_of_Photography/Red.html

 

[Steve Sanacore]

Gregor;

Have you ever tested Canon or Nikon Pro DSLR's in the same manor as this test? I would love to know the differences in the results. Some of your results here seem to conflict with my experiences with Canon raw files.

I think we are going to need someone from RED to decipher this information and explain the details to us. If RED placed more image information in the bottom half of the exposure then I think this is directly opposite of what Canon and Nikon have done.

I just want to make sure I know the best way to set exposure. I have not yet used Red Alert or Red Cine, so maybe this isn't as complicated as it sounds?

Thanks for the great tests and explanations Gregor.

 

[Gregor Hagey]

Gregor;

Have you ever tested Canon or Nikon Pro DSLR's in the same manor as this test? I would love to know the differences in the results. Some of your results here seem to conflict with my experiences with Canon raw files.

No I haven't tested DSLRs at all. This is the testing I would run for HD, but I don't know if it's appropriate to use a waveform/vectorscope to analyse stills work because a print is your finished format.

In general I set Red to 320 and meter at 320 ISO, but if I have complete control of highlights I keep Red set to 320 ISO and set my meter to 160 or 100 ISO to get a cleaner image.

 

[Esmaeil Neisi]

Wooow

 

[Benjamin Rowland]

Would like to hear someone from the red team weigh in on Gregor's test results. Interesting stuff...

 

[Dj Joofa]

Well in RGB, saturation is determined by the ratio values between red green and blue. Low contrast images are by nature low saturation. So simply by adding contrast to a RED image, you are adding saturation overall. You don't neccessarily need to increase the actual saturation value in REDcine. Though you might, and noise isn't a problem when adding a small amount of saturation.

IBloom

Saturation is not directly the ratio of combinations of R,G, B to each other. It is measured as the length of "outwardness" of the chroma vector. In (device-dependent) RGB space, fully saturated R would be [1,0,0], and a blue green would be [0, 1, 1], hence, it is not directly a ratio.

 

[lordtangent]

My experience form shooting DSLR RAW for quite some time now is that you want to expose as far to the right of the histogram as possible. Even though digital cameras are much more linear in the shadow areas compared to film, the shadow area is where the noise is. Or rather, the noise is more obvious there. It's the random base noise of the sensor but since the "signal" (you image) part is weak compared to the "noise" in the dark areas what you get when you boost the dark areas is more visible noise. So really you want to push the "meat" part of your exposure as high above the noise as possible to get the best picture. (i.e. "expose to the right")

The problem is that doesn't leave you a lot of room to protect for highlights. It's really not practical to underexpose enough to protect for spec hits and direct light hits anyway. Film can only handle gross overexposure to a limited extant as well. But at least over exposed areas don't look as ugly on negative film as digital due to the more organic roll-off of film.

One solution would be if Red switched to a LOG CMOS for the image sensor. There is a technology called LinLog that combines traditional LOG CMOS with programmable roll-off and an instantaneous shutter. It would fix two problems at once! (Rolling shutter and hard clipping highlights) What log CMOS does is add a little analog amp (more like a valve) to each pixel. The voltage of the pixel can be fed into the "valve" and be used to create a analog fall-off on the pixel. The curve is programmable on LINLOG, so part of the exposure can be linear (the meat) and the top can be programmed to roll off. The coolest part is it all happens before the A to D step, so the digital circuits don't even need to know anything about it. (THough it'd probably be a good idea to keep some meta data about what roll-off curve you recorded with) I wish I could get this on my DSLR. It solves what I consider the last remaining curse of digital acquisition.

Here is some LinLog info:

http://www.photonfocus.com/html/eng/cmos/linlog.php

 

[combatentropy]

Veterans of the 35mm negative know to protect the shadows. This is because after a certain point of darkness no emulsion is left on the negative -- i.e., no texture. Meanwhile, there is a great degree of latitude in the highlights.

With digital, you need to drop that habit, er, like a bad habit.

A key word of the medium should tell you that its rules may not apply: negative. That's right, while film is a negative reflection of what you expose, an electronic sensor is a positive. And as it turns out, the rules for exposure are the opposite.

After a certain point of overexposure, an electronic image clips. And just like when no emulsion is left on a negative for the dark areas in a scene, no variation is left in the electronic signal for the bright areas of the scene.

According to the experts who write the software that converts raw files in Photoshop, "50% of the image data lies in the brightest F stop captured"

That is technically true, yes. (http://www.normankoren.com/digital_tonality.html) But gamma and the ultimate conversion to 8-bit from 12-bit help a great deal in evening it out.

The fact is, digital images hold up quite well to boosting the shadows:

http://www.clarkvision.com/imagedetail/dynamicrange2/

 

[Dj Joofa]

( But gamma and the ultimate conversion to 8-bit from 12-bit help a great deal in evening it out.

The fact is, digital images hold up quite well to boosting the shadows:

Actually, the process here is a bit destructive for a sensor that is matched in bit-depth to number of stops it can offer. Firstly, it will *not* create more information in the darker areas (shadows if you will), which it should have ideally, and will only adjust whatever information it has available to the right gamma level. For e.g., Red One has a 12-bit ADC and about 11 stops to offer, and hence, about over 70% percent information is thrown away in the range from standard reference black to standard reference white, under certain assumptions.

Therefore, the situation is a little bit tricky in shadows. On the other hand, it is fine to throw away extra information in the brighter stops as along as you don't intend to do tone mapping later, say during color correction.

Some readers may be perplexed at my figure of 70% information being thrown away when the digital camera images they acquire look very good. Well the reason is that most of the discarded (not captured) information is in the darker stops, where as a lot of time people are not operating in that range, say shooting outdoors during the day, or reasonably-lit interiors. In the range of mid tones to whites the information loss is less perceptive, as the distribution of discarded information is less dense. However, under low illumination the loss in the darker stops shows up as posterization, which needs to be reversed.

 

[lordtangent]

The fact is, digital images hold up quite well to boosting the shadows:

That's true... but as I mentioned even when underexposing it's tough to protect for gross over exposure anyway (spec hits and other stuff that tends to look "clippy" when shooting digital) Putting your exposure to the left of the histogram to gain headroom is a bad trade-off unless you are willing to accept the noise you'll get as a result. The Red One sensor is pretty clean, so it might be possible to trade off some noise to get more head room and still get an acceptable result. But I doubt you will ever be able to get enough headroom to make any difference on grossly over exposed stuff anyway. (i.e. the things you ARE NOT exposing for. Not the picture but stuff like spec hits etc) I know I've never been able to really emulate film-like roll off that way with my Canon 20D though. (Sensor is not clean enough to "under expose" enough to leave headroom for that stuff)

Digital exposure is essentially linear. Reducing exposure by one or two stop in digital only buys you one two stops of headroom (not much at all when talking abut stuff like spec hits and other grossly over exposed areas). Noise is linear also but PICTURE, is gamma corrected. A one or two stop under-exposure usually increases the apparent noise considerably. the reason is because when you make "picture" from that linear data you are pushing up the noise by a power function in the process.

IMHO, the best approach is to try to put the "meat" of the exposure as far to the right as possible. The spec hits are gonna clip anyway. Don't worry about shadows at all. With digital the toe is very linear so you will get detail there. As long as you don't push it up too much the noise will stay buried down there and you wont see it too much. But your goal should be to get the cleanest mid range exposure ( without over exposing of course). You want as much "signal" as possible over the noise of the sensor.

My dream is a digital sensor with some sort of roll-off to help take the "curse" off of over exposed areas. As I've mentioned several times LinLog would be a solution but no camera manufacturer seems interested in putting it in a production camera.

 

[Dj Joofa]

LordTangent, I think you are mixing up a few things. Firstly, when you say putting up "the exposure as far to the right as possible", there are three gross ways of doing it:

(1) Increase the size of the aperture
(2) Increase the exposure time
(3) Boost up analog gain

Under additive noise model, (3) will always increase the noise (say your camera increased the ISO rating up by boosting analog gain), and hence, the assertion "the noise will stay buried down there", shall not be correct.

For (1) and (2), if the noise statistics show a tendency to converge to a fixed number given sufficient time, then it can help, and you can move your exposure to the right using (1) or (2). However, (1) is normally fixed by dof and other requirements, and (2) is bounded by the frame rate and the amount of motion blur you want to tolerate.

In addition, in reference to your comment on LinLog, non-linear CMOSes are being investigated for some time. They have both advantages and disadvantages compared to a linear CMOS.

 

[mikeburton]

Here is an interesting article that stu maschwitz wrote a few days ago. Interesting read and some good information in this write up,
http://prolost.blogspot.com/2008/02/digital-cinema-dynamic-range.html

 

[Dj Joofa]

Here is an interesting article that stu maschwitz wrote a few days ago. Interesting read and some good information in this write up,
http://prolost.blogspot.com/2008/02/digital-cinema-dynamic-range.html

Stu's article is very interesting and informative. I could be wrong but this is my take on that issue: The last time when I read it appeared to me that Stu was taking the notion of the implementation of a particular manufacturer's implementation of Rec 709 space and how they treated the white point as a guide to expose for the log space. Graeme Nattress has already pointed out that the implementation of white point for the Red Camera is done by a 0.4 EV boost up, and hence, to "see" that extra highlight in Rec 709 that one sees in Red Log one should use a -0.4 EV value in software.

 

[Mitch Deoudes]

(1) Increase the size of the aperture
(2) Increase the exposure time
(3) Boost up analog gain

What about (4) Add lights / remove ND? As you noted, 1 & 3 aren't really options in this case, and changing the shutter angle isn't the first thing to try.

 

[Dj Joofa]

What about (4) Add lights / remove ND? As you noted, 1 & 3 aren't really options in this case, and changing the shutter angle isn't the first thing to try.

Yes, you are right about (4). I was not thinking about it. I thought LorgTangent was aiming for a fixed lighting and only going for an in-camera adjustment.

But, again, your point is well-taken, (4) should be there.