RAW workflow - a discussion

[Hans von Sonntag]

For reference something in the ballpark of the "RED reference" (being RC3/RG3/800/daylight) settings, should be fine.

Personally I think that Red's mantra of ISO 800 is a disservice. The minute Scarlet was in the fields many threads popped out here and there that were dealing with noise. People are used to WYSIWYG. The LCD/EVF is bright, there is ISO, Red says ISO 800 is the ballpark, let's push ISO a bit. After all it's RAW! If Red's advice would be ISO 400-500 we would see much, much less noise and people in trouble.

BTW, I shoot all stuff at ISO 500. Sometimes I close the the iris for half a stop for better DR. Same what I did with film just the other way round. ISO 800 is a noise-prone setting. And in RedCineX ISO 800 kills highlights. ISO 500 is the much safer bet, IMHO.

But ISO 500 sounds slow. Is ISO 800 a marketing setting?

Hans

 

[Gunleik Groven]

I think there are other reasons for the 800 advice, and like 320 or 500, you just have to learn to make it work.

But here are my reasoning around 800.

The camera calibration point is 320/daylight. (Not its "native" 18% greypoint. THAT is another story and seems to be debated a bit between those who have tried to measure it technically. But 320 is a good startingpoint for practical use, and I actually happen to think that the advices on how to use the camera is generally very good!)

If you ballance the cam from daylight to tungsten, you'll set a clamp on the red channel and when that clamp/limiter/knee is too hash, you get the usual "videoish" yellow/metall skincolors in the top, because the RED channel is not clipped really, but too tight.
@ 800 you have the underexposure headroom (1,25 stops) needed for a whiteballancing to tungsten NOT to be destructive for the RED channel, and you can thus use the cam much as a videocam and still get excellent results.
Which is good for an awfull lot of users.

With RG3 (which looks like it is slightly more "linear" in the top, and has a bit of a harder curve in the bottom than RG2) the highlights are better brotected as "viewable without fiddling" compared to RG2 and especially RG. Thus you can have a visual represntation of more of the highlights available, and a nice and punchy image "out of the box". And that image can be whiteballanced without significant problems, as long as you shoot 800.

That's how I interpret this.

Noise is a different story all together. You can create more or less of it depending on how you light and expose independently of ISO setting.
As the ISO is already made through floatingpoint curves, there is no longer a noise gain by lowering the iso and compensating with a curve. There can be other reasons to do this, but noise is not part of that equation anymore.

So I think 800 is a good middleroad that can make most shots look good if you have shot them well...
No significant tradeoffs and a lot of operational gain and ease of use introduced.

And you still "shoot RAW" so you can tweak whatever you're not happy with, but you have a pretty "safe-bet" startingpoint.

But I do agree on your "RAW" remark...

But... As I said... This is just my ramblings on the subject...

And I don't worry much about the difference between the cameras. I allways found M and MX much closer than MX and Epic...
On the show I am currently working on, I can spot all the Epic shots from the other two cameras right off the bat.
And for my taste, the Epic images outshines the R1 MX ones with a wide margin...

 

[Gunleik Groven]

Yes, but it gives their cameras a look, rather than letting the user pick. It is as much a burned in look as Canon's cameras because you cannot work around it if you want to, and its a look that changes with revisions to their SDK.

Sure, to some degree. You can allways choose one of the other matrixes, though, and I haven't found this a huge practical drawback or limiting factor.
As to the revisions... THAT is one of the very good reasons to "burn" RGB files when starting post on larger productions.
Colorconsistency.

We like that, don't we?

 

[Lliam Worthington]

Hans, I agree...

And I want to bring whomever wants to along the road, without getting too geeky and quick...

I think a bit of this info can be usefull to a whole new generation of R3D shooters and posters these days...

It's not really REDucation. It's just a slow walk down signal paths...

Cheers!

G

While to be honest, I still struggle a lot with some of the language/knowledge levels and posts on threads like this, your intent is greatly appreciated Gunleik. Thank you. Without threads like this, it's very hard to see myself ever being able to consistently get the best out my images and this thread has also certainly helped to put things into full perspective when it comes to transcoding and given much food for thought.

I have a RAW/metadata grading question that I hope makes sense. With a more extreme grading choice or "look" such as in the "Book of Eli" If the relevant creatives had already agreed upon the look, how far would you consider seeking to push the metadata settings towards optimally achieving "that look" before say transcoding to DPX files for further grading in Resolve or the like? If so what advantages would be most apparent? If not, why not?

I see some people talk of doing just what they require to get a nice even palette / full digital negative and not pushing a lot further with the RAW files. There's a number of very well made points regarding the true "power" of raw, but I was under the impression there was an adaptive power when it comes to creating more extreme looks?

Thanks for your time everyone.

Best

Lliam

 

[Graeme Nattress]

"The camera calibrationpoint (not really "native point") is 320/daylight. 800 Is a good practical place to start."

The native white balance, for want of a better word is nearer daylight than tungsten. That is because silicon itself is blue deficient. As for the ISO debate - I don't really see how you can ascribe any ISO as native when the ISO themselves say for a RAW camera that ISO is the manufacturers choice of preferred exposure. ISO doesn't really have a meaning beyond that for a RAW camera.

Now, to picking an ISO v ETTR. On our RAW cameras, they're two different exposure methods. In ISO you pick an appropriate ISO and adjust exposure according. With ETTR you use the clipping indicator tools to expose, then adjust ISO to get reasonable image brightness.

Graeme

 

[Gunleik Groven]

"The camera calibrationpoint (not really "native point") is 320/daylight. 800 Is a good practical place to start."

The native white balance, for want of a better word is nearer daylight than tungsten. That is because silicon itself is blue deficient. As for the ISO debate - I don't really see how you can ascribe any ISO as native when the ISO themselves say for a RAW camera that ISO is the manufacturers choice of preferred exposure. ISO doesn't really have a meaning beyond that for a RAW camera.

Now, to picking an ISO v ETTR. On our RAW cameras, they're two different exposure methods. In ISO you pick an appropriate ISO and adjust exposure according. With ETTR you use the clipping indicator tools to expose, then adjust ISO to get reasonable image brightness.

Graeme

Thanks Graeme!

Liam. Just have to get some work done, first...
The question makes a lot of sense!

Gunleik

 

[Gunleik Groven]

Just from looking at the jpeg's, are you sure all had the same gamma curve?
If so, can anyone (Graeme) explain the different response (same sensor, same gamma, same color matrix) of the cameras?

Thanks.

If you look at the jpeg's - the clippingpoint at the top seems to be pretty uniform, so I guess that is another hint I didn't mess up the aperture....

The lenses are not the same, but same generation (delivered within a year) Masterprimes 35mm from sets we have used to interchange for productions...

The whole AD/processing thingy is quite different between the two cameras, and I don't expect RED to have spent all that R&D on making the cameras "the same". It definitely doesn't feel like that. But there are some interesting differences in the highlights and lowlights which I may comment a bit on if I feel like it. But THAT is not the main purpose of this thread...

 

[Graeme Nattress]

Although the sensor is the same, the circuitry driving it is rather different in Epic than a Red One M-X.

Graeme

 

[Lliam Worthington]

Liam. Just have to get some work done, first...
The question makes a lot of sense!

Gunleik

Thanks Gunleik, just whenever you have the time

 

[Hans von Sonntag]

As for the ISO debate - I don't really see how you can ascribe any ISO as native when the ISO themselves say for a RAW camera that ISO is the manufacturers choice of preferred exposure. ISO doesn't really have a meaning beyond that for a RAW camera.

Greame, if Red's preferred ISO is ISO 800, what does that mean in terms of "sweet spot of the sensor" ? If I have an overexposed picture that I develop at ISO 800 I will lose information in the highlights in comparison to a development at ISO 350, regardless, of course, what ISO I used when shooting because ISO is metadata anyway. If I try to develop even lower, say at ISO 100 I don't gain anything in the highlights. Hence my assumption that the sensors "sweet spot" is around ISO 250-320.

The problem for the user arises when in the SDK when ISO is not metadata anymore but a very important figure for a correct translation of the sensor's RAW data to RGB.

If I would transfer ANY picture at ISO 320, regardless of exposure, to RGB using a meaningful gamma curve such as RedLogFilm is there a scenario that transcoding at ISO 800 would give some benefit for later grading?

Interesting!

Hans

 

[Gunleik Groven]

While to be honest, I still struggle a lot with some of the language/knowledge levels and posts on threads like this, your intent is greatly appreciated Gunleik. Thank you. Without threads like this, it's very hard to see myself ever being able to consistently get the best out my images and this thread has also certainly helped to put things into full perspective when it comes to transcoding and given much food for thought.

I have a RAW/metadata grading question that I hope makes sense. With a more extreme grading choice or "look" such as in the "Book of Eli" If the relevant creatives had already agreed upon the look, how far would you consider seeking to push the metadata settings towards optimally achieving "that look" before say transcoding to DPX files for further grading in Resolve or the like? If so what advantages would be most apparent? If not, why not?

I see some people talk of doing just what they require to get a nice even palette / full digital negative and not pushing a lot further with the RAW files. There's a number of very well made points regarding the true "power" of raw, but I was under the impression there was an adaptive power when it comes to creating more extreme looks?

Thanks for your time everyone.

Best

Lliam

It's a trick question! With two very definite lines of answers.
I'l refer to one, and emphasis on the one I prefer. But I guess there will be as strong arguments for "the other way" of looking at this. And both can give good results. But both lines of answers starts on set, not in post.

And if you choose one way, my opinion is that you should stay on that road until delivery.

Thus this is more of a philosophical question than a purely technical one...

Remember that every color decission you do in post, is technically destructive. You throw away data. The further you deviate from the original exposure, the more data you throw away, and as such - the "thinner" your image will be.

This is true independently of which of the two ways you choose.

And artistic vision, is all about making choices. And stick to them. At least until you consciously choose to choose another approach. And stick to that.

But your files can only carry so much info, so you have to make a priority as to what is the most important to you:
Either
• maximum flexibility in post,
or
• an exposure which is as close to your intended result as possible.

Both these exposures will contain the same amount of info, but the second will contain more of the "right" info and less of what you do not finally want. The catch is that you cannot really change your mind THAT much in post.

So my answer starts on-set…

To illustrator the two ways, I'll call the one the "ETTR" way, and the other the "photographic" way.

When exposing ETTR primarily, you ensure image-integrity and levels available to your best knowledge. The image may not look too finished, but will be very flexible in post, as you have taken care for your lowlights - as to not exposing them too low, and exposing crucial parts of the image in a part of the expire where you have high precision and can gently pull them back where you want them in post.

For such an exposure, your main focus would be to cary all the image data through unharmed and trust the colorist to finnish it off in the intended way.

NOW. That sounds all nice and pretty.
Only catch is that you deliberately take a lot of info you already know you don't want into post, and that a post shaping of the image doesn't necessarily look as good as an image already photographed the way you want it…


Short anecdote to follow.
Last year I got to work with a really great and pretty famous photographer. And he loves to expose dark images. Or: Images that have really dark/black areas.
His AC was pointing out to him that that could be an issue for post. Nothing would look good down there. He responded along the line of:
WTF. I don't want those areas to be seen. I really want to ensure they're black.

So… There is a huge difference between a "low exposure" and an "underexposure"…
And there isn't really any "right" exposure, just "the right exposure for this particular shot".

And I am huge fan of this way of thinking…
Paint the image as far as you can on set with lights and shadows and colors WITHOUT touching the metadata in camera. Stay as close to what you know to be the center/neutral settings on the camera as you can and use lights, and filters to get the image you want.

Another anecdote from a feature shot on the original M.
There was a dark club-scene that was supposed to look rather gently blueish with tungsten lights.
The DoP had WB'd the cam to tungsten and set the lights and blue intensity for the monitors.
Only catch was that in post, the blue was noisy, and despite wanting to have a blue look to the scene and enough lights available, he ended up underexposing blue, because he white-ballanced the camera and lit for that. And there has been a few of these shots lugging around, that are totally unnecessarily compromised by "grading on camera" approaches…

So, if you can:
Have a vision.
Expose for that.

If you do that, you do not have to tweak the metadata so much to get where you want in the firstlight, and you'll have more of the info you need. You can still force yourself in the ETTR direction for technical reasons, but you try to focus not so much on that, as to the image you ultimately want to get home.

If you do that properly and have familiarized yourself with the camera before you take it to work, chances are that metadata will not mess up your online/color-grading.

The more radical your approach is, the more important is this for your images integrity.

So I think the best way to avoid problems with metadata is to shoot for the sensor and light for what you want. The best way to avoid problems is to not create them in the first place. That way, you'll have a real fat and relevant negative when you come home.

Others may disagree with me on this.




And now, I'll take the weekend off and continue to update this thread on Monday.

The plan is to through several ways of developing the files and look at their advantages or tradeoffs for different purposes.
But please download themm and run your own tests...

The RAW files and DPXs can be found here:
http://www.reduser.net/forum/showth...a-discussion&p=1001507&viewfull=1#post1001507



Have a nice one!

G

 

[Graeme Nattress]

Hans, you can't define based on how much detail you see in highlights. That is entirely content dependent, and although you can always lower gain to see "more" up to a point, that only means what you're seeing is the difference between the lower dynamic range of the display and how the tone curve maps the data to that display.

RLF retains all detail over a wide wide range of ISO. At that point ISO sets "rendering intent" to give an indication of how bright or dark the image is. There's absolutely no need when using it to set ISO lower than the rendering intent to try to capture more highlight range as it's all captured regardless, with only a slight curve bending it in between 3200 and 6400 and beyond.

Graeme

 

[Noel Sterrett]

The native white balance, for want of a better word is nearer daylight than tungsten. That is because silicon itself is blue deficient..

In my opinion this is one of the most important and all too often overlooked facts about silicon-based cameras.

Those of us who grew up shooting film always understood that the white balance of film was "baked in" at the time it was exposed, and that filters should be used to adjust daylight balanced film when shooting in tungsten (80A,B,C) and tungsten balanced film when shooting in daylight (85). If you didn't, there was a rather severe price to pay.

Now that has all been forgotten, and replaced with the myth that silicon can be adjusted in camera with the press of button (AWB), or later in post with a magical "white balance" slider. It can't. Just like film, it's actually baked in by the sensor when exposed.

Obviously, you can adjust anything in post, including white balance (either from RAW or after de-bayering), but not without a cost. If you expose a daylight balanced sensor in tungsten light without filtration, you will necessarily sacrifice dynamic range at both ends: blues will become nosier or crush, and reds will clip earlier (and produce ugly yellow peaking).

The downside to proper filtration, of course, is that you'll need more light (and it's also a pain). But that has always been the tradeoff, and at least with film we learned to deal with it.

So the point for RAW workflow is that the ability to adjust white balance is not really preserved by staying in RAW, it is set when shot.

Cheers.

 

[Graeme Nattress]

Noel, there's three basic ways to adjust white balance - optically, analogue gain and digital gain. All have advantages and disadvantages. With the M-X sensor we found digital gain offered the best balance. We may change that in the future as every sensor is different, or we may find that it continues to be the best balance. All we can say about it is that we do evaluate it and act accordingly.

Graeme

 

[Gunleik Groven]

Thanks Graeme and Noel!

I'd say that if I have the chance, I'd have lit the scene for as close to native WB as I could, as that will create less distortion in post, though. That is NOT to say that REDs WB doesn't work really well. But we'll look into that next week.

Cheers!

G

 

[Hans von Sonntag]

balanced film when shooting in tungsten (80A,B,C) and tungsten balanced film when shooting in daylight (85). If you didn't, there was a rather severe price to pay.

I humbly disagree. I shot many projects on tungsten film such as the boring 5293 and used never a 85 for daylight. The later colour timing gave the footage a distinct desaturated look but was right for he eye. Much nicer than with the 85. 5393 was most boring when shot in a tungsten environment. Hate it. 5298 looked much nicer but was clearly grainier, actually not ISO 500 but ISO 250.

I'm still searching for LUTs and such that will make my Epic's matrix somehow funky. Perhaps I should start using filters from the past such as coral, tobacco, etc... This will influence the RAW severely. WB it later will probably bring interesting results. Off to my trusty rental house.



Hans

 

[Hans von Sonntag]

I'm still searching for LUTs and such that will make my Epic's matrix somehow funky.

There is no matrix in the Epic. It's RAW! Oh well...

Hans

 

[Hans von Sonntag]

Hans, you can't define based on how much detail you see in highlights. That is entirely content dependent...

I'm understanding partly what you are saying. But isn't a clipping white a clipping white? Clipping is not content dependent, is it? And the stop before clipping isn't either - probably? ISO in the Red SDK determines where the whites, the mids, the blacks are set in an RGB histogram. Up to ISO 250 I can find more information in the whites, below ISO 250 the picture becomes only darker. Above 250/320 (not sure really, hard to determine between the two) I CAN compress the whites if the footage is exposed TOO much to the right.

What do I miss?

Hans

 

[Graeme Nattress]

Hans, you seem to be mistaking clipping for tonal re-distribution. Normal R3D development doesn't clip whites, but instead distributes which code values in the output image get assigned to various ranges within the image. As monitors have a low dynamic range compared to the input R3D data, that means as you adjust the ISO you'll see more in the shadows or more in the highlights, but at no point do the highlights or shadows get clipped. You may get more precision in the highlights but as in linear light data they're so over-precisioned anyway, that's hardly and issue.

When we map high dynamic range data from the R3d down to the low DR of the display we use a tone curve. That tone curve "pegs" shadow and highlight points and remaps in-between. I can test this on fake sensor data with known detail in the highlights and see that such detail which may appear lost at first glance in a ISO6400 export is actually there still, although at a reduced precision than it is at ISO800 and lower ISOs offer no real increase in precision.

I just re-wrote to put a small pattern in the brightest area of my test data at 0.2 stops down from the main pattern and it's still utterly visible at ISO6400 and hasn't been clipped away. Yes, it's now represented at less precision, but it's still there and still visible (even to the naked eye without adjustment).

Graeme

 

[Lliam Worthington]

It's a trick question! With two very definite lines of answers.
I'l refer to one, and emphasis on the one I prefer. But I guess there will be as strong arguments for "the other way" of looking at this. And both can give good results. But both lines of answers starts on set, not in post.

And if you choose one way, my opinion is that you should stay on that road until delivery.

Thus this is more of a philosophical question than a purely technical one...

Remember that every color decission you do in post, is technically destructive. You throw away data. The further you deviate from the original exposure, the more data you throw away, and as such - the "thinner" your image will be.

This is true independently of which of the two ways you choose.

And artistic vision, is all about making choices. And stick to them. At least until you consciously choose to choose another approach. And stick to that.

But your files can only carry so much info, so you have to make a priority as to what is the most important to you:
Either
• maximum flexibility in post,
or
• an exposure which is as close to your intended result as possible.

Both these exposures will contain the same amount of info, but the second will contain more of the "right" info and less of what you do not finally want. The catch is that you cannot really change your mind THAT much in post.

So my answer starts on-set…

To illustrator the two ways, I'll call the one the "ETTR" way, and the other the "photographic" way.

When exposing ETTR primarily, you ensure image-integrity and levels available to your best knowledge. The image may not look too finished, but will be very flexible in post, as you have taken care for your lowlights - as to not exposing them too low, and exposing crucial parts of the image in a part of the expire where you have high precision and can gently pull them back where you want them in post.

For such an exposure, your main focus would be to cary all the image data through unharmed and trust the colorist to finnish it off in the intended way.

NOW. That sounds all nice and pretty.
Only catch is that you deliberately take a lot of info you already know you don't want into post, and that a post shaping of the image doesn't necessarily look as good as an image already photographed the way you want it…


Short anecdote to follow.
Last year I got to work with a really great and pretty famous photographer. And he loves to expose dark images. Or: Images that have really dark/black areas.
His AC was pointing out to him that that could be an issue for post. Nothing would look good down there. He responded along the line of:
WTF. I don't want those areas to be seen. I really want to ensure they're black.

So… There is a huge difference between a "low exposure" and an "underexposure"…
And there isn't really any "right" exposure, just "the right exposure for this particular shot".

And I am huge fan of this way of thinking…
Paint the image as far as you can on set with lights and shadows and colors WITHOUT touching the metadata in camera. Stay as close to what you know to be the center/neutral settings on the camera as you can and use lights, and filters to get the image you want.

Another anecdote from a feature shot on the original M.
There was a dark club-scene that was supposed to look rather gently blueish with tungsten lights.
The DoP had WB'd the cam to tungsten and set the lights and blue intensity for the monitors.
Only catch was that in post, the blue was noisy, and despite wanting to have a blue look to the scene and enough lights available, he ended up underexposing blue, because he white-ballanced the camera and lit for that. And there has been a few of these shots lugging around, that are totally unnecessarily compromised by "grading on camera" approaches…

So, if you can:
Have a vision.
Expose for that.

If you do that, you do not have to tweak the metadata so much to get where you want in the firstlight, and you'll have more of the info you need. You can still force yourself in the ETTR direction for technical reasons, but you try to focus not so much on that, as to the image you ultimately want to get home.

If you do that properly and have familiarized yourself with the camera before you take it to work, chances are that metadata will not mess up your online/color-grading.

The more radical your approach is, the more important is this for your images integrity.

So I think the best way to avoid problems with metadata is to shoot for the sensor and light for what you want. The best way to avoid problems is to not create them in the first place. That way, you'll have a real fat and relevant negative when you come home.

Others may disagree with me on this.




And now, I'll take the weekend off and continue to update this thread on Monday.

The plan is to through several ways of developing the files and look at their advantages or tradeoffs for different purposes.
But please download themm and run your own tests...

The RAW files and DPXs can be found here:
http://www.reduser.net/forum/showth...a-discussion&p=1001507&viewfull=1#post1001507



Have a nice one!

G

Thank your for your excellent post and time Gunliek. Invaluable. Have a lovely weekend off.

Best

Lliam