Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

RAW workflow - a discussion

Precisely

Precisely

I've always thought about ETTR as a signal to noise issue as Graeme states, and perhaps that's the only practical advantage.

This idea of using exposure choices to enhance precision seems attractive from a purely mathematical perspective. In practice, unless key scene values are so under exposed they start competing with the noise floor, I'm not seeing degradation with the naked eye. That said, if there was some increase in precision made possible by ETTR, that could improve compositing/post processing results. Perhaps the difference would still be too subtle to matter.

So if ETTR yields more real data in the bottom of the scale and enhances S/N ratio then is there any reason not to do it?

In some scenes where there are speculars, tons of top light in hair, etc I have taken the approach of leaving some of the right side of the histogram open hoping to capture a bit more image data I can curve into the processed image. As long as I can see even a hair of open space on the left side of the histo I figure I'm not paying a price so why not? The why not would be if I was actually moving the meaty part of the image into a part of the linear code value scale that robbed me of precision.

Based on Graeme's note, it looks like I can continue to use the latitude above my key values (skin tone, client's product, etc) for greater highlight protection even if that places them father left. Good to know.

Cheers - #19
 
It's all about SNR. Don't think about bit precision at all. The gotcha with ETTR is that some people will follow it to a rule and under-expose (insufficient SNR) the main body of the image to protect speculars that have no real detail in them.

Graeme
 
Graeme Nattress said:
It's all about SNR. Don't think about bit precision at all. The gotcha with ETTR is that some people will follow it to a rule and under-expose (insufficient SNR) the main body of the image to protect speculars that have no real detail in them.
Graeme
Click to expand...

If you mean that dropping the exposure because some speculars make the RAW stoplight show clip without regard to the scene values and proximity of key values to the noise floor, I agree.

To put my previous note in a more conversational way; as long as I have a bit of "elbow room" on the left so that I am out of the noise floor I might as well leave myself more potential to capture some detail and chroma in the high end. An example would be an exterior where the ground (sand for example) is fairly reflective so there is lots of natural fill and therefore no deep shadows - in that case, the only potential "lost" image information would be up high. Yes, HDRx might be a better way to have your cake and eat it too, but in most cases the extra media burn/fps limitations/compression limitations of the second stream aren't offset enough to justify it. That said, there are times when HDRx can enable really amazing imagery.

Cheers - #19
 
Graeme Nattress said:
It's all about SNR. Don't think about bit precision at all. The gotcha with ETTR is that some people will follow it to a rule and under-expose (insufficient SNR) the main body of the image to protect speculars that have no real detail in them.

Graeme
Click to expand...

LOL

This I have seen a lot. That's why I advocate photographers to be exactly that, Make images they want to see. ETTR is a tool one can apply as appropriate whenever a shot calls for it or it is a practical solution. When it becomes a weirdo "rule" that makes people expose images they don't really want to get them fixed in post, it is more of a hassle than anything else.

Like lights and flags, ETTR is part of your photographic toolset, not something that should allways be the practical way to do stuff..

Just my 2$

G
 
Graeme Nattress said:
The gotcha with ETTR is that some people will follow it to a rule and under-expose (insufficient SNR) the main body of the image to protect speculars that have no real detail in them.
Click to expand...

I think people doing that is a classic example of following a rule blindly without understanding the reasoning behind it. That is generally never a good idea.

It's always important to look at meter readings in the context of what the actual scene in front of the camera is. Something as crude as using your hand in front of the lens to mask out the sun or a bright specular when you see a clipping indicator can be very handy.
 
Nick Shaw said:
I think people doing that is a classic example of following a rule blindly without understanding the reasoning behind it. That is generally never a good idea.

It's always important to look at meter readings in the context of what the actual scene in front of the camera is. Something as crude as using your hand in front of the lens to mask out the sun or a bright specular when you see a clipping indicator can be very handy.
Click to expand...

Wow... :)

Flags and grad-filters are so underrated these days...

LOL
 
Gunleik Groven said:
Wow... :)

Flags and grad-filters are so underrated these days...

LOL
Click to expand...

To be absolutely clear, I do just mean temporarily putting your hand in front of the lens to confirm the clipping indicator disappears, as the clipping was caused by that particular thing ;-)
 
Nick Shaw said:
To be absolutely clear, I do just mean temporarily putting your hand in front of the lens to confirm the clipping indicator disappears, as the clipping was caused by that particular thing ;-)
Click to expand...

Perfectly understood...

I just amuse myself a bit on the notion that when entering digital high-end into photography, somehow the skills of photography aren't needed anymore, as it is so "rich/fat/RAW/whatever" and thus can be fixed at a later state...

While I look more at it like it's getting back to the main crafts.

I really like what happened with the audio-industry when going digital.
A (very) few new kids popped up with talent and knowledge to match the democratisation of toolsety, but the real good guys knows how to record audio before the "digital revolution" and still do and still the most credited engineers around (deliberately not saying "studios" here)

Shooting RAW is just getting back to photography on a very basic level for me. That's why I enjoy it so much.

The rude and unforgiving simplicity of taking an image... :)

The analogy about understanding your filmstock, is a good one, I think, even though a sensor and a stock work pretty differently...
 
Gunleik Groven said:
But what I'd like to ask first is for different definitions of "RAW" workflow.
Click to expand...

Formally the RAW workflow is related to what is called linearity in mathematics, and linearity means something is directly proportional to something else. For instance, when lifting weights, the work required to lift something to a certain height is directly proportional to mass. Double or triple the height, and the work is doubled/tripled as well and so on. The human eye is non-linear with respect to light. If the amount of light is doubled, the perceived brightness is not doubled.

Linear maps have the convenient feature that they can be "reversed", or more precisely , one can find an inverse map. If something is doubled, the inverse map is about multiplying by one half. This makes things handy with the CMOS sensor which is linear with respect to light; Open the lens by one stop or close by one stop and the signal is is doubled or halved, respectively. This implies, while shooting instead of matching the exposure (say the mid grays) precisely to the desired spot of the gamma curve it is enough to make sure the image is not clipped (or excessively underexposed to noise, see Graeme's comment above about ETTR). For, thanks to linearity, all pixels (or more precisely their color channel information) of a raw image scale in the same manner when the exposure coefficient is adjusted in post. In principle setting the exposure in post is the same as setting the iris of the lens.

Since the eye is not linear, one has to apply a gamma curve at some point. "Baking in" involves the idea one has already applied the gamma curve. This means, the pixels do not scale uniformly any longer. Say, one adds the exposure by 1/3 stop. The effect on some pixels is negligible while some others may change a lot. It's no longer the same as adjusting the iris of the lens by 1/3 stop. Accordingly, grading a "baked in" image becomes quite a lot more difficult. Consequently, it's a much better idea to retain "uniformity" as long as possible and to apply the gamma-curve as late as possible along the color grading chain. Especially so, as the gamma setting depends very much on the conditions on which the image is taken.
 
Lauri Kettunen said:
Formally the RAW workflow is related to what is called linearity in mathematics, and linearity means something is directly proportional to something else. For instance, when lifting weights, the work required to lift something to a certain height is directly proportional to mass. Double or triple the height, and the work is doubled/tripled as well and so on. The human eye is non-linear with respect to light. If the amount of light is doubled, the perceived brightness is not doubled.

Linear maps have the convenient feature that they can be "reversed", or more precisely , one can find an inverse map. If something is doubled, the inverse map is about multiplying by one half. This makes things handy with the CMOS sensor which is linear with respect to light; Open the lens by one stop or close by one stop and the signal is is doubled or halved, respectively. This implies, while shooting instead of matching the exposure (say the mid grays) precisely to the desired spot of the gamma curve it is enough to make sure the image is not clipped (or excessively underexposed to noise, see Graeme's comment above about ETTR). For, thanks to linearity, all pixels (or more precisely their color channel information) of a raw image scale in the same manner when the exposure coefficient is adjusted in post. In principle setting the exposure in post is the same as setting the iris of the lens.

Since the eye is not linear, one has to apply a gamma curve at some point. "Baking in" involves the idea one has already applied the gamma curve. This means, the pixels do not scale uniformly any longer. Say, one adds the exposure by 1/3 stop. The effect on some pixels is negligible while some others may change a lot. It's no longer the same as adjusting the iris of the lens by 1/3 stop. Accordingly, grading a "baked in" image becomes quite a lot more difficult. Consequently, it's a much better idea to retain "uniformity" as long as possible and to apply the gamma-curve as late as possible along the color grading chain. Especially so, as the gamma setting depends very much on the conditions on which the image is taken.
Click to expand...

I like this way of talking about it.
Lately I have started off with linearity as the base of explaining stuff for both photographers and post people.
What you do not cover in your description, is the relation between linearily mapped stops and mapped precission. THAT is where I see/think/feel the biggest difference between the R1 and trhe Epic.

Usually I start odd describing a sensor as a photon-weight, and each time you double the intensity (add a stop), those new intensities are mapped to a new bit with double the precission of the latter bit. The nice thing with the Epic, is that it has "stop overhead" compared to bits.

But what you describe is a linearlight signalchain, not neccesarily a RAW workflow.

I do definitely like this approach, though, and think it is much more significant than whether you choose to transcode or not...
 
Gunleik Groven said:
What you do not cover in your description, is the relation between linearily mapped stops and mapped precission. THAT is where I see/think/feel the biggest difference between the R1 and trhe Epic.
Click to expand...

Thanks to the wave-particle duality light can be understood either in terms of electromagnetic radiation or photons. In this context it does not matter which interpretation is employed.

Every stop doubles the amount of light, or equivalently one may say every stop doubles the amount of photons. Let the smallest amount light that can be recognized from noise be N photons. In this case the camera stores 1 (for this pixel). If one opens the lens by one stop, then the amount of photons is 2N. For, every stop doubles the amount of light. Accordingly, in this case the camera stores value 10. If you open the lens by another stop, the amount of light/photons is 4N, and in this case the camera stores value 100. So, whenever the lens is opened by one stop, the amount of light doubles, and a new bit is "added". (What I mean is, 1 is actually 0000...0001, 10 is 0000...0010, 100 is 0000...0100 and so on.) This is why at least as many bits are needed as is the dynamics of the camera measured in the amount of stops. Both go in the power of two. Or the over way around, this is the reason why it does not make sense to add a lot more bits than is the dynamics measured in amount of stops.

There is no difference in this respect --how the number of stops corresponds with the amount of bits-- between R1 and Epic. All cameras of this type share this property.

The linearity between the amount of light and the signal stored by the camera is very weakly if at all related to this. To understand this, one needs first to realize that linearity is a property of a mapping, and mappings are always from one set to another. In this case the domain ("input set") is the amount of light and the range ("output set") is the set of numbers representing the signals. Now, recall that the smallest amount of light that can be recognized from noise is N. If this amount of light is 2,3, 4, 5, 6, 7, 8-folded (that is 2N, 3N, ...., 8N) the output, or in other words the stored signal is 0000...0010, 0000...0011, 0000...0100, 0000...0101, 0000...0110, 0000...0111, 0000....1000, respectively.

In my understanding term "raw" means that this linear relationship between the stored values and the actual amount of light while capturing the image is retained. This is important because as soon as one introduces a non-linear mapping, such as the gamma-map, this global property of the image is lost. This is why in all color grading software systems, such as RDC-X, the linear mappings are applied first and the other ones only there after.

In the end of the day the question is also of the flexibility of the visual perception. Our visual perception can adjust itself in amazing amounts between darkness and brightness. For this reason an appropriate gamma-mapping is rather much a context dependent issue. If the gamma-mapping and the other nonlinear mappings are "baked in" in the camera, there will always be conditions in which these settings are not that good. Another issue is, the non-linear mappings compress some part of the signal and expand others. Since these values are processed and stored as digital numbers, it is not necessarily possible to decompress afterwards but instead some information is lost.This is why it's always better to leave such things into post where floating points can be exploited and one is in a better position to make the fine-tuning.

So in short the raw workflow is like leaving as much head space into the post and making a good use of the computing resource of the post system. The linearity of the sensor is a key ingredient. Another one is the color grading system which makes a good use of the linearity and uses floating numbers or a bit precision high enough.

This is the basic principle, but the whole thing can be implemented in different ways. Graeme already confirmed there is a difference between R1 and Epic. In addition, it's a good question, how raw are the raw files.
 
Gunleik, my pleasure. If the raw workflow has to be condensed into a small amount of words, I would say:

The raw workflow means while shooting one focuses in capturing light and while grading one focuses in adjusting the desired colors. The old system means, while shooting and while color grading, one worries both for the amount of light and for the colors. ;-)
 
Nice summing up :)

I guess I'll do some of what I planned with this anyway... But it so nice to get all these very knowledgeable inputs.
Love it. I am here to learn, and I do!
 
Just passed two hours reading that thread...Pffeeeww.....And wanted to say thank you to all the posters.
That thread deserve a bump aswell.
Cheers
B.
 
If it's a project that I care for, this does not happen to often. Most clients does not have the money time or interest. So most of times i do not bother.

Normally we get 10 bit DPX files form Resolve and finnish in Flame. This to me is not really the best way to get things done. Grading and online should be a combined process. And I find it quite amazing that people think that they can create a better grade in resolve then in flame. That is nonsense, I find flame superior to any grading toolset. There is simply so much more thing that you can do in a flame and the colors are all the same. This is something clients will get their head around soon but now still people thing the grading tools are sharper for grading than the compositing tools. Thats wrong. I think it comes from the time when we still used film and the clients DP's etc got amazed over how the colorist could tweak the colors in the scanner, now they are still quite amazed and they think it's the balls in the grading suite that's the trick... But if People do not leaner then I will not teach..

So I let them do their grading with the colorist, here in sweden it's depending on budget, if the budget is good they take the plane to london to get a "good grade" if they do not have that money then they do it at The Chimney Pot or Stockholm Post Production, if they do not have that money then they do it some where else... I actually think it's quite nice to get a disk with dpx files to do the online and efx from. the grading discussion can be a rather long one and most times i find it better to stay out of it, and do my composites etc and take the money and leave.

But then there are the projects I do care about, then I do this.

I bring in the DPX files from their resolve session with a XML. Then I use the same XML and reformat it in flame so the XML points at the r3d and conform from them as well.

Later in flame I make containers for each clip holding both the r3d and the HD 10bit DPX. Then I have all the options in the world, I can tell the flame to chew on the 5k r3d or look at the pre graded dpx files for any given clip. If I do green screen and such I can use the native source and make a key-inline pass that is from a different iso /whitbalance etc. I can also make sure I know that the highlights info in the graded shot is all thats there etc.

I can also use the flame color match functions to perfectly match and save the colorist grade. This I find quite amusing... it's like... thanks super expansive london colorist.... I just snagged your grade :)...

And while working as above Im quite supriced how sloppy they are in the grading sessions and how little they know about colors. Some try to grade bluscreen material and then to help me out they key back the blue in the screen so that I should be able to key it... thats nuts. They also zoom and crop and such because they think that it's better done in the grading toolset than in compositing... ( this is something left from the time when there was a film strip in the scanner, and people naturally did pan in scan to get as much out of the SD signal as possible) today it's a different story... zooming into and panning in a alexa frame before online only does one thing... it ruins it for 3D tracking...


So to me I think the future will be the following.
Grading will be advanced lut's and metadata that is forwarded to the online sessions together with the raw files.
Flame can do this allready, I only have flame 2012 so I did not get Gamma3 and such just yet. But I can actually choose the redlinexpro seting for each clip or camera seting or flame seting or make a mix of the 3 etc. And thats how it should be.

I would like to have resolve just creating r3d meta data. If there is advanced layering and such that should Im sorry be done in composting I think. But I guess some splines etc could go through in future metadata swell.


For me I get happy every time when I see a r3d node in one end of my compositing tree and an "final output node" in the other...

WE master all in 10bit uncompressed Quicktime files. So in the end we are not raw anyway. I think that is good enough, 16bit is only for things that later should go through extreme grading or keying processes.

that how we role. :)
 
Björn Benckert said:
Grading and online should be a combined process. And I find it quite amazing that people think that they can create a better grade in resolve then in flame. That is nonsense, I find flame superior to any grading toolset. There is simply so much more thing that you can do in a flame and the colors are all the same. This is something clients will get their head around soon but now still people thing the grading tools are sharper for grading than the compositing tools. Thats wrong. .....
Later in flame I make containers for each clip holding both the r3d and the HD 10bit DPX. Then I have all the options in the world, I can tell the flame to chew on the 5k r3d or look at the pre graded dpx files for any given clip. If I do green screen and such I can use the native source and make a key-inline pass that is from a different iso /whitbalance etc. I can also make sure I know that the highlights info in the graded shot is all thats there etc.
Click to expand...

Bjorn, I get the distinct impression that you work almost exclusively in short form material, i.e., commercials, music videos, etc., as opposed to longform material (i.e., television series, features, etc.). Because everything you've just said might apply to short form, but in long form, it's all about continuity (which you don't have nearly as accessible in Flame as you do in a grading program like Resolve or Baselight) and to some degree, speed. That's what grading programs are designed for. Flame is designed for tweaking individual shots to death, and manipulating multiple elements to create one finished image. In most long form color grading, the finished image already exists. It's the colorist's job to match scenes to achieve a smooth flow.

I can also use the flame color match functions to perfectly match and save the colorist grade. This I find quite amusing... it's like... thanks super expansive london colorist.... I just snagged your grade :)...
Click to expand...

It might be amusing, but I wouldn't lose sight of the fact that the London colorist came up with that look, you didn't. :sneaky2:

And while working as above Im quite supriced how sloppy they are in the grading sessions and how little they know about colors. Some try to grade bluscreen material and then to help me out they key back the blue in the screen so that I should be able to key it... thats nuts. They also zoom and crop and such because they think that it's better done in the grading toolset than in compositing...
Click to expand...

That doesn't make them ignorant about colors, it means they know less about compositing than they probably should. As for scaling, depending on the elements they have to start with, it might very well be better done in the grading session. For instance, if they're working with 4K or 5K R3D files but passing on HD or 2K DPX files to you, they have a lot more to work with and any resizing would by definition be a lot cleaner. And the scaling engines in both Resolve and Baselight (I'm not going to speak for Lustre, Nucoda, or others) are very, very high quality - equal to or superior to that of Flame. So in many cases, it IS better done in the grading toolset.

I'm not trying to read too much into your comments, but it does sound as if you're being a bit disrespectful of colorists in general. Everyone has a job to do. A good colorist brings just as much of themselves to the table as a great compositor. Much of their work is done in real time with the director/cinematographer/producer/associate producer/post supervisor present and collaborating with them to bring the proper mood and visual intent to the screen. That kind of real time involvement is not typically done in compositing sessions on long form material. But the truth is that if all the color grading on a show was given to the compositor, the result would very likely have a lot less continuity and would take 10 times longer. Everyone makes their own contribution, and denigrating the efforts of those who are, in fact, your collaborators is not very constructive. Perhaps talking to them about the compositing issues you face, and informing them of things they can either do or stop doing that would make your work go smoother would be a lot more productive. I have a great appreciation for both skill sets, as I've done both jobs, as well as spent time as a visual effects supervisor. They are not the same, and neither one exists in a vacuum.
 
Thanks Mike. I hoped you would join!

There are a couple of very different perspectives depending on POV.

Longform vs shortform

Compositor vs DP vs colorist

And from my POV in many situations (which means longform, BTW):
Keeping it all together and flowing smoothly.

What I like about this thread, is that most points are valid, even when contradicting eachother. Because they are valid depending on POV.

I like that!

Thanks to all who have contributed.
 
You must log in or register to reply here.
Back
Top