Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

RAW workflow - a discussion

Gunleik Groven

Gunleik Groven

REDuser Sponsor
Joined
Dec 28, 2006
Messages
13,465
Reaction score
5
Points
38
Location
Norway
Website
quine.no
Just thought this could be a productive thing to discuss...

There seems to be a bit of confusion to what a RAW workflow implies, and there are defintely different ways of doing things that can get you where you want.

This is not inteneded to be an Adobe vs FCP/whatever discussion, but more about what can give which results and about the signalchain.
I'll post examples as the discussion hopefully evolves...

But what I'd like to ask first is for different definitions of "RAW" workflow.
To me it mainly means that you don't do a transcode. It says very little about image integrity, as that can be messed up in both a RAW and a non-RAW workflow.

There seems, though - to be a bit of misunderstanding about how the metadataparameters affect the signalchain, from the "well it's raw so I have everything available" statements... Which is only partilally true, as that is totally dependent on how you set your metadata.

And I thought it could be fun - and possibly a bit enlightening - to discuss these issues.
Basically:
How does your metadata affect your image in a RAW workflow

and:
How does that differ from a transcoded workflow

And:
If one chooses to transcode, how and why would we do that for different scenarios.

Hope this is of some interest and that people can join in with examples.

I'd love examples posted to be in r3d single frames, and transcoded media of choice, not only jpg's, as that isn't very informative to the subject. If someone wants to post files, but don't have serverspace - I'll gladly host the files.

Just to be clear, what I want to discuss is different approaches to technical integrity of the image.
The aestethical values and choices, are just that. And cannot be measured in any way I know of.
I tend often to do rather harsh aesthetically choices... :)

Now.

Let the discussion begin!

I don't mind discussing different toolsets, I just don't want the standard Adobe vs the world discussion.



Cheers

Gunleik
 
It could be argued that all RED workflows are RAW workflows, since RAW refers to the original recording. In my experience grading is rarely, if ever, done using the RED SDK parameters.


Any "baking in" of settings in a way which cannot be undone with the grading toolset could be considered harmful to grading flexibility. And it does not matter if this baking in is done by transcoding to RGB files, or during the RAW decode into the RGB frame buffer of a grading system.


If the conversion to RGB is done without clipping or crushing any image data, and at a sufficient bit depth, I consider that it does not matter if this is done in the grading system, or in a separate transcode prior to grading. This is of course providing the conversion to RGB is done with consideration as to what SDK parameters are appropriate for each particular shot.


To me the best solution is conversion to 16-bit log DPX or unclamped float EXR by somebody who knows what they are doing, and then grading with those while still keeping the RAW files available so that a re-transcode can be done if required for individual shots.


I think I just basically said the same thing three times!


Everything I just said with reference to grading applies really to the entire finishing process, so applies equally to VFX.
 
I happen to agree with all three statements.
Which probably means I think Nick is a very wise man... :)

But I have seen others disagree with this.
I think it could be practically informative with some examle-images put through different approaches...



Whiteballancing of the image is also part of the soup...
 
I have a question which I've been testing but am failing to really understand. Take this situation.....I shoot at 800 ISO, protecting some highlights then drop back to 320 ISO in RCX and use curves (and Red Log Film) to balance the contrast of the image...is this a bad practice? Useless? Perfect? And once that is done, then I would head into DaVinci for grading and refining of the look. I want to know if this is good practice or if there alternatives that are better and why.
 
Nick shaw had kindly cleared up some questions i had about the red raw workflow in post. and as you both agree that setting the white balance is an important first step and then after that transcode for use in a more 'comprehensive?' grading program such as davinci, then what is the real use of all the other RAW parameters? it is interesting that you say that the technical integraty of the image be discussed and not asthetical. personally what draw me to red was the Raw parameters i found them more intuitive to understand to do a grade as the parameters are like that which you would set before shooting.shortly i will post some transcoded images and my process for discussion
 
I vote for RAW all the way!
Having the RAW settings available through all the grading sessions is a must I think.
In the real world you get shots that without a WB tweak you will not be able to achieve.
Some footage that was shot recently with RG2 and RC2 get an over all bust when going RG3 and RC3, and you want to discuss this decisions with the DP on the grading session and not as apre decided workflow.
 
@ Mathew..

That can be a good approach. You do not destroy your image that way, as ISO is created through floating point curves operations.

You might (or may not) find it as easy to go to redlogfilm and grade from there, as that gamma (cineon) is created to protect both the highs and lows with headroom for grading both ways.

If you go for direct to delivery, your approach is as god as the results you see. Which can be excellent!



@ Louis
As to whiteballancing.
Usually it is a good idea to whiteballance the image. Especially for quick turnarounds. Whiteballancing can in some situations compromise the signal in an unwanted way though, so if you have an image that doesn't perform as you want it to, that could be a place to look for solutions. But generally and for most applications, a WB is a good thing to do.


@ Ido
As to RAW all the way... Well it isn't whatever way you turn it.

I agree that having access to the metadata is a good thing, if you know what you do, which you obviously do Ido.
But in larger operations with specialists doing editing/VFX/grading, consistency of the signal is an important thing - and everyone do not know what happens with the signal when seemingly "undangerous" metadata is altered, in that case what Nick describes is the foolproof way.
Also, editing RAW isn't really practical in a few scenarios I know off, both from a dataintegrity, competence and quick turnaround perspective.

In smaller operations, I can see it has a better value.

Conforming to RAW for grading is good, if you get a proper signal out of the RGB buffer in the grading app, but that is easilly messed up, and can thus give unwanted results.

The signal that is graded is RGB/YUV whichecver way you see it and does perform exactly like what it is. No magic soup there.

But I think image-examples will be neccesary to come along the statements here...



Cheers!

G
 
Ido Karilla said:
In the real world you get shots that without a WB tweak you will not be able to achieve.
Some footage that was shot recently with RG2 and RC2 get an over all bust when going RG3 and RC3, and you want to discuss this decisions with the DP on the grading session and not as apre decided workflow.
Click to expand...

That is why I advocate having the RAW files available to go back into for particular problem shots.

Things like the the introduction of REDcolor3 are also IMHO an argument in favour of transcoding, as it takes a while for new colour science to become available in grading systems, whereas using REDCINE-X to transcode allows you to use the latest colour science (if appropriate).
 
The way I see it its best to start with red gamma 3, red colour3 . It's my assumption that the guys at red have done much for testing and image scrutinizing than you are I. They would have maximise the amount of DR possible from the sensor whilst maintaining a pleasing contrast ratio.

I like editing raw because I like to change the exposure of an image using the iso setting without degrading the image unless I go beyond iso800. I also like being able to change the WB.

I do find myself spending a lot of time playing with looks in RCX when I probably could spend that time on the edit.I use adobe so I stay in raw until the final export but I do wonder if my footage would look better if I rendered out of RCX using the extra tools on hand.

All in all I like shooting red raw but trust redgamma 3 is doing a very good job.
 
I come from a motion picture background and mindset and much of the allure of the raw workflow has come from that.


Editing Perspective:

The very first advantage for me with RedCode Raw in particular is the ability to edit without transcoding.

This for me speeds up the editing workflow, but others may feel differently. I still work with studios that transcode to HD proxies to view on other devices and what not. There are certainly reasons to transcode early in the pipeline and that's one of the best. Sharing smaller dailies through ftp, emails, drives, etc.

Editing and making selects for me happens much quicker on the front end and I only transcode what's necessary.

I've worked on productions that have shot between 1-3 million feet of film. That's landing in the 200-550 hours of material. Some teams can make features out of 60 hours or less. Whatever style of shooting and type of content you're trying to capture I certainly don't want to transcode 60-550 hours of material if I can avoid it. I'd like to transcode enough for the rough edit which often lands in the 6-17 hour range.

Disk space still costs money and in this cast even with several Red Rockets, time. I suggest to most to review dailies and edit in raw whenever it's possible. These days though proxies can be made on set pretty damn quickly. So my dailies opinion is changing on that front. Depends on the project though.


Colorist Perspective:

The raw workflow is the friend of digital colorists everywhere. Most high end suites support RedCode Raw natively at this point and you can certainly get the most out of the image through that medium. I have worked with studios that prefer transcoding to DPX, EXR, TIF, and proprietary formats to help with their pipelines. When dealing with large scale data management and many assets I can understand that. RedCine-X Pro exports and that nifty command line converter usually are tools of choice in those situations.

The flexibility of working natively in raw can't be understated. Shooting color charts, white balancing, tweeking your ISO, etc... That's a given. Color tools have gotten better and better at dealing with material from "not so good" sources though.

I remember back in 2004 I had a shot that we scanned in that was lit one way, but I literally had to sweep the color with an extreme grade for one sequence. We were working with 4k uncompressed 16 bit scans from a Super35 source and this was possible. But that's film and uncompressed scanning in this case. Minus a source like that, if for instance we were dealing with a lossy codec, this is exactly the type of grade that would be very difficult to make look good.

As we get into fatter and fatter digital negatives it gets even better for colorists.


True Control

The big selling point with raw in general is that you have the choice where you want to transcode, if you even need to. That's the real power behind the raw workflow. Choice and control. The raw file is always there to go back to when you need it. Redcode Raw is like a good friend. It's always there when you need them the most.

While we wait on 3rd Party support for RedColor3 and RedGamma3 there's nothing stopping anybody from completing your edit using RC2/RG2 then transcoding and relinking back into your color suite or project with newer color science processing exported through RC-X Pro.
 
scott william said:
The way I see it its best to start with red gamma 3, red colour3 . It's my assumption that the guys at red have done much for testing and image scrutinizing than you are I. They would have maximise the amount of DR possible from the sensor whilst maintaining a pleasing contrast ratio.

I like editing raw because I like to change the exposure of an image using the iso setting without degrading the image unless I go beyond iso800. I also like being able to change the WB.

I do find myself spending a lot of time playing with looks in RCX when I probably could spend that time on the edit.I use adobe so I stay in raw until the final export but I do wonder if my footage would look better if I rendered out of RCX using the extra tools on hand.

All in all I like shooting red raw but trust redgamma 3 is doing a very good job.
Click to expand...


Good stuff!
All the power to you, and I am all for everything that gives the result you want!

A thing to remember, though - is that - "RAW" (which it isn't really) or not (which it is) - that you cannot pull anything out of the blacks in a grading session, if you have crushed them with RG3. Even when grading "RAW", as the signal is allready down there. Or if you can pull them up, they'll probably be a lot more noisy than if you started with a fuller signal.

It doesn't help that the color tool is working in "float" or whatever, if the signal it is fed is not there to be treated.

Thing to remember is that it is all about choices, and choices are good. But every choice you do in a digital signalchain is destructive. You weigh your data and throw out what you don't want, and keep what you want.
For a lot of people in a lot of situation, that pre-weighing (like RG3) is exactly what you want.

What this thread is about, is ways to work with an image when that isn't exactly what you want, or when an image behaves "badly" or unexpectedly. In those scenarious it can be good to understand what goes on under the hood in the signalchain.
 
@ Phil.

Agreed on all points. Mostly. As to editing, I still think that depends on the environment one is working (not software).
One cannot expect that everyone knows how to treat the signal in all positions, but you can still be extremely good in dramturgy as an editor.
I'd take a good storytelling editor over one who can tweek metadata the right way anyday.

As to colors and VFX, I am a bit hung. But as long as you ensure consistency, that's all fine, one way or the other.

Point that you obviously understand - comming from film - that seems to be overlooked a bit by a few others, is that the signal we grade actually isn't RAW in any way, and neds the same care for being optimum as it does when transcoded to DPXs. And if you do not posess the machine capacity to do that in realtime at full quality, a transcode might still be a good option for speeding up things, securing consistency and keeping integrity of the images in place.

But what I more want to dive into, is what happens with the image with different metadatasettings.
It is not so much about diskspace saved...
 
Since my workflow 90% of the time involves either adding VFX or slight noise reduction or lens correction before grading... I always find myself going to RGB earlier than RED advocates. Personally I think that's how everyone working on big projects does it too.

Phil Holland said:
I remember back in 2004 I had a shot that we scanned in that was lit one way, but I literally had to sweep the color with an extreme grade for one sequence. We were working with 4k uncompressed 16 bit scans from a Super35 source and this was possible. But that's film and uncompressed scanning in this case. Minus a source like that, if for instance we were dealing with a lossy codec, this is exactly the type of grade that would be very difficult to make look good.
Click to expand...

Well, that's the real thing for me: RGB files, if correctly made, can withstand massive amounts of grading changes.

After all, you have to consider that the camera's sensor is basically an intensity-sensing device with R, G, B color filters. If those color filters do a good job and the color primaries are vaguely approximate to the primaries of your RGB file, you really shouldn't lose color info, right? The only place you'd lose color info would be if the incorrect gains on RGB cause the DeBayer algorithm to do odd things, maybe?

The question is: why is it so easy on RED to create bad RGB files if the R3D development settings aren't right?

Why is setting the white balance right in the RAW file so important?

I suspect it is because the color filters on the M and MX sensors are a little... odd. They filter some wavelengths too much and others to little, or something like that - and there is crosstalk / missing color info. Perhaps this was a compromise to reduce noise? I don't know. Anyway, to me, natively they don't work quite right in conjunction with certain light sources and subject matter. I suspect that Graeme or someone is doing very smart compensation in the R3D development process to fix for this. So this is why RED is advocating staying in RAW so much - because they know that, more than other cameras, if you get the white balance wrong, you kill detail.

So personally I try to take care to get the white balance and tint right because it seems oddly sensitive in this regard.

However in other cases I prefer to just decode at 5000 Kelvin and fix the color balance in the grade. I really need to sit down and figure out if it's just my imagination but I think you can really get better results that way sometimes.

Bruce Allen
www.boacinema.com
 
@ Bruce, as to WB... I Have found that you get fairly undistorted images in the WB realm of 4400 - and 6000. I haven't managed to see any baddies introduced through tint.
So I do agree. WB is a bit of a choice of approaches.

I can def get less noise by staying more centered with WB and ballance it in RGB land.
BUT that is probably not a good GENERAL advice.

Like 800 and WB is a good shooting GENERAL advice, I can see why people choose to shoot 320/daylight.
 
At some point in the finishing chain you have to move from camera raw space to the target display space format for DR and color gamut. Regardless of encoding bit depth or format, display spaces are more limited in both DR and color gamut compared to your raw camera data: typically about 7-8 stops linear DR at best and whatever the defined standard gamut for the encoding format is. The most critical metadata elements that have to be nailed in the debayer process are middle grey point, white balance, and gamma curve to match intended scene values to the display space. You have much more latitude to manipulate these in camera raw space than downstream.

If you are handing your work off to a colorist used to whatever software and workflow standard, accommodate his requirements. A good colorist is a fine artist. Artists become highly skilled at working in particular mediums. You don't want to hand the guy a palette of acrylics when he prefers painting with oils. A color monitor calibrated and matched to the target display format is essential for best results when you establish your debayer metadata settings.
 
Basically Red puts the need of RGB transcoding from the camera to the post production where usually more computing power is available. This, and the ability to compress the RAW files makes Red's approach unique and unquestionable clever. But there are pitfalls.

I would say that at least 50% of the DOPs that use a Red camera now and then don't understand the meta-data workflow entirely, let alone the meaning of RAW and something that even Red has problems with to describe: native sensitivity to light and colour temperature. In the good old days of film that was easy to understand, just reed the notice on the can and shoot accordingly. On the Epic there is nothing written on such as 320 ASA, 5000K.

On the post production side supervisors hate to take responsibility for the footage's equality. Historically that has been the responsibility of the DOP. That changed with Red's workflow which makes transcoding at some point necessary, either in the beginning or in the end. All post houses I know favour Alexa, simply because the responsibility is laying soley on the DOP's shoulders, it's RGB, no metadata, etc... She/he shoots, they edit and finish. Clear and simple.

What I do? I transcode all the stuff I shoot by myself before things go into post production. Usually 4KHD. The settings: 320 ASA, 5000K, no WB, Half Good. This way I never lost any information in the highlights in the transcoding process and I don't kill moods with WB. It actually is that simple: just dial in the settings of the "native" sensitivity, regardless at which ISO you use for exposure and off you go. There is hardly something like "power of RAW" - if correctly transcoded.

IMHO, a simple RedCineLight, basically an applet, would do the trick for most users and take away complexity no one really needs. RedLog, RedLogFilm? What? Just offer RedColour3, RedGamma 3 and RedLogFilm for the Alexa/DR lovers.

I think that RAW is overrated. It has been great in the beginning when every second month a new colour science improved the quality vastly. But now? Red would attract many producers and DOPs if there were a ProRes 1080p module.

Hans
 
Nick Shaw said:
To me the best solution is conversion to 16-bit log DPX or unclamped float EXR by somebody who knows what they are doing, and then grading with those while still keeping the RAW files available so that a re-transcode can be done if required for individual shots.
Click to expand...
I too think this is a good way to do it. Not to say keeping it RAW 'till the end is a bad way, but especially if you're hampered by the inability to do realtime playback when staying RAW, there's no major problem in doing things like Nick suggests. The key here being that the conversion is done by somebody who knows what they are doing.
 
OK. This discussion could go in two directions.

1. is about the dataspace and efficiency gains by staying all RAW.
Even though I think that the efficiency gains are not equal for all situations, that is not where I want this to develop.
It's interesting, but from a more datacentric standpoint. I am interested in a more image-centric standpoint for this particular discussion.

Also we have to make a distinction between direct to delivery production, basically NOT going through a full Color/VFX chain and productions that do.

For a lot of productions I fully think that "if it looks good in the monitor and it looks good in RCX/adobe/whatever as it is - that is good and do your direct delivery".
I am all for not complicating stuff when that is not a neccesity.
I advocate such workflows a lot these days. RED can be incredibly simple to work with compared to any cam I know of using this approach. But that is also not where I want this discussion headed...

Where I wanna go is:
1. What happens with the image under different metadatasettings
2. how do you keep the integrity of all/most data when that is adviceable
3. How do you keep consistency throughout a larger post path with several dedicated artists involved
4. Do you really loose anything but diskspace through a transcoding path

Somehow it has been advocated that as long as you stay RAW, you have all the flexibility in the world.
My point is more or less:
Not neccesarily.

Can transcodes be destructive?
Sure

Can staying "RAW" be destructive
As possible, as what we see and work with actually allways is transcoded.
No magic
No soup there
Just RGB

But I am open to be convinced I am wrong... as long as it can be documented... :)

Cheers

G
 
David Rasberry said:
Regardless of encoding bit depth or format, display spaces are more limited in both DR and color gamut compared to your raw camera data: typically about 7-8 stops linear DR at best and whatever the defined standard gamut for the encoding format is. The most critical metadata elements that have to be nailed in the debayer process are middle grey point, white balance, and gamma curve to match intended scene values to the display space. You have much more latitude to manipulate these in camera raw space than downstream.
Click to expand...

That would only be the case if you are working in your display space in a gamma coded integer form intended for direct display, such as REDgamma3. If you work log or unclamped float linear with a display/output LUT, you are not constrained to the 0-1 range of your display space's primaries.

That is to say, just because your display can only show 7-8 stops you don't need to constrain the material you work with to that range.
 
Nick Shaw said:
That would only be the case if you are working in your display space in a gamma coded integer form intended for direct display, such as REDgamma3. If you work log or unclamped float linear with a display/output LUT, you are not constrained to the 0-1 range of your display space's primaries.

That is to say, just because your display can only show 7-8 stops you don't need to constrain the material you work with to that range.
Click to expand...

As long as you are working in a linear space this is true. When you move to a log space you are in fact applying a compression curve to the data. The full value DR range is represented in a smaller DR space with decreasing value range between each step as you approach the upper and lower limits.
 
You must log in or register to reply here.
Back
Top