RAW workflow - a discussion

[Phil Holland]

Agreed on all points. Mostly. As to editing, I still think that depends on the environment one is working (not software).

As to colors and VFX, I am a bit hung. But as long as you ensure consistency, that's all fine, one way or the other.

Point that you obviously understand - comming from film - that seems to be overlooked a bit by a few others, is that the signal we grade actually isn't RAW in any way, and neds the same care for being optimum as it does when transcoded to DPXs. And if you do not posess the machine capacity to do that in realtime at full quality, a transcode might still be a good option for speeding up things, securing consistency and keeping integrity of the images in place.

Exactly. Every job is different and most houses are different. And on top of that there are variances still on whether a colorist wants RedGamma and RedLogFilm depending on how the media is going to be used. We have all of these in the .r3d, which is very nice.

When it comes to the grade I agree, but I still think there's advantages working with native .r3ds in the grading suite or even doing a first light in RC-X Pro and then transcoding. On Baselight, and likely Speedgrade, I do my first and secondaries in the suite on the .r3d It's having the flexibility to make the raw workflow work for you that I like. There isn't just one way to do it. Transcoding can be done whenever you feel it's necessary.

The question is: why is it so easy on RED to create bad RGB files if the R3D development settings aren't right?

Why is setting the white balance right in the RAW file so important?

@ Bruce, as to WB... I Have found that you get fairly undistorted images in the WB realm of 4400 - and 6000. I haven't managed to see any baddies introduced through tint.

The sensor is rated at approx 5000k and I've found through using optical filtration under tungsten to bring it up into the 4400-6000 range that Gunleik mentions you can indeed produce a superior image. Especially where color noise is concerned. That said, without filtration I haven't seen an issue of getting the WB right or wrong in camera (yet). If for some reason I shot 3200k without blue filtration and if it's a low light shot I don't mind doing a bit of noise reduction to the shot.

I notice that native sensor kelvin effects pretty much all CMOS based tech in the same way.

Noise reduction is where the real difference comes into play between Red and other folks. They leave it up to you to get it done. For me this is fine. For many others it's not. I could totally see Dragon having some sort of general on chip reduction, but it's certainly not something that I feel is necessary.

 

[Blair S. Paulsen]

Digression alert

Digression alert

I think the concept of RedCine "Light" is an interesting one. Mike Most and others have pointed out that the power of RAW is desirable in theory, but in many instances it just makes it far too easy to screw things up. For people like Gunliek and most of us reading a thread like this on "RedUser", the power of RAW would have to be pried from our cold dead fingers, but as Peter Parker learned the hard way...

I would suggest that a version of RedCine (or maybe just a basic/advanced mode toggle) that was as simple as possible might really help the legions of folks who just want a "pretty picture". What if all the user had to do was put an eyedropper target on something white/gray, click on some skin tone in the image if there is any and hit "enhance" or "alchemy" if they want a bit more "pop" and then transcode using a list of pre-sets aimed at the 20 most popular formats.

Cheers - #19

 

[Gunleik Groven]

I have made a little preset for our colorists getting lowbudget quick turnaround RED shots in from non-drama/promo parts of the house.

RedLogFilm/RC2 (for now)/5000k/640ISO

Works allmost shamefully well for an incredible amount of shots.

Not perfect for every, but "good enough" for more than most. Allmost embarrasingly so...

Even though a proper meta firstlight outshines this, that setting kind of works "very well" for "allmost all" shots out of the box.

And gives the colorists all the RGB fun they don't have with any other camera around here...

As a by-the-by and a bit OT to this thread, we looked at how efficiently Resolve could key out IR pollution from most of the shots we tried it on.
Quick key and desat cleaned up a lot.

This was a bit of a "for dummies" approach, but that can be rather effective.
Stuff doesn't HAVE TO be hard...

 

[Gunleik Groven]

As a BTW. I do firstlighs in RCX-P (whether the shots are to be transcoded or not)
Trying to integrate with Hiero/Nuke, but do all the color tweeking in RCX. The interface is just getting good these days and the conforming capacity is only improving.

But if resolve could have a similar "metagrade/firstlight" window to its color window with controll over all SDK parameter it internally supports, I'd probably stay in resolve all the way...

 

[Blair S. Paulsen]

Metadata nuances

Metadata nuances

This subject can get deep pretty fast so I will just offer one approach that has worked for me on 75% of the material I deal with.

Shoot with the ISO at 800 and the WB at either 3200 or 5600. Move the iris up and down the scale while watching the histogram and look for an exposure that spreads out the values. Use other tools to see if that exposure holds up.

When I open the image in RCX Pro or Scratch I set the WB and tint first, then I try different ISO settings until the mid-tones are just a little low. (Most of the time I end up at 320 to 500 depending on the scene values.) Then I use the curve to emphasize or de-emphasize parts of the tonal range to get the image I desire.

A fairly standard starting point for me is to "stub the toe" at the bottom just enough to find a good black level (never more than a smidgen of crush), move the mid point up until skin tones or other key values seem right, then make a big broad shoulder in the highlights. This creates a curve that is steeper in the 7-8 stops that make up 95% of what can be seen on an actual display in the real world which typically helps separation and contrast, which by extension helps perceived sharpness without adding any sharpening that might bring unwanted artifacts.

Obviously the "secret sauce" is in knowing when and how to deviate from this starting point "formula" to best service the particular image or creative intent at hand.

I further stipulate that this approach is going to throw some information away, but if done well, not much of anything useful. It also allows me to deliver the intended look into the post pipeline in a way that suits the proscribed workflow. If there is going to be a proper final color pass then I keep my curve adjustments more subtle and render out to DPX or OpenEXR in RC3/RLF. If the time and budget for color is limited or nonexistent then I work the curves a bit more aggressively and suggest delivering in a format that can be both offline and hero media - ProResHQ, DNxHD175, etc. For these more "direct to finish" workflows I suggest RC3/RG3 but be careful with RG3; its great on a lot of stuff, but for some images the particular shape of the RG3 gamma curve does not interact well with the specific image at hand, or more critically, applies greater "pressure" on a part of the curve that is weak to begin with. This is not a "red flag" warning, even those images that suffer a little from the RG3 profile will still look pretty close to as good as they ever could have. Plus, most well exposed material looks better with the RG3 gamma curve than RG2 or other options.

Not sure if I have properly followed one of Gunleik's paths of discussion, but I hope I offered a meaningful data point.

Cheers - #19

 

[Gunleik Groven]

Thanks Blair.

No Cencorship I know of here, and I think I can follow your approach.

I was planning to post some images here with R3d and different development settings and analysis, but if this thread is only read by those who are comfortable with these things... It's not much of a ppoint.

Just planning to take one over and one underexposed Tungsten shot.

Is that helpfull?

At least for people to take a shot at and discuss approaches?

Would in case be a parallell Epic/R1MX shot in 4khd. Could also get into discussing a bit of the differences between them...

Cheers

G

 

[Ido Karilla]

@ Bruce, as to WB... I Have found that you get fairly undistorted images in the WB realm of 4400 - and 6000. I haven't managed to see any baddies introduced through tint.
So I do agree. WB is a bit of a choice of approaches.

I can def get less noise by staying more centered with WB and ballance it in RGB land.
BUT that is probably not a good GENERAL advice.

Like 800 and WB is a good shooting GENERAL advice, I can see why people choose to shoot 320/daylight.

I will give you a fresh example of miss use of tint.

A production came to me today with Epic footage that seemed very noisy to them.

It was a studio shot of a stage with windows and a green screen behind.

Every thing was lit properly, never the less the stage gray walls where moving around with noise.

I reset the metadata dialed on set and a full green cast was over everything, on set they tweaked the tint so much that the actors looked fine since they where in a uniform light. but the dirty gray walls on the other hand became noise from color oscillations not well maintained by this tweak.

So with a two pass raw grading and some simple composting the shot was saved.

This are the real life situations that small budgets, not so informed crews bring to our desk on daily basis.

 

[Erich Ocean]

People seem to think of a transcode as a bad thing. It's not.

A properly done transcode is equivalent to caching the debayered output with whatever metadata settings you chose.

Since you rarely, and I mean rarely, change the RAW metadata settings after you've done your one-light, the transcode is effectively a way to speed up everything later by caching the results of those decisions.

Put another way, you never, ever lose anything by doing a transcode -- it's just a cache. If you later decide you chose the wrong development parameters originally, you'd go back to the RAW file from the camera, pick new paramaters, and do a new transcode. (You may have to adjust some creative decisions down the line as a result.) This is the value of shooting "RAW" – nothing more.

If you think of the transcode as a cache, you'll see why it's still a "RAW" workflow all the way down.

There's nothing "cool", "better" or "smart" about repeatedly regenerating that cache because you choose not to transcode your RAW file. You're basically just wasting time and energy (literally). You keep doing a transcode, and then throwing away the results, only to move the playhead over that frame and do it again!

There is no value in not generating the transcode cache up front. None. The only thing you could possibly gain is reduced storage space (by not having to store the transcode cache). But storage is cheap, and processor and your own time are not. The cost of storage is literally the only possible argument against not generating the transcode cache, and it's a seriously weak argument.

(Transcode time is not an argument, because the transcode always happens regardless of whether or not you write out the cached files to disk or not. You can't get away from it.)

 

[Gunleik Groven]

@ Ido
Yes... I didn't think of that possibility when I came with my tint remark. I generally try to make people shoot as close to native as they can and never tweak the camera like that on-set.
I see how tint can be destructive when used like that.

@ Erich
Kind of one of the main points of this thread.
Thanks for summing it up so consistently. In a multi-editor/VFX-artist/colorist environment, I'd say that a full transcode with proper meta-settings is a very rational and effective AND lossless way of treating the RAW images.
It just isn't rational here to buy in the ballpark of 20 Rockets to serve (allmost) the same purpose....

(allthough we do go without transcodes when that is a rational/practical solution...)

I think it is good to look a bit at the

RAW = magic
RGB = messed up

statements, as both are false in themselves and really not a very interesting way of discussing RED post IMHO...

 

[Ido Karilla]

For us as a small facility doing mainly feature films, with 2 colorists and one technician the choice of staying RAW is more of an artistic choice.

We feel that having the raw at hand let you experiment with the DP and director more, it is much easier to offer them different approaches to the scene discussed.

I find kelvin a much more sensible and intuitive when following changes in time of day. It can be achieved in RGB but there is something much more subtle when you are defying thing by changing the kelvin by hundreds. It is also a great tool when matching sides that where shot outside at the end of the day with a half an hour between them.

Just my 2 cents.

 

[Jaakko Rinne]

I was planning to post some images here with R3d and different development settings and analysis, but if this thread is only read by those who are comfortable with these things... It's not much of a ppoint.

Just planning to take one over and one underexposed Tungsten shot.

Is that helpfull?

At least for people to take a shot at and discuss approaches?

Would in case be a parallell Epic/R1MX shot in 4khd. Could also get into discussing a bit of the differences between them...

Please do post them. Case studies are always interesting.

 

[shashbugu]

Basically Red puts the need of RGB transcoding from the camera to the post production where usually more computing power is available. This, and the ability to compress the RAW files makes Red's approach unique and unquestionable clever. But there are pitfalls.

I would say that at least 50% of the DOPs that use a Red camera now and then don't understand the meta-data workflow entirely, let alone the meaning of RAW and something that even Red has problems with to describe: native sensitivity to light and colour temperature. In the good old days of film that was easy to understand, just reed the notice on the can and shoot accordingly. On the Epic there is nothing written on such as 320 ASA, 5000K.

On the post production side supervisors hate to take responsibility for the footage's equality. Historically that has been the responsibility of the DOP. That changed with Red's workflow which makes transcoding at some point necessary, either in the beginning or in the end. All post houses I know favour Alexa, simply because the responsibility is laying soley on the DOP's shoulders, it's RGB, no metadata, etc... She/he shoots, they edit and finish. Clear and simple.

What I do? I transcode all the stuff I shoot by myself before things go into post production. Usually 4KHD. The settings: 320 ASA, 5000K, no WB, Half Good. This way I never lost any information in the highlights in the transcoding process and I don't kill moods with WB. It actually is that simple: just dial in the settings of the "native" sensitivity, regardless at which ISO you use for exposure and off you go. There is hardly something like "power of RAW" - if correctly transcoded.

IMHO, a simple RedCineLight, basically an applet, would do the trick for most users and take away complexity no one really needs. RedLog, RedLogFilm? What? Just offer RedColour3, RedGamma 3 and RedLogFilm for the Alexa/DR lovers.

I think that RAW is overrated. It has been great in the beginning when every second month a new colour science improved the quality vastly. But now? Red would attract many producers and DOPs if there were a ProRes 1080p module.

Hans

I like the Applet idea.

 

[Gunleik Groven]

I like the Applet idea.

It's nice... Could do with a couple of foolproof presets, though...

 

[shashbugu]

It's nice... Could do with a couple of foolproof presets, though...

Honestly speaking 90% of the tweaks I've ever done in RCX are always thrown out in favor of a smoke or resolve grade. Redlogfilm, white balance, RGB, temperature and ASA are the only controls ever really used, and we actually tweak those in PPro cs5.5. So yes a redcine light with lots of presets, kinda like the settings for raw in speedgrade will go a long way. In a sense you wont even have to do much. Now that speedgrade is shipping with PProCS6, white balance and debayering could be done at the grading level.

 

[Graeme Nattress]

I suspect it is because the color filters on the M and MX sensors are a little... odd. They filter some wavelengths too much and others to little, or something like that - and there is crosstalk / missing color info. Perhaps this was a compromise to reduce noise? I don't know. Anyway, to me, natively they don't work quite right in conjunction with certain light sources and subject matter. I suspect that Graeme or someone is doing very smart compensation in the R3D development process to fix for this. So this is why RED is advocating staying in RAW so much - because they know that, more than other cameras, if you get the white balance wrong, you kill detail./QUOTE]

Nope, not odd. Go take a look at how Adobe do raw conversion colorimetry for DSLRs - they profile at A and D65 if I remember rightly, and then interpolate on the mirek of the white balance between those two for any other colour temperature. In other words, rather than profile the colorimetry for every possible colour temp, the main ones are profiled and the rest are interpolated. We work similarly but take more profiles and less interpolation. That is why setting white balance is important. Of course, white balance only effect colorimetry, which is the conversion of the rather ill-defined camera raw space (ill defined because no camera is colorimetric device) to XYZ, and from there to any defined colour space of choice.

RAW is advocated because it keeps decisions as live as possible for as long as possible. It's like the difference between the old days of Photoshop when every levels operation you applied was burned in, compared to today where we have live layer effects keeping that decision alterable until you bake off a file for print. Or how RAW has been very successful in the photographic world. As Jim and I very much enjoy the RAW workflow with stills, we worked hard to figure how to do the same for motion.

Graeme

 

[Gunleik Groven]

@ Graeme
I do actually appreachiate that a lot, and now - for a short while - derailing my own discussion...

There are two issues really

1. I work in a rather largish organisation (in Norwegian context, it is kinda the largest as to moving image production...) where we strive to replace inferior solutions (read: "DSLR crap") with better, yet cost-effective solutions.
And I happen to be in the "well but RED could be that as well as the C300" camp.
To prove that point, I have to make simple foolproof workflows.
The conform is not much of an issue these days. Happily.
Colormanagmement is.

So what I tell the aspiring S35 photographers is: Make a beautifull image @ 800 ISO with in-camera WB and the contrast you'd like to see, and chances are pretty good that it will look good after post.
Don't fiddle with anything else, and your chances of a lovely result are pretty high...

So... These productions go out more or less "as is", (simple transcode to edit in RCX RG3/RC3, OR quite a few of them go through some kind of colorcorrection.
Thus: Simple preset, that has to take images from R1 M/MX and Epic - no scarlets yet.

And thing is: It works.

I know the flexibility you have built into the system, but I have to make more or less foolproof solutions for a productionchain delivering content to 3 national broadcast channels. And by dumbing it down like this, RED is getting some traction outside Drama/promo. The alternative is C300/DSLR, some XDCAM. Obviously the users are happy....

2. At some point you'll have to lock down for consistency.
I am currently (and as I write) working on a 24 episode show. We'll be in post on that until November - give or take. Episodes are edited while others are finnished. They shall look the same. There are several colorists, editors, VFX artists etc involved. Thus I lock anything critical into fixed RGB formats (EXRs mostly, some 16-bit DPXs bouncing around, too) with strict parameters as to how they are treated colorwise in VFX.

That way The graders can grade the VFX shots (mostly) long before they get them, and the result will still be accurate and consistent (without major tweaks after VFX delivery).

As Erich pointed out, rendering is just saving the cache.... I cannot see a worse result from the EXRs/DPXs compared to what comes out of the rocket. Actually - at times it looks better (even though it isn't really) because the rocket cannot manage more than half debayer in realtime with resolve, and the macs we use don't really have slots for many more rockets, even though we use the expansion cases. (Now: Why we still use Macs for this is another discussion... )


There are so many assumptions as to signalchains out there, and people tend to be a bit more religious than practical about the issues.
My point is: It doesn't harm to be practical. Sometimes that means RAW all the way, sometimes not. You'll need to have an idea about what you do either way, because the results will be extremely similar with the same signalchain/metadatasettings, whether you transcode or not.

(digressing anecdote: A guy at one of the workshops had learned through an internal study how "bad" prores is - which it of course to some extent actually is. But the practical result of his argument is: Shoot DSLR/XDCAM - as those are the options... Kinda... why not just do the prores then?

And that's why I am in favour of some baseline settings for transcodes, as that insures a result which will never be worse than X. And that X, is actually pretty cool... THEN experiment to get it better and get familiar with the format.

Kinda.

Thanks for chiming in BTW!

Appreachiated.

Cheers!

G

 

[Martin Stevens]

:emote_popcorn::emote_popcorn::emote_popcorn:

Great thread Gunleik.

Also, how do you pronounce your first name?

 

[Andy White]

:emote_popcorn::emote_popcorn::emote_popcorn:

Great thread Gunleik.

Also, how do you pronounce your first name?

Wow - was just reading through this thread and thought the same thing! have been wondering since seeing Gunleik's Epic tests last year

 

[Gunleik Groven]

Thanks and hahahahaha to the popcorn...

My name is pronounced like...

Gun is with the same vocal sound as in "goon" but short, like it had a double "n". Thus "goonn"

leik is pronounced like the "leic" in "leica".

so...

Goonnleic

Hahahaha

Back On Topic!

LOL

 

[Hans von Sonntag]

It's nice... Could do with a couple of foolproof presets, though...

Most DOPs, producers and directors are non tech-savvy people. RedCineXPro is what is says: a pro application. Way too complicated for many. If there is room for improvement on the non-hardware side then by simplifying things. But this is not about RWA really. Don't want to interfere with your interesting thread.

Hans