RAW workflow - a discussion

[Gunleik Groven]

Most DOPs, producers and directors are non tech-savvy people. RedCineXPro is what is says: a pro application. Way too complicated for many. If there is room for improvement on the non-hardware side then by simplifying things. But this is not about RWA really. Don't want to interfere with your interesting thread.

Hans

Cannot ever remember you interfering with anything, while I...

And it is very relevant to the RAW subject, as it happens to scare a few people to inferior solutions...

NOW I'll dig up some images, as noone else have volunteered...

 

[Steve Sherrick]

Definitely post the test shots/comparisons. I will be very interested to see your analysis. I love these types of threads. They are specific and they aim to help people understand. I always learn something. Thanks for doing it Gunleik!

 

[Matthew Scott]

@ Mathew..

That can be a good approach. You do not destroy your image that way, as ISO is created through floating point curves operations.

You might (or may not) find it as easy to go to redlogfilm and grade from there, as that gamma (cineon) is created to protect both the highs and lows with headroom for grading both ways.

If you go for direct to delivery, your approach is as god as the results you see. Which can be excellent!

Thanks mate, good to know. I'm getting great results like this, just wanted someone with a higher knowledge to confirm that it's safe Rg3 is beautiful out of the box, no doubt, but it's very rare I want an out of the box image. I grade everything I shoot and having the flexibility to get contrast where I want it with RedLogFilm is much better IMHO. Thanks again for the response.

 

[Gunleik Groven]

Ok... Here goes.

These are 4 exposures shot in parallell with R1 MX and Epic.
Both used 35mm Masterprimes

I tend to like to look a bit at the extremes... (I have a lot more exposures and direct light into the cameras and and and... but to get them as similar as possible, without the difference of angle totally ruining the comparison, I'll stick to a basic situationnn), and like images with some dynamical challenges, so here goes...

To introduce the images, I have chosen to develop them at 5600/800/RG3/RC3 as that is the current "most advocated flavour". (I know... NOT for tungsten... But I had to start somewhere.... ) )
I have developed 16-bit dpx's you can download with the shots alongside the R3D's, and you can load them into your favourite grading app alongside the R3D's and see if you get significantly different results with the R3D's and the DPXs. EXRs would probably be better, but there is an additional level of knowledge involved with linear files, so I chose DPXs for the examples.

First batch is just presenting the raw files and images. Not much to comment on, really...

The Epic was shot with 5:1 compression and 3 stops HDRx for all the images.
That may have affected the lower exposures negatively somewhat. But that's not relevant for what I want to look into.
All shots at 4k HD, as that is our format of choice for TV (mostly)

ah...
I see from the images, that it can be good to add that no NDs were involved in these particular shots...

The jpg's are BTW 1080, but you'll have to rightclick and "show image" to view them like that...

Here goes.

@ t 1.3 - R1MX

@ t 1.3 Epic M

@ t 2.8 R1 MX

@ t 2.8 Epic M

@ t 4 R1 MX

@ t 4 Epic M

@ t 11 R1 MX

@ t 11 Epic M

All the RAW stillframes are in this zip:
http://gunleik.com/RAW/R3D.zip

And the 1st batch of DPXs can be found here
http://gunleik.com/RAW/dpx_800_rg3_rc3.zip

Have fun and nJoy

I am not sure how much further I'll be able to dive into this tonight, but please download and discuss...

Cheers!

G

 

[Frank Weeks]

Great thread Gunleik. Thanks for your efforts.

 

[Gunleik Groven]

Thanks Frank.

Where I want to go from here is actually two places...

1. Use the different DPXs I supply and the different metadata I suggest in your grading app, and see if the RAW and RGB files perform any differently, if so how?

2. Look at different metadatasettings and how the images are affected.

But I will most likely not finnish this in a day or two, as I do this just on some very limited spare time and for fun...

Just to be clear, these first developments aren't really good for grading.
They're just "baseline settings" for reference... and that's kinda the point with them...

But please come with your analyses and questions.
I am sure there are somewhere here who can answer.

Would love to have you guys join in, if there are corrections/suggestions to what I do here.
I'm here to learn!

It's not really intended as a deep comparison between R1 and Epic, but as there are a few of both cameras out these days, it can be fun to look into what happens with both with identical settings and setups.

But if there is anywhere better to take this, just suggest!

Cheers!

G

 

[Gunleik Groven]

Just a quickie, before I have to take a nap...

'couple of small writeups, that ultimately quickly become relevant to this thread.


The advice of shooting 800 is a practical one. The camera calibrationpoint (not really "native point") is 320/daylight. 800 Is a good practical place to start. To set your ISO higher, forces your exposure down. Especially good for tungsten shots, to avoid clipping RED. It also defeats ETTR. Check this:

http://www.gunleik.com/tests/alexmx/RED1_MX_and_Arri_Alexa/ISO_&_Kelvin_-_an_introduction.html

Here's a couple of things to think of when fiddling with in-camera metadata:
http://www.reduser.net/forum/showth...RedCINE-Curves&p=995758&highlight=#post995758

BUT, as long as you get your desired results, you are fine, whatever you do!

Cheers!

G

 

[Ido Karilla]

Ok... Here goes.

These are 4 exposures shot in parallell with R1 MX and Epic.
Both used 35mm Masterprimes

I tend to like to look a bit at the extremes... (I have a lot more exposures and direct light into the cameras and and and... but to get them as similar as possible, without the difference of angle totally ruining the comparison, I'll stick to a basic situationnn), and like images with some dynamical challenges, so here goes...

To introduce the images, I have chosen to develop them at 5600/800/RG3/RC3 as that is the current "most advocated flavour". (I know... NOT for tungsten... But I had to start somewhere.... ) )
I have developed 16-bit dpx's you can download with the shots alongside the R3D's, and you can load them into your favourite grading app alongside the R3D's and see if you get significantly different results with the R3D's and the DPXs. EXRs would probably be better, but there is an additional level of knowledge involved with linear files, so I chose DPXs for the examples.

First batch is just presenting the raw files and images. Not much to comment on, really...

The Epic was shot with 5:1 compression and 3 stops HDRx for all the images.
That may have affected the lower exposures negatively somewhat. But that's not relevant for what I want to look into.
All shots at 4k HD, as that is our format of choice for TV (mostly)

ah...
I see from the images, that it can be good to add that no NDs were involved in these particular shots...

The jpg's are BTW 1080, but you'll have to rightclick and "show image" to view them like that...

Here goes.

@ t 1.3 - R1MX

@ t 1.3 Epic M

@ t 2.8 R1 MX

@ t 2.8 Epic M

@ t 4 R1 MX

@ t 4 Epic M

@ t 11 R1 MX

@ t 11 Epic M

All the RAW stillframes are in this zip:
http://gunleik.com/RAW/R3D.zip

And the 1st batch of DPXs can be found here
http://gunleik.com/RAW/dpx_800_rg3_rc3.zip

Have fun and nJoy

I am not sure how much further I'll be able to dive into this tonight, but please download and discuss...

Cheers!

G

Just from looking at the jpeg's, are you sure all had the same gamma curve?
If so, can anyone (Graeme) explain the different response (same sensor, same gamma, same color matrix) of the cameras?

Thanks.

 

[Gunleik Groven]

Just from looking at the jpeg's, are you sure all had the same gamma curve?
If so can anyone (Greame) explain the different response (same sensor, same gamma, same color matrix) of the cameras?

Thanks.

Go check...

I might have messed up...

The r3d's and RMD's are in the same folder and not big at all...

I re-looked at the 1.3 just now, and have the same results...

I cannot see how I can have managed to miss that marking on the lense...

Anyhow, with the current choice of in-camera gammas, the cameras obviously look even more different on set...

I am BTW in no way surprised that they differ. More surprised that others have thought that the cameras are basically the same...

Same silicone, but after that, everything is different...

 

[Dan Hudgins]

Non-white-balance TIF or DPX 48bpp?

Non-white-balance TIF or DPX 48bpp?

Is there a way to output non-white-balance 48bpp TIF or DPX as "linear light" that have no-clipping and 100% of the tonal data from the de-Bayer interpolation of the R3D data? (That is without both ISO adjustment and without white balance so the meta-data for ISO and K have no impact on the conversion from R3D to 48bpp TIF or DPX)

Normally the conversion of Bayer data goes through these steps:

1) Record Bayer pattern uncompressed as a monochrome image, or break the four Bayer colors into four half size monochrome images and wavelet compress those smaller images, or interpolate the two greeens and save green full size and red and blue as half size with wavelet compression.

2) Decompress or otherwise load the Bayer data back into its original pattern, or make four uncompressed images, red, blue and two greens half size, or uncompressed to a full size green image and half size red and blue.

3) Interpolate the red and blue images and maybe the two green images if they are recorded half size so you end up with three full size images red, blue and green.

4) At this point you have a choice of to white balance or not, if not then its "raw" if so then its RGB workflow.

4A) Save the non-white-balance as 48bpp DPX, TIF or linear DNG. (in this case there is no burned in ISO or K)

4B) Save the image through a whitebalance LUT to any RGB or YUV format. (the white balance LUT may also correct midtone for 46% etc. to establish the ISO speed as well as the K balance to neutral subjects)

What I am asking about here is if there is a setting in REDCINE-X (tm) to export the R3D interpolated or inflated uncompressed data without the white balance being done to make sensor native color balance matching signal levels that are 100% matching the sensor signal levels including sensor black level etc. or at least a linear mapping of the sensor data without white balance so that the tones that are normally clipped are left in the color channels so that say the blue and green in the result files have the 90% white card signal level at about 60% to 75% of the green signal as is normal for actual raw sensor data in a Bayer DNG file or RAW data dump from the ADC output of the sensor circuits.

If you can save the linear sensor data out of REDCINE-X (tm) to a so called RGB file format like 48bpp DPX or TIF, then you can use that as an actual type of raw workflow with RGB tools.

The primary difference between what should be called R3D workflow rather than raw workflow is that you rely on the RED SDK (tm) to re-map the image tones from sensor signal level rather than doing it in some other program, so its about the placement of the white balance and other LUT in the signal chain.

If you can bypass the LUT in REDCINE-X (tm) and output the sensor signal levels without white balance to a high bit depth uncompressed 48bpp DPX or TIF as linear data then you have other options than using the SDK for the white balance and first correction of midtone to 46% etc.

So what settings in REDCINE-X (tm) are used to bypass ALL of its impact on the relative sensor signal levels?

Getting unbalanced data out as log is not as good a solution since you need to re-linearized before processing and there may be some mismatch in the anti-log curves, so saving in linear from REDCINE-X (tm) as gamma 1.0 from the sensor data, that as far as was said in the past is always recorded with fixed sensor gain, gives an external copy of the sensor data that can be worked on outside the RED SDK (tm) with tools of the users choice, something like making a Linear DNG file.

(for those who don't know there are two kinds of DNG files, Bayer DNG and Linear DNG, Bayer DNG is like a monochrome TIF file of the Bayer pattern sensor data, since the sensor is monochrome it saves just one color monochrome image full size that happens to have the Bayer pattern filtered on its pixels in four pixel groups, the Bayer DNG does not do anything with the colors in recording so its just a monochrome checker board image and so 1/3 the size of a Linear DNG file that is just like a RGB TIF file that has RGB data that is not white balanced so it captures the interpolated Bayer pattern making it 3x as large s the Bayer DNG file, R3D can be converted into Linear DNG since after the wavelet compressed images are inflated through de-compression you end up with red, blue and green full size images that are representations of the sensor signal levels, saving those without ISO or K corrections to a Linear DNG or 48bpp TIF or DPX file lets those K and ISO corrections be done with tools other than the RED SDK (tm), who good those results are depends on what tools are used and the skill of the person using them, in my case my workflow can produce good results from uncorrected image data from any source that ends up in a 48bpp TIF or DPX file as long as it is an accurate representation of the original sensor linear signal levels and has not been mucked up by clipping or chroma matrix in some way beforehand ect. In the case of my workflow the camera meta-data or the DNG meta-data have no impact on the results as they are not used for ISO, K, or white balance of the graded images as such adjustments are based on the sensor signal levels only)

 

[Gunleik Groven]

@ Dan

AFAIK that is currently not an option, and RED encrypted the format first and foremost to avoid exactly this.

I do not see a huge problem with discussing a R3D workflow as "RAW", though - even though it to some extent is matrixed. It will be matrixed anyway, and RED has chosen to keep controll on that. By looking at noisegeneration at different ISO and kelvin settings I guess one could come far as to reverse-engineer that matrix, but then: Why?

By now they are really good, and we need the sensor to be calibrated before we have any fun anyhow...

 

[Hans von Sonntag]

To introduce the images, I have chosen to develop them at 5600/800/RG3/RC3 as that is the current "most advocated flavour". (I know... NOT for tungsten... But I had to start somewhere.... ) )

Thanks for the discussion!

Now I disagree with you. Personally I find that Epic's sensor is best pictured in the realm of 250 - 320 ISO. Red's advice to shoot at ISO 800 forces about 1 stop of underexposure and shows their confidence in the MX-sensor's noise level. In many circumstance this makes sense. However, it does not mean that pushing one stop above "native" ISO in the development process by using the metadata would be always desirable, on the contrary, IMHO.

If we are talking about an environment that is obviously challenging the sensor, meaning the full DR of the camera is needed, pushing roughly 1 stop by dialing in ISO 800 in RedCine will kill 1 stop DR ind the highlights.

Conclusion: if your environment has a lot of DR underexpose 1 stop, ISO 800 in camera will help here. But development at e.g. ISO 320 will safe your highlights. If you are shooting in a lowlight environment and exposure is thin pushing 1 stop might give an edge in the grading process, or not. Actually I doubt that. If your environment is not DR-challenging chose ISO 320 for exposure and develop at ISO 320.

Hans

 

[Gunleik Groven]

Hans... I agree, sorta

Point here is that I will be doing a lot of different developings... And look at the result.

For reference something in the ballpark of the "RED reference" (being RC3/RG3/800/daylight) settings, should be fine.

I wouldn't use these settings for anything practical, though, but let's say they are a noise and DR baseline (not best-line)

Point here is partly to show that even in an "all RAW" workflow, you have to take care of your data.

I haven't done that with the baseline settings...

 

[Hans von Sonntag]

Just from looking at the jpeg's, are you sure all had the same gamma curve?

Ido has a good point here. If in all shots the same gamma curve is used, e.g. RedGamma3, then the RedOne MX has an edge over the Epic regarding DR.

I would love too see the two shots @t4 developed at ISO 320/250 and in RedLogFilm.

Thanks Gunleik!

Hans

 

[Dan Hudgins]

@ Dan ...even though it to some extent is matrixed. It will be matrixed anyway, and RED has chosen to keep controll on that.

That brings up the point of if the sensor data is matrix-ed before wavelet compression or not? There could be an advantage in doing it before compression, as there is for green interpolation.

In order to apply the EDR in my processing their should be un-matirxed sensor data in the source because applying normal white balance color matrix pollutes the luma information with noise from the chorma interpolation issues.

 

[Hans von Sonntag]

For reference something in the ballpark of the RED reference settings, shpuld be fine.

I wouldn't use these settings for anything practical, though, but let's say they are a noise and DR baseline (not best-line)

Point here is partly to show that even in an "all RAW" workflow, you have to take care of your data.

Gunleik, my point is that there is no such thing as "power of RAW" when its comes to DR. The only reason why people find here an then miraculously missing information in the highlights is the fact that they miss the right development by dialing in a high ISO Number. When they drop down the ISO they find missing information and call that "jaw dropping". Dropping below ISO 250 won't dig out any more information in the highlights, above ISO 400 can kill highlights, depending on your footage. Using ISO 320 will keep you safe in any circumstance.

Hence my proposal for an applet that has only 1 ISO setting (e.g. ISO 320), 1 colour science, 3 Kelvin settings (3200, 4600, 5600) and 2 gamma curves, RedGamma3 and RedLogFilm. Every DOP, every camera assistant would understand these numbers because they are common basic knowledge.

Hans

 

[Gunleik Groven]

Hans, I agree...

I am just moving towards the conclusions at a bit slower pace...
And want to cover a couple of other things for those who want to play it on their own...

One point being: "RAW" and "transcoded" is technically equal.

If you push things out in top or bottom prior to grade, you will get the same results with an "untranscoded" signal as you get with a "transcoded".

But that is only one of the points I want to make...
Thus my slower and less enthused pacing of this.

And I want to bring whomever wants to along the road, without getting too geeky and quick...

I think a bit of this info can be usefull to a whole new generation of R3D shooters and posters these days...

It's not really REDucation. It's just a slow walk down signal paths...

Cheers!

G

 

[Gunleik Groven]

Ido has a good point here. If in all shots the same gamma curve is used, e.g. RedGamma3, then the RedOne MX has an edge over the Epic regarding DR.

I would love too see the two shots @t4 developed at ISO 320/250 and in RedLogFilm.

Thanks Gunleik!

Hans

The R3D's are there to be downloaded...

Run them and see what you find...

 

[Gunleik Groven]

That brings up the point of if the sensor data is matrix-ed before wavelet compression or not? There could be an advantage in doing it before compression, as there is for green interpolation.

In order to apply the EDR in my processing their should be un-matirxed sensor data in the source because applying normal white balance color matrix pollutes the luma information with noise from the chorma interpolation issues.

Agreed.
But RED has chosen deliberately NOT to open for this approach.
And to be frank: I don't blame them.

From a purely engineering perspective, that would be fun and exciting.
But R3D is NOT an open format.
It's a deliberate - and I guess very well thought through - choice...

 

[Dan Hudgins]

and I guess very well thought through - choice...

Yes, but it gives their cameras a look, rather than letting the user pick. It is as much a burned in look as Canon's cameras because you cannot work around it if you want to, and its a look that changes with revisions to their SDK.