Cineon 10bit Log encoding for DPX discussion thread

[Evangelos Achillopoulos]

I decided to start a new thread in order to clear up what Cineon encoding is, to RED team, below is my opinion, Mike Most, Mitch Gross and others please tell me if I'm wrong.

Everyone's got a different opinion about what's best for post and there really is no "one simple way". We will continue to add more options and flexibility for log workflows though because we often get requests for different types of outputs for log workflows (and they're all different).

That is what made me start this thread... I thing that there is one log Cineon workflow (which is equivalent to scanned film DPX's) and the encoding of it is described below...

When we scan film we measuring negative film densities, status M that is, in RGB... film has a base density that we call "base", this is when the negative is developed, in the unexposed area at the perforation of the negative, we have an area that is the total black (unexposed) the base... above that base RGB values (densities) are recorded to negative... so to scan it we zeroing the scanner to the base density that we presume its the total black.

So on Cineon encoding which is 10bit Log we have the following structure on RGB values (0 to 1023 RGB) that are corresponding to negative densities above base.

From 0cv to 94cv (95) is nothing (in real life that is used after color grading that black can go below film black as a headroom)... the black Code Value (cv) is 95cv or 95Bcv (B=black point)... then every 0.002 negative densities, above base, is one Code Value (cv)... so for red negative density 1.002D the cineon value is 1.002D/0.002D=501cv then 501cv+95Bcv=596cv, for green negative value 0.75D which is the 18% gray value the number is 0.75D/0.002D=375cv then 375cv+95Bcv= 470cv... that's the LAD gray card number for those who understand what I'm saying... these values are going all the way up to 1023cv even though 1023cv is equal to 1.856D which is a very rare in real life applications...

Now how do we match these values to sensor linear data?

With scene brightness is the answer.

Study the following table and you will see what is the relation...

Or in that link for more info...

http://www.acvl.org/digital_intermediates/dicompanion/ch02.html#id591490

In the above table someone can realize that 90% white, is just a white with 90% scene brightness, and that there are whiter (brighter is better) whites than the 90%... that happens to be the other magic number that we commonly see and it called white point 685cv... so thats why the PDlog685 clips everything above 685 white... because, for a weird reason that I can't understand, someone on the RedAlert code (or REDCine code) presumed that above that white is nothing... or that it must compress all the values between 95 and 685...

Also its understood that in order to make the conversion of linear sensor data to Cineon values you have to totally disclose the maximum scene brightness that RED sensor can withstand... If that is not being done, totally aligned with the above method, then there will be problems like Rainer is describing...

The rest its just mathematics... that I thing Graeme is very capable of solving...

I just described a very precise description of what we all want... (I thing...)

This is how you going to create DPX files according to Cineon 10bit Log spec.

That is what all want.-

 

[Evangelos Achillopoulos]

I'm the only one that needs a proper Cineon 10bit Log DPX???

Anyone that needs something similar???

 

[M Most]

I'm the only one that needs a proper Cineon 10bit Log DPX???

Anyone that needs something similar???

Anyone who does DI work does. And the lack of a proper Cineon Log interpolation LUT from Red has led to a rather chaotic situation in which each facility rolls their own solution. This is not optimal, and Red has seemingly been very reluctant to supply a "blessed" solution themselves. Why, I couldn't say. But maybe Deanan or Graeme can. The statement that they get different requests for different types of outputs for log workflows only serves to tell me that they've been talking to the wrong people. There is only one Cineon format, it is well documented, and it is exactly as you have described. And anyone who's finishing for a film deliverable uses that format. That's pretty simple to me.

 

[Marcus Struzina]

Just a question, aren't some film outputs made directly from log DPX files that do not follow the cineon spec, and aren't some of these optimised to spread the red data more fully through the 10 bit range ? Are these the log workflows that Deanan is referring to in the quote ?

 

[Evangelos Achillopoulos]

Just a question, aren't some film outputs made directly from log DPX files that do not follow the cineon spec,

Film outputs always follow the Cineon spec because film only understands densities...

and aren't some of these optimised to spread the red data more fully through the 10 bit range ?

Its not needed, it wastes our time to workaround it, and it makes log function to misbehave, read below why...

Are these the log workflows that Deanan is referring to in the quote ?

Its better to let Deanan to answer that... as far as I'm concerned, I thing yes... although I don't understand the use of them...

Human perception of light is a logarithmic expressed phenomenon as most of the human senses, film densities are tracking this phenomenon accurately, Cineon by tracking film densities is tracking the same phenomenon that is stimulating human perception of light...

So there is no need for extra info... because this extra info is rejected by human perception of light...

It is that simple...

ahhh... I forgot a small detail... it is also messing up our Log workflows and making all of us to do unwanted workarounds and trade offs...

And for those that want all the linear info there is always the 16bit TIFF's...

Let me elaborate a bit in what mis behavior is with the help of Rainer Fritz:

At PDlog685 we saw that the blacks modulating very nice, like you know it from scans, but the highlights have not that headroom as in redlog. Vice versa the blacks in redlog are modulating less than in pdlog685 but the highlights have much headroom. The colors and contrasts in pdlog685 are for my opinion a little tighter than in redlog. So it was our choice to do a linear grading.

So the guy at the end forced to do linear grading because none of the "log" export formats was adequate...

That is what I call mis-behavior...

 

[Marco Graziaplena]

ok... I need some explanations please...
what's the best settings to make a log-to-lin LUT if I went for Redlog dpx 10bit?
and for PDlog 985?
What's the differnce actually between redlog and pdlog985?
why RED doesn't give the right values for a redlog LUT for all compositing & color correction applications?
Thank you very much

 

[Kaku Ito]

Evangelo thank you very much for this thread.

 

[sander kamp]

Hi Evangelos, I have the following problem: here in this country the post houses are very reluctant to work with RED footage. They will do it but make it very expensive. For instance to get a edl of a 30 second clip into there Scratch or other system they ask for a 1000 dollar or more transcoding fee - that is before they do any actual grading. Since as far as I know Scratch doesn't need transcoding I wonder what they are doing.

To get around this I was thinking of giving the posthouse DPX files. But then I am getting into all of the problems you are describing of which I know nothing about since I have never worked with film scans. What would be the best way to make sure the post house gets something they can work with? And how can I see for myself if that works or not?

 

[Evangelos Achillopoulos]

To get around this I was thinking of giving the posthouse DPX files. But then I am getting into all of the problems you are describing of which I know nothing about since I have never worked with film scans. What would be the best way to make sure the post house gets something they can work with? And how can I see for myself if that works or not?

All the workflow solutions that are out there, like Autedesk white paper i.e., are trying to create a workable Log DPX first...

But this is a dream... if you use the REDLog -2 exposure it creates something workable but as you read from Rainer post you losing the quality in the blacks...

So as long as RED keeps this arrogant behavior and is refusing even to post something in this thread I thing you will be tied up in either compromise and try the above WORKAROUND or use 16bit Tiffs...

I use capitals in order to be very specific that there is NO REAL LOG EXPORT from any of the software that RED is providing up to day.

I hope in the new RedCine they will address this issue...

And again I hope that their arrogant behavior is not driven by thinking that I am a bustard that is always pitching them... yes I'm not going to behave like a funboy, its very simple, I'm a professional that every day is supporting RED owners to output the best looking images for their movies and as long as this problem continuous to exist, I will be very stubborn on that subject, because is my duty to do so.

Jim, throughout the history the worst thing that happened to all King's, was the flatterer's around them, that was always hiding the reality and they led them to take wrong decisions... I'm not going to be another flatterer Jim... I'm showing something here... please open your eyes, its for the best of your child... please be certain that we are on the same side...

 

[Rainer Fritz]

First I have to say, that I'm not that experienced as Evangelos or other people here in the forum, which working for many many years in filmlabs, what I can describe is, what I saw so far in the last feature projects when we went into DI and made tests to get a proper workflow there.
We made our first tests on a filmlight baselight 8, which has a calibrated christie cp2000 beamer. The hole pipeline at this filmlab is calibrated and they have their LUTs to work proper on log scans every day. Also a ARRI filmrecorder with newest hard and software runs there. http://www.listo.at
We exported footage with all output LUTs which are provided by the redconverters. We also tested the autodesk recommended workflow for PDlog985 in the autodesk whitepaper, which I think was created in a early time of red. It is the PDlog985 -2 stops thing. So, we put the log/lin LUT on the material to grade it with the linear viewing environment first on redlog, then pdlog685 and also pdlog985 as well the autodesk version of pdlog985. We had also a lot of scans there with similar conditions (int/ext in high and low contrast as well lowlight conditions) to see how it works there.
When we started with redlog the first impression was, that the blacks are too tight. When we looked to the histogram of the signal we saw that the modulation in the blacks wasnt that much. When we printed the redlog test log graded it was the same there. Very tight blacks and a lot of modulation in the highlights.
The next test with pdlog685 looked from the source material nearly vice versa. I must say for a print with a log workflow better then redlog. Good modulation in the blacks, but tighter in the highlights then redlog. The interesting thing was that the control of secondaries was harder, also the contrast much more different then redlog, and in my opinion the colorspace also different specially in the skintones. Same result on the print. PDlog985 also with the autodesk version was no option, because the headroom in grading specially to the highlights were so tight, that we decided to not use it.
After this testing we knew, that we cant work with the calibrated LUTs for the workflow they use on scans. So we were in the situation that we need a truelight LUT or we go to a linear workflow and do the lin/log conversion for the filmout on the output of the grading or at the input on the ARRI recorder.
For the first way we decided to do a linear workflow and the lin/log conversion on the ARRI recorder, because it provides a 3D LUT which give amazing results. So we tested again redlog material and pdlog685. On the grading suite we graded in full range without LUT and made for both versions a basegrade that we see a proper straight linear picture. After that we started to grade. As well here we saw the same effect with the blacks described above, but had much more room to grade in both directions as well in the mids. Then we printed both versions redlog and pdlog685 linear graded. And got very nice results. Sure not the same as you would have in a log pipeline, but amazing at all. The redlog material was showing us in the blacks at the print a lot more then we thought at grading. PDlog685 performed also very well, but as I said not that good in the highlights as redlog.
As I know from other projects, for example Antichrist, they went through log workflow on scratch. They had success on it, and if you see the pictures you will know what is possible, but they made a LUT calibrated to their environment with the red footage i think from cinespace.
One big effort is to direct RAW conform into the CC, where you could bring in all the bandwidth which the sensor captured. Which we tested on speedgrade and nucoda. Works very well there so far and you have the controll of all the RAW settings and 12bit color depth.
The only thing I can say at the end is, test, test, test and get experience with this footage.
We have the rolls stored, if anyone want to see them I could ask the lab to get those.

best,
rainer

 

[Evangelos Achillopoulos]

Rainer, to sum it up...

With RED there are four ways to work...

RAW, with Log workflow, but with what Log conversion?
RAW, with linear workflow
Linear with Tiffs
Log DPX with what? there is not one simple way to do it...

My problem Rainer is that sentence...

"And got very nice results. Sure not the same as you would have in a log pipeline, but amazing at all"

Why we have to compromise? and not be able to get the best out of RED?

 

[Daniel Dávila]

Hy Evangelos, we talked some time ago...
What do you think about the Autodesk (pdlog985 -2 exposure) route? As I understand, the R1 don´t capture so much highlight headroom as film, but at least, the Autodesk whitepaper indicates how to map the values near a tipical cineon scan inside the 95-685 range, and there's even a little bit over 685 to play around.
I agree with you that is very important to work in the standard log space, why RED does give us a standard REC709 output and don´t give us a standard log output? I thought RED was a "Digital Cinema" company, not a "Digital Video" company.
Ok, many times on this forum Deadan and Graeme has told us that the best (best = maping all the dynamic range all along the value range of a 10-bit file) way to get a log image from Raw data is using the RedLog gamma, but, in my opinion, right now we don´t need the "best" way, we need a standard (that means as close as posible to a Cineon Log scan) output, because we have to use pre-RED tools, like conforming and CC software, film recorders and, by the way, film stock, lab and film projectors!
I have to say that it´s very common, even before RED, that there´s some ignorance in the film post world, so, we always have to make tests and tests. But, it would be very nice to have a RED official STANDARD way to map the Mysterium sensor values into a log file (I mean, clearly usable in RED software, not a simple post in this forum).
I personally use the Autodesk route, convert to a AJA 10-bit RGB log movie and CC with Apple Color. I finally convert it to DPX to film out in Arri Laser. It gives me nice results...
Just wondering, why they didn´t put the pdlog985 gamma inside the RED tab in Color??? Just can´t use the Autodesk route in there.

Just waiting what will be the dynamic range improvements in the new sensors and how to use it correctly.

Greetings, Daniel, from Santiago, Chile.

 

[Carsten Fenstermacher]

But, it would be very nice to have a RED official STANDARD way to map the Mysterium sensor values into a log file (I mean, clearly usable in RED software, not a simple post in this forum).

I'm hearing two different goals in this thread. Rainer wants the best workflow and Evangelos and you want a standert one, is that correct?

I think from the point of best workflow, 16bit linear is going to be superior. 10bit is more of a compromise because you have to figure out what to throw out and there are tradeoffs. cineon was not designed for digital and the documentations does not provide answers excepts for 8bit video. This is why we end up with so many different log formats. the toe and shoulder that's not there in digital is not a known transfer to cineon log.

In our posthouse's testing, redlog throw out the least and if you linearize on ingest then you get the advantages of a 10bit space savings.

Rainer, were you linearizing the redlog or putting it straight into a print emulation lut? That could explain why you saw less range in the shadows? looking at the distribution in redlog, it holds more range in the shadows but i belive it should be used with a linearization step. It's more like a transportations format like S-log rather than a film grading format.

Evangelos, I think we may not see a response from red in this thread. When you are aggressive and hostile in a public forum you force the other side to not respond. Then maybe no one benefits from the intention of your post.

 

[Evangelos Achillopoulos]

I'm hearing two different goals in this thread. Rainer wants the best workflow and Evangelos and you want a standert one, is that correct?

If you read the whole history of Rainer posts you will understand that he wanted to do the typical Log Cineon workflow... but he got the problems that all have... non standard Log exports with bad color grading behavior... Carsten we all want the same thing... there is a linear export with Tiffs - DPX for REC709, why wont we have a proper Log export for film work?

Evangelos, I think we may not see a response from red in this thread. When you are aggressive and hostile in a public forum you force the other side to not respond. Then maybe no one benefits from the intention of your post.

The whole thing started when I ask Deanan very politely about the subject and I got back the copy/paste style answer that is in the top of the thread...

I have a language barrier, I'm not a native English... my language is Greek... I can only answer straight, the political blah blah is not the way I talk...

On the other hand read Mikes post and you will realize that Deanan answer was inadequate... For non understandable reason's they keep doing this :emote_headwall:... and they don't understand that they hurting their clients, their product and their heads...

I don't expect an answer from them, I'm expecting an action towards to the solution of this bug... because I thing its a bug...

It would be wonderful if they get involved, with a positive attitude and really start discussing the subject, you see I just want to do my job well, between professionals that they want to do their job and they share the same goals I don't thing there is room for hard feelings.

Maybe I'm wrong...

You see I don't own a RED, I'm just helping SILENT RED owners to work and create beautiful images with their cameras... if that is so bad then let it be unsolved...

I AM ON THE SAME SIDE OF THE FENCE... I'm just speaking louder... I'm calling the Fire Chief to help me put out the fire :grouphug:

 

[Evangelos Achillopoulos]

I think from the point of best workflow, 16bit linear is going to be superior. 10bit is more of a compromise because you have to figure out what to throw out and there are tradeoffs. cineon was not designed for digital and the documentations does not provide answers excepts for 8bit video. This is why we end up with so many different log formats. the toe and shoulder that's not there in digital is not a known transfer to cineon log.

Cineon doesn't care about toe and shoulder... its about the way human perception understand light... Its the logarithmic expression of scene brightness... so it is very obvious what needs to be kept and what needs to be thrown... read carefully what I write and you will understand... at least one has understand it... Mike that is... its very simple.

With 10 bit Log you can get 14bits linear easily... RED is just 12bits...

 

[Carsten Fenstermacher]

The whole thing started when I ask Deanan very politely about the subject and I got back the copy/paste style answer that is in the top of the thread...

I have a language barrier, I'm not a native English... my language is Greek... I can only answer straight, the political blah blah is not the way I talk...

It's not mine either but ranting transcends language

 

[Steve Shaw]

There are a number of problems with the discussions within this thread...

First Red doesn't have the dynamic range of film, so cant be mapped to the full Cineon profile which is based on a larger DR.

And second, it really doesn't matter...

A DI operation has to use a calibtarion LUT based on the deliverable chain (film recorder, film stock, lab processing, print stock...)

The source material is not a factor in the generation of this LUT.

You can feed the DI system any image and the process of grading it while viewing through the print LUT 'pushes' the image data into Cineon LOG space.

The issue is making sure the image has the best granularity possible before grading.

This means working with Red Lin Light data at its native 12 bits (16 bit file format for most systems as 12 bits is unusable) - or 10 bit log (RedLog) as a 10 bit Log image has a granularity equavelent to a 14 bit Lin image.

This means you have the required quality of source to then grade as required using a true calibration LUT for the delivery chain.

I hope this makes sense as it is rather important to understand.

 

[Evangelos Achillopoulos]

There are a number of problems with the discussions within this thread...

First Red doesn't have the dynamic range of film, so cant be mapped to the full Cineon profile which is based on a larger DR.

It doesnt need to... Panasonic has an exporter from FilmRec AVC-Intra footage on 3700 that gets up to 850cv... so its not full range since it doesn't need to... these DPX's from the P2 exporter are working like a dream...

And second, it really doesn't matter...

A DI operation has to use a calibtarion LUT based on the deliverable chain (film recorder, film stock, lab processing, print stock...)

The source material is not a factor in the generation of this LUT.

You can feed the DI system any image and the process of grading it while viewing through the print LUT 'pushes' the image data into Cineon LOG space.

The issue is making sure the image has the best granularity possible before grading.

This means working with Red Lin Light data at its native 12 bits (16 bit file format for most systems as 12 bits is unusable) - or 10 bit log (RedLog) as a 10 bit Log image has a granularity equavelent to a 14 bit Lin image.

This means you have the required quality of source to then grade as required using a true calibration LUT for the delivery chain.

I hope this makes sense as it is rather important to understand.

Yes for the need to use a film LUT or a print LUT... But nope to the rest... read the 10 post on this thread... its a good example of the problem http://www.reduser.net/forum/showpost.php?p=442762&postcount=10 Rainer had a difficult time to do it and at the end he followed a workaround... Even in grade from RAW, on every shot we have to try to achieve the optimal... since in the RAW tabs there is no optimal Cineon color space selection...

Steve, I don't agree neither, lots of people that doing DI for film out also...

We have to do workarounds... I don't like that... sorry...

 

[Charles Angus]

Jump in and correct me if I'm wrong, but if there was a button in one of the Red transcoding apps (Alert, Cine, or Rushes) that simply output a properly encoded Cineon DPX file, you could have a traditional film-style post from that point on. A workflow that everyone knows, is comfortable with, and understands...

 

[Daniel Dávila]

Cineon log button

Cineon log button

Yes, I think this thread is about that, a clear way to get a "properly encoded Cineon DPX". From a properly exposed RED RAW, we want a DPX LOG file that, without adjustments, can be read by any LOG application, and, with the same LUTS used with 35mm scanned material, it looks "correct". And more relevant, the printed copy also looks "correct". Sure it can be done now, but after much research and testing.
What's your way, using Autodesk's road or starting with redlog and make some adjustments?

Greetings, Daniel