BSC Film and Digital Image Evaluation 2009

[Gavin Greenwalt]

There is a slight difference between Log and Li in my tests.

Maybe there should be like this always by default but I'm "investigating" that...

There was in my REDCine tests this morning as well. Which tells me that there is something wrong with REDCine's Log curves. I got the best results out of REDLog and some experimentation.

There doesn't seem to be a way to get the a proper cineon file. REDCine -> Lin -> Log (in another program... any other program) produced the best looking log files.

But that's a REDCine problem. And like I said... the reason why I prefer linear. Everyone now adays rolls out their own definition. In your examples you're trying to get the lin to look like the log. That's actually a little backwards because it's RED's log files not its lin files that are 'wrong'.

Something which Evangelos also seems to have discovered: http://reduser.net/forum/showthread.php?t=31727

 

[Evangelos Achillopoulos]

There was in my REDCine tests this morning as well. Which tells me that there is something wrong with REDCine's Log curves. I got the best results out of REDLog and some experimentation.

There doesn't seem to be a way to get the a proper cineon file. REDCine -> Lin -> Log (in another program... any other program) produced the best looking log files.

But that's a REDCine problem. And like I said... the reason why I prefer linear. Everyone now adays rolls out their own definition. In your examples you're trying to get the lin to look like the log. That's actually a little backwards because it's RED's log files not its lin files that are 'wrong'.

Something which Evangelos also seems to have discovered: http://reduser.net/forum/showthread.php?t=31727

Its already one year and two months since my first RED flag... http://www.reduser.net/forum/showthread.php?t=13110 the above is the second try...

The think that Gavin is not understanding, is the use of film derived 3DLUT in the process... if you start from the same point and include a 3DLUT in the pipeline you will get similar results in both lin or log... the 3dLUT will give that different look immediately in both cases...

But there is more to it, If you do grade in linear and have your waveform on, you will realize the difference in the highlights of what you getting with log... that is behavior... behavior leads to a different artistic result...

 

[Sanjin Jukic]

I think that there is not instant match between Log (PDlog685) and Lin (Liner Light) just on one click.

Use whatever grade app with 3D LUT but you always need some tweaking to match look between Log and Lin.

I "played" again with some different 1DLUTS (panalog2lin, lin2panalog, GenArts Sapphire film-effects) and tried to match it.

The results are pretty close but always different and it's good because make more choices in a whole grading process.

Have a look at the pictures that shot on R1 with Leica-R Summilux 50mm f/1.4 @ f/2.8 + Hoya (screw-in) Circ-Pola+ND4+ND2+IR/UV Leica 55mm filters (time at noon in full sunny day):

1.Panalog2Lin

2.Lin2Panalog

3.Log2FilmEffects

4.Lin2FilmEffects

Panalog2Lin

Lin2Panalog

Log2FilmEffects

Lin2FilmEffects

 

[Craig W. Bickerstaff]

Sajin, have you ever made like a short film or something? It seems to me you only ever shoot random objects.

 

[Gavin Greenwalt]

The think that Gavin is not understanding, is the use of film derived 3DLUT in the process... if you start from the same point and include a 3DLUT in the pipeline you will get similar results in both lin or log... the 3dLUT will give that different look immediately in both cases...

But there is more to it, If you do grade in linear and have your waveform on, you will realize the difference in the highlights of what you getting with log... that is behavior... behavior leads to a different artistic result...

Again. For the 100th time to be clear. I'm not talking about what representation of the data you grade in. I'm only exclusively, solely, singularly talking about the way the data is exchanged between apps. My problem with LOG is that not all LOGs are created equal. Which creates uncertainty as to what the file is supposed to look like. Lin doesn't have that problem. Even if you want to grade in LOG I would still prefer a LIN file because then you know for certain exactly what the LIN->LOG conversion was. And if then I handed those files off to the next artist or post house they too would have complete confidence that they too were working on the intended data.

Linear Data and LOG data are completely interchangeable as long as you know the exact inverse curve for the LOG data.

LIN == (LOG * (Inverse Curve)). Even a 3DLUT utility like Truelight in Nuke *Always* operates from linear color space. It just applies an inverse log curve if it's log footage (an inverse log curve which would be incorrect with any of log file saved out of REDCine).

Lin -> Log-> Lin -> Log ... you can do it all day without damaging the data as long as your precision is high enough. Log and Lin files when created correctly contain the exact same data. If you like grading in LOG then just apply a lin->log conversion. If you need to blur or mix in lin just leave it in lin.

Whether you start with a lin file or a properly layed out log file the data should be identical. Not 'close' with some fidgeting'. Identical. In this case it's a redcine 'bug'.

I'm not certain why you keep bringing up 3DLUTs. Since you could feed the 3DLUT a log or lin file at any step of the process just as easily as the other. And log lin conversions shouldn't involve a 3DLUT.

 

[M Most]

I'm not certain why you keep bringing up 3DLUTs. Since you could feed the 3DLUT a log or lin file at any step of the process just as easily as the other.

No, you can't. The LUT has to be built specifically for one or the other. Just because the specific Truelight node in Shake has a switch to allow the input to be either one doesn't mean that a normal color pipeline does the same thing. In almost all cases, 3D LUTs that are created for film print preview purposes expect Cineon Log input. And Cineon Log format is very, very standardized in professional circles, unlike the way you seem to want to keep characterizing it. Just because you might not understand it doesn't mean the rest of us don't.

There are other advantages to working with log images during color correction for a film targeted DI that I've avoided talking about - things like the ability to use offset and contrast as a primary color control rather than telecine type lift, gamma, and gain allow a trained colorist to deal with the material in a manner that's more akin to film exposure and processes, allowing better retention of film characteristics and a less "electronic" result. That's why systems like Baselight and Lustre have both film modes and video modes on their primary controls. These are things that a colorist understands, but a compositor would not. And they are very real reasons why the DI pipeline in almost every major DI facility is designed around cineon log images as the source.

And log lin conversions shouldn't involve a 3DLUT.

Your version of log to lin involves only a 1D LUT. That's fine if you're not changing color spaces. However, if you're trying to simulate a film print from a digital file, the only way to do that is with a 3D LUT. That is the task that is required in digital intermediate work, whether it means anything to you or not. That's why Evangelos (and me) keeps trying to tell you that this is not as simple as you seem to think it is. If all you're doing is compositing, and your only deliverable is electronic, life is much simpler. If you need to target film as a primary deliverable, and look at images that are representative of that (which most of us working in the professional end of the industry do), it's not that simple and a lot of your simplistic solutions no longer apply.

 

[Gavin Greenwalt]

No, you can't. The LUT has to be built specifically for one or the other. Just because the specific Truelight node in Shake has a switch to allow the input to be either one doesn't mean that a normal color pipeline does the same thing. In almost all cases, 3D LUTs that are created for film print preview purposes expect Cineon Log input.

1. Except converting from one to the other is completely trivial. So trivial as to be non-existant. I don't know where you read me saying the same 3DLUT can be directly applied to either and get the same result without converting first. I keep saying over and over and over and over and over and over and over and over again (to deaf ears) that the source doesn't matter because as long as you know what the source is you can switch back and forth at will effectively no penalty.

My entire argument can be sumarized in thise one statement. Pray tell me how I am too simple minded and inexperienced to have screwed it up:
Log -> GRADE -> 3DLUT -> Log -> Film == Lin -> Log -> GRADE -> 3DLUT -> Log -> Film

"It doesn't matter what you start with."

And actually I'm going to take it one step further. Here is the REAL comparison:

Lin (World) -> Log (film) -> Log (scan) -> Log (dpx) -> GRADE -> 3DLUT -> Log -> 3DLUT (Electronics! Voltage! Laser!) -> Lin (laser photons) -> Log (film) -> Lin (Projector) -> Lin (Screen) -> Lin (Iris) -> Log (Neurons)
==

Lin (World) -> Log (film) -> Log (scan) -> Lin (EXR) -> Log (File Load) -> GRADE -> 3DLUT -> Log ->3DLUT (Electronics! Voltage! Laser!) -> Lin (laser photons) -> Log (film) -> Lin (Projector) -> Lin (Screen) -> Lin (Iris) -> Log (Neurons)

And Cineon Log format is very, very standardized in professional circles, unlike the way you seem to want to keep characterizing it.

2. Cin is standardized. Except when someone makes a mistake. Or it's not (See Redlog, panalog, filmstream log etc).

Note that a discrepancy even in REDCine causes an incorrect output of log files. Case in point.

Just because you might not understand it doesn't mean the rest of us don't.

Would you be kind enough to tell me which of my above two statements is incorrect before adding more snark? :cheers2:

 

[M Most]

Pray tell me how I am too simple minded and inexperienced to have screwed it up:
Log -> GRADE -> 3DLUT -> Log -> Film == Lin -> Log -> GRADE -> 3DLUT -> Log -> Film

Okay. You seem to be under the impression that in a film DI environment images are converted between formats. They are not. Film records essentially a log image, which is scanned and recorded to a 10 bit log file. That is used directly by the color correction device. The result of that is passed directly to the film recorder, which records the values represented by that standard Cineon log file to a new film negative. The only use of a 3D LUT is for viewing purposes, not for file conversion (at least not for the film deliverable - a different version of a film targeted LUT is baked in to create video deliverables). There is no file conversion. You're assuming complication that doesn't exist.

2. Cin is standardized. Except when someone makes a mistake. Or it's not (See Redlog, panalog, filmstream log etc).

Professionals don't mistake Panalog for cineon log, nor do they assume that Redlog, Viper Log, or any other curve is the same as cineon, either. If you use a proper print preview LUT designed for the particular monitoring device you're looking at it's rather obvious when the file format is incorrect. If one doesn't have such a LUT - or worse, doesn't understand what it is or why it's necessary - they have no business dealing with digital intermediate work in the first place.

Gavin, I have to say that I'm getting really weary of saying the same things only to have you keep implying in your own way that neither I nor Evangelos nor Steve Shaw nor many other pretty experienced people here know what they're talking about. So I'm done with this. Hopefully you've learned something from the discussion.

 

[Gavin Greenwalt]

MMost we aren't even disagreeing on anything! I'm so confused what you're suggesting I don't understand!

Are you saying that Lin or Log files are empirically better than each other? I don't care about what "best practices are". I'm just talking about empirical quality discrepancies.

Okay. You seem to be under the impression that in a film DI environment images are converted between formats.

Well... it is internally converted to linear photons when outputted from the ArriLaser (and I'm sure run through a 3DLUT of some sorts for the analog response curve of the laser). And if the grading software developers wanted they could take (and I might add most of them do take) linear files and convert them invisibly and perfectly to log behind the scenes.

So you don't disagree: Lin and Log are equivalent file formats. That's the only assertion I've been saying this entire time. I'm tired of being insulted for suggesting that a Linear file format and Log file format won't produce the exact same image. I've demonstrated twice that fact.

Gavin, I have to say that I'm getting really weary of saying the same things only to have you keep implying in your own way that neither I nor Evangelos nor Steve Shaw nor many other pretty experienced people here know what they're talking about. So I'm done with this. Hopefully you've learned something from the discussion.

The assertion that "starting with a LIN file instead of a LOG file will necessarily result in an inferior output" is DEAD WRONG. DEMONSTRABLY WRONG. And no amount of appeal to authority can change math. MMost you haven't made that claim and neither has Steve! And I haven't been argueing with you! I've been argueing with those two who claim there is an empirical difference between a log file format and a lin file format. That the data represented is inferior in the Lin file format.

I keep saying it doesn't matter what color space you save a file in-- you can get identical results. Then you keep calling me an idiot. Why!? The only thing here that we should be in any disagreement in is which is EASIER and that's subjective but my case can be outlined as:

Film is essentially a Log <-> Lin 3DLUT.

Every projector on earth is prejecting LIN.

Every screen on earth is reflecting LIN.

Every monitor on earth is displaying LIN.

If you record to film the laser is a LIN projector. And the film itself is essentially an wonky lin -> log conversion. Then the film in the projector is essentially a log -> lin conversion. Then your retina does a lin -> ~srgb conversion.

If you see it. It's Lin.

 

[David Rasberry]

Then your retina does a lin -> ~srgb conversion.

All human senses are log responsive, not linear. It is log response that allows us to cope with such large dynamic ranges in stimulus, whether light or sound. The point of a log curve in common output formats, like REC709, is to compress a linear dynamic range that exceeds the capability of the medium, in this case HDTV bandwidth, into a more compatible limited range so important highlight and shadow detail can be preserved and transmitted. Midrange tonal values should remain pretty much unaltered in a conversion, but highlight and shadow values at the extremes are significantly altered.

 

[Gavin Greenwalt]

All human senses are log responsive, not linear. It is log response that allows us to cope with such large dynamic ranges in stimulus, whether light or sound. The point of a log curve in common output formats, like REC709, is to compress a linear dynamic range that exceeds the capability of the medium, in this case HDTV bandwidth, into a more compatible limited range so important highlight and shadow detail can be preserved and transmitted.

All of which I 100% agree with. Which is why I said about the same as a lin to sRGB (gamma 2.2) conversion.

Light is Lin. Our perception of said light is... complicated but log/vid ish. Our brain puts a LUT on it.

 

[David Rasberry]

All of which I 100% agree with. Which is why I said about the same as a lin to sRGB (gamma 2.2) conversion.

Light is Lin. Our perception of said light is... complicated but log/vid ish. Our brain puts a LUT on it.

I also agree that a lin>log conversion should be a straight forward mathematical conversion that produces minimal change in the overall look of an image. If there are significant midrange color or value shifts something has gone astray.

 

[M Most]

So you don't disagree: Lin and Log are equivalent file formats. That's the only assertion I've been saying this entire time. I'm tired of being insulted for suggesting that a Linear file format and Log file format won't produce the exact same image. I've demonstrated twice that fact.

All right, one more time. You keep referring to "log" and "lin" as equal. In terms of actual visual information that is the result of the values in the file, that's only true if you have about 16 bits of Lin but only 10 bits of Log. In other words, to have images that are the equivalent of their log representations, you need a hell of lot more space, a hell of a lot faster storage, and a hell of a lot of processing power to play those images at 24 frames per second. In the real world, we don't have that, at least not in current DI systems. Not to mention that there's little to be gained by using those additional bits because in terms of perception there is little to no difference in the image itself.

I keep saying it doesn't matter what color space you save a file in-- you can get identical results.

Lin and Log are not color spaces. They are methods of value coding, with a log representation using a gamma curve. A color space is defined by a color gamut and white point, not by whether the file the information is saved in is Log or Lin. Moving visual information between color spaces definitely yields different results.

If you see it. It's Lin.

Engaging in a continued conversation revolving solely around semantics is pointless. There is light. There is perception of light. But in terms of computer images, film processes, and digital intermediate post production practices, that is somewhat irrelevant. If you want to live in the real world and deal with other professionals in a way that gets things done properly, you really need to understand what it is they're doing and saying and not try to impose your own interpretation.

NOW I'm done.:cheers2:

 

[Gavin Greenwalt]

All right, one more time. You keep referring to "log" and "lin" as equal. In terms of actual visual information that is the result of the values in the file, that's only true if you have about 16 bits of Lin but only 10 bits of Log.

Thanks. That's the point I disagreed with being advanced by Sanjin and Evangelos:

That a 16bit lin image was visually inferior to a 10bit log image. That starting with a 16bit lin image would result in an inferior result. Which is why I was terribly confused why you weren't disagreeing with me but telling me I was wrong and to shut up.

In other words, to have images that are the equivalent of their log representations, you need a hell of lot more space, a hell of a lot faster storage, and a hell of a lot of processing power to play those images at 24 frames per second. In the real world, we don't have that, at least not in current DI systems.
NOW I'm done.:cheers2:

See that's where I disagree. I see a 240 core 2GHZ GPU capable of insane shading calculations per second and the new i7 CPUs and think that the compression overhead in modern linear file formats like OpenEXR are insignificant. OpenEXR can get about 2:1 lossless compression which means it's actually *less* bandwidth than a 10bit Cineon. Or at least that's been our experience.

And I think it's time for people to re-evaluate their options. Especially as they look to future changes in their pipelines. I think log workflows are obsolete with modern systems and that we should standardize on one single formating. Seeing as film projection is slowly dissapearing the importance of a linear projector will be the future. Digital sensors are out evolving chemical and will probably take the majority of features the input will natively be linear. I just don't see a future for log except as an intermediary step in the grading software hidden from the user.

As I said in my first post. This should be completely invisible to the user and painless. I would rather spend CPU cycles and HDD reads than introduce any hastle to a colorist. Let them worry about making it look great, not worrying if they got all the data possible out of a RAW file.

With a 16bit RAW and no clipping on either end you are guaranteed to have ALL the data. No worries. No accidents. Let artists focus on art not tech.

 

[M Most]

See that's where I disagree. I see a 240 core 2GHZ GPU capable of insane shading calculations per second and the new i7 CPUs and think that the compression overhead in modern linear file formats like OpenEXR are insignificant. OpenEXR can get about 2:1 lossless compression which means it's actually *less* bandwidth than a 10bit Cineon.

It's not a matter of pure processing power. Reliable real time playback involves caching schemes and requires file formats that are at least in part designed for that purpose. DPX is one of those. The current spec Open EXR is not, although there is a subset version that ILM has proposed as a real time playback file format (no compression - the decompression would involve its own overhead), and that the Motion Picture Academy's Science council has shown interest in as the basis for their Image Interchange Format proposal. But at this point in time - July, 2009 - that is not a reality.

As for the future, the DI pipeline will change only when film is the secondary deliverable. At the current rate of adoption, and with the current economic climate, that won't be for quite some time, at least a few years at the earliest. However, there are productions - specifically 3D animation - that use a pipeline that is digital cinema centric now. And yes, that largely eliminates the need for logarithmic image representations.

 

[Gavin Greenwalt]

It's not a matter of pure processing power. Reliable real time playback involves caching schemes and requires file formats that are at least in part designed for that purpose. DPX is one of those.

Fair enough. :cheers2:

There are many properties of the OpenEXR spec that I'm very fond of. But I could see how it's possible the specs might not be ideal for RT.

Although in 4 years our computers will be 8x faster. SSDs will be commonplace, GBs RAIDs will be affordable even at small studios and you'll have enough RAM in your average workstation to play back a minute of uncompressed 16bit 4k .rgb images. System memory buses will continue to increase in speed. With enough pure speed system wide any imperfections can be hidden by clever software.

I'm curious which will win. Standardizing a new file spec or advances in hardware.

It's always easy to fall into the trap of False Scarsity with computers. I ran into that this last week while archiving old footage. "Why am I spending half an hour to save $2 worth of storage?"

 

[Jonas Rejman]

An epic battle...

Thank you all, very informative...

 

[M Most]

Although in 4 years our computers will be 8x faster. SSDs will be commonplace, GBs RAIDs will be affordable even at small studios...System memory buses will continue to increase in speed...

Well, yes. And cars will get 50 MPG, cell phones will shoot HD video, and there will be Epics and Scarlets. But the reality is that right now none of that exists (well, I guess the 50MPG part does if you buy a new Prius...). Those of us doing real work have to live in the real world. Awareness of future developments is good and mildly important, but that awareness doesn't help get the job done today.

 

[Gavin Greenwalt]

Well, yes. And cars will get 50 MPG, cell phones will shoot HD video, and there will be Epics and Scarlets. But the reality is that right now none of that exists (well, I guess the 50MPG part does if you buy a new Prius...). Those of us doing real work have to live in the real world. Awareness of future developments is good and mildly important, but that awareness doesn't help get the job done today.

True. But software development lead times are often several years.

For instance if Caustic delivers their Raytracing card this year and it delivers as planned there are no 3D packages ready to use it to its full potential nor will there be for a couple of years. Always kind of dissapointing when the hardware beats the software. "All dressed up an no place to go".

 

[M Most]

True. But software development lead times are often several years.

For instance if Caustic delivers their Raytracing card this year and it delivers as planned there are no 3D packages ready to use it to its full potential nor will there be for a couple of years. Always kind of dissapointing when the hardware beats the software. "All dressed up an no place to go".

You mean like Red Rocket? :toetap05:

Sorry, couldn't resist.