IIF discussion thread

[Mark L. Pederson]

Okay, I suspect I'll regret starting this thread ... but what the hell. I've seen a number of posts lately referring to the IIF (Image Interchange Framework) - and I'm interested to get comments and possibly more information in one place.

MMost and Luki seem to be drinking the IIF Kool-Aid already - but I'm not getting a warm, fuzzy feeling just yet.

Maybe this is just because I don't know enough details. I'm an open minded guy and I've been wrong many times before and have said (and posted) plenty of stupid things - I just go with my gut. Maybe it's just that I have a general fear of ... Councils.

Anyway, my very basic, general comments/concerns are:

My understanding is that the file format is half float is not high bit... which would actually be 10-11 bits of real data spread over a larger range ... the more you spread the range, the more gaps you get in the data, etc. (10bits for data, 5 bits of exponent (spreading the range, not real data) and one sign data)

I have gotten the impression (and have heard comments) that the Council wants to see media forced into IFF format right after, or during acquisition. Not sure if this is true, but I can surely tell you that RAW, as an acquisition format has arrived. RAW workflows have arrived.

RAW is the new digital negative. RAW should stay RAW until export/render/delivery. I think we all know that, right?

I have a lot for respect for the Academy and it's members, but it's also a rather basic fact that there's a LOT of political power there. I wonder if there might be some "pressure" to adapt ...

Anyway, I'm all for putting a nail in the 10-bit LOG DPX coffin.

Convince me IIF is the right technology, and the right answer.

 

[Pawel Achtel]

Is there any technical information available?

 

[Sanjin Jukic]

Interesting quote from IFF "The Digital Dilemma" papers:

"The baseline storage costs used for this study are:

• $4.80 per physical item per year for archival storage
• $1.80 per physical item per year for working library
• $500 per terabyte per year for near-line data tape storage

...

The film-capture/digital finish production and 4K-captured productions produce
4K masters, and the 2K-captured productions produce 2K masters.(12)

Three copies of the digital master are assumed, given the recommended practice of data replication,
and this is consistent with the practice of archiving between two and five film masters,
although three is most typical: a YCM, a finished negative, and an interpositive. The
cost of producing the three film masters ($80,000 amortized over 100 years) is added
to the annual storage cost of film. The finished master is assumed to be 120 minutes
in duration for all scenarios; a shooting ratio of 25:1 is assumed as an industry average
to calculate the amount of source material, and two copies of all digital source material
is assumed to reflect current industry practice and insurance requirements.

Using current preservation methodology,

the cost of storing 4K digital masters was found to be

enormously higher – 1,100% higher – than the cost of storing film masters.

The overall costs increase further with the use of magnetic hard drive data
capture systems at 2K, and further still with 4K digital capture systems.

Although 1920 x 1080 capture using HDCAM SR videotape appears to be a costeffective
alternative to 4K, it is worth repeating that this cost reduction comes with a
corresponding reduction in certain performance characteristics relative to film. These
performance characteristics, and their effect on perceived
image quality, are beyond the scope of this
report, although there is significant and ongoing debate
about these tradeoffs. Furthermore, the decision to
migrate HDCAM SR source material would likely be
made in approximately 10 years.(13)

If the choice is made to copy the tapes to some newer videotape format
(assuming such a videotape format is developed in the future),
the cost to do so is estimated as follows:

• Total number of original production videotapes (from case study): 5,347
• Cost of new tape stock: $100 per tape
• Cost of copying to new videotape: $400 per tape
• Total cost of migration: 5,347 x ($100 + $400) = $2,673,500 (14)

----------------------------------------------------------------------

(12) For this report’s calculations, a 4K frame is composed of 4096 x 2160 pixels, 48 bits per pixel; a 2K frame is
composed of 2048 x 1080 pixels, 30 bits per pixel; and a 1920 x 1080 frame is composed of 1920 x 1080 pixels,
30 bits per pixel.

(13) The perceived longer “shelf life” of videotape as compared to data tape is attributed
to the (generally) longer useful life of videotape formats, lack of dependence on
computer hardware, operating systems and application software, and the use of
error concealment techniques to counter increasing bit error rates over time.

(14) Tape stock and copy costs based on current high volume dubbing rates for HDCAM SR tapes."

Get more>>>

 

[Carsten Fenstermacher]

The archival report continues to fail to factor in the constant storage density growth for a single piece of media and relative cost drop for duping over 100 years. The assumption that you'll be dubbing an SR tape to another SR tape every 5 years for the next 100 years is silly at best.

While the density goes up and cost goes down in $/TB, so does the duplication cost for each iteration every few years. So 100 tapes or drives for a production becomes 30 pieces, then 10 pieces, and so on until you start getting into multiple production for a single piece of media.

 

[Thomas Mathai]

The archival report continues to fail to factor in the constant storage density growth for a single piece of media and relative cost drop for duping over 100 years. The assumption that you'll be dubbing an SR tape to another SR tape every 5 years for the next 100 years is silly at best.

While the density goes up and cost goes down in $/TB, so does the duplication cost for each iteration every few years. So 100 tapes or drives for a production becomes 30 pieces, then 10 pieces, and so on until you start getting into multiple production for a single piece of media.

You are right, about the density growth and and price drop, but does the report mention that extra clones may be made?

I can see that if a 100 tapes/drives a few years later only need to take up 30 tapes/drives, then it would be logical to make an extra set or 2 of the 30 tapes/drives for safety.

I'm sure no one would want to destroy the original 100 tapes/drives, but there is the assumption that a percentage of the original 100 may be damaged or unrecoverable, so it would make sense to make extra sets of clones.

 

[Thomas Mathai]

Okay, I suspect I'll regret starting this thread ... but what the hell. I've seen a number of posts lately referring to the IFF (Image Interchange Framework) - and I'm interested to get comments and possibly more information in one place.

MMost and Luki seem to be drinking the IFF Kool-Aid already - but I'm not getting a warm, fuzzy feeling just yet.

Maybe this is just because I don't know enough details. I'm an open minded guy and I've been wrong many times before and have said (and posted) plenty of stupid things - I just go with my gut. Maybe it's just that I have a general fear of ... Councils.

Anyway, my very basic, general comments/concerns are:

My understanding is that the file format is half float is not high bit... which would actually be 10-11 bits of real data spread over a larger range ... the more you spread the range, the more gaps you get in the data, etc. (10bits for data, 5 bits of exponent (spreading the range, not real data) and one sign data)

I have gotten the impression (and have heard comments) that the Council wants to see media forced into IFF format right after, or during acquisition. Not sure if this is true, but I can surely tell you that RAW, as an acquisition format has arrived. RAW workflows have arrived.

RAW is the new digital negative. RAW should stay RAW until export/render/delivery. I think we all know that, right?

I have a lot for respect for the Academy and it's members, but it's also a rather basic fact that there's a LOT of political power there. I wonder if there might be some "pressure" to adapt ...

Anyway, I'm all for putting a nail in the 10-bit LOG DPX coffin.

Convince me IFF is the right technology, and the right answer.

Who knows if IFF is the right technology, I do know that there is a need for an intermediate format to deal with RAW from Red, Arri, SI and all future cameras.

Adobe created DNG to deal with the various RAW DSLR versions, maybe Cinema DNG will do the same.

http://labs.adobe.com/technologies/cinemadng/

Also is the Academy the only ones looking to create an intermediate format?

Is SMPTE working on this too?

 

[M Most]

MMost and Luki seem to be drinking the IFF Kool-Aid already - but I'm not getting a warm, fuzzy feeling just yet.

I don't like Kool Aid. Not the sugary kind, not the organizational kind, not the film kind, not the Red kind, and not the IIF (it's not IFF, that's an old graphics format that was used by New Tek for Lightwave renders....) kind. I like to judge things on merit, come up with pros and cons, and consider appropriate usage based on those pros and cons. That said...

My understanding is that the file format is half float is not high bit... which would actually be 10-11 bits of real data spread over a larger range ... the more you spread the range, the more gaps you get in the data, etc. (10bits for data, 5 bits of exponent (spreading the range, not real data) and one sign data)

You're confusing bit depth per channel with overall image processing capabilities. While the half float format (basically invented by ILM, which also came up with the Open EXR file format to contain it) does use 10 bits of basic data, the exponent and sign bits allow this to describe 30 stops of visual information, and an additional 10 stops beyond that with reduced precision in the lower values. It essentially allows a description of a vast range of exposures in order to allow for true HDR imaging that can be manipulated in post. This may sound a bit like RAW to you, but it actually goes considerably beyond that in the sense that it can contain values far beyond what a single sensor exposed once can achieve. So it goes far beyond the practical limits of any current individual camera. This allows for all of the super white and super black values one could ever practically have. Not to mention that the proposal is only that - a proposal, a work in progress. You can rest assured that many of the more influential members of the Council are well aware of the need for more bits, and are working to incorporate that into the final proposal, although the logic behind the 16 bit half float is pretty sound, not to mention that it just happens to be identical to Nvidia's half float format in their Cg graphics language. Not that this fact should be a determining factor, especially with the pace of technological innovation these days...

I have gotten the impression (and have heard comments) that the Council wants to see media forced into IFF format right after, or during acquisition. Not sure if this is true, but I can surely tell you that RAW, as an acquisition format has arrived. RAW workflows have arrived.

RAW is the new digital negative. RAW should stay RAW until export/render/delivery. I think we all know that, right?

No, we don't. At least I don't. RAW is a format that cannot directly be used for anything as it doesn't represent a visible, RGB image. Only after it is processed can it be useful. Now, I do agree that it does contain all of the original information that was captured. But if you've got a 12 bit linear sensor, and you use a proper debayering algrorithm, and keep the values in linear light, and use 16 bit precision math - all of which Red's software does - you wind up with an RGB image that is a direct representation of the RAW information. So there is no difference between the "compressed" version (i.e., the RAW record) and the "decompressed" version (i.e., the 16 bit RGB image) in terms of useful information. Now, once you get past that, there are choices as to how to practically use that RGB image depending on your post path, some of which compromise more of the data than others.

One of the things about the proposed ACES space is that it is based on what the Council is calling "scene linear" data - in other words, linear light values that are scene referred rather than output referred (the output is handled at the other end of the process). And although you might think that the Council is dominated by film related ideology, the use of scene linear data relates much more directly to digital capture than it does film. In effect, a Red camera already captures what the ACES space wants to see, so in that case, the input data transform applied to convert to ACES would do only very minimal transformations, mostly related to the "unlimited gamut" color space that ACES represents. Any data captured on film and provided via a film scan would need to undergo a "reverse engineering" via a more complex input data transform to yield information that is in the expected format, since film does not capture in linear light. The idea behind the proposal is to allow for a common container that does not compromise the ability of different capture systems to imprint a "look" specific to that capture system, but does eliminate the need to treat each device in a completely different manner in order to achieve proper results. The fact is that more and more, different capture systems and devices are being used, often on the same project, sometimes in the same scene. Many projects will be shot on a combination of devices - film, a Red camera, a Phantom, various video formats - and all of these formats currently need to be treated differently in order to be used together. The IIF eliminates the need for "secret sauce", and it eliminates the need for color correctors to operate differently for different formats, as well as the need to tailor the DI work for a specific primary output target. It allows for a highly expanded range of possible values and possible colors, regardless of the output device. And it allows for standardized transforms, provided by the manufacturers, to ensure that all input devices and all output devices work in a predictable manner.

I understand that you currently live in a very Red-centric world. But even on your projects, you're probably using things like the Phantom for certain situations, and you're sometimes buying stock footage that might have been shot on film or another digital camera. The fact is, though, that the film industry in general is not and likely will not be nearly as Red-centric as your company is. The problem isn't how to best handle one camera's footage, it's how to best handle all footage, regardless of the source, in a way that allows everyone to be on an even playing field, and in a way that provides more transparency to the original photography. That's what IIF is all about.

Convince me IFF is the right technology, and the right answer.

I long ago gave up trying to convince you of things, but hopefully I've provided a bit of food for thought.

 

[Ketch Rossi]

In deed IIF is in a way a reminder, to me at list of what I have recently gone trough with Stills, as I have being working with RAW, RC2, to be exact, and recently have being shooting with several other formats form various manufacturers of MF cameras, so I ended up believing that DNG was in deed the answer to one format for all.

I admit that DNG has brought several advantages to my workflow, and especially that of sharing files amongst the various media and agencies involved, but as I'm not really a technical kinda of guy, I have yet to see if as some say you do really loose important info that can't be recovered, and or if what you loose is just extra info that was not really that important to the final IQ any way, so for me the same now will stand for IIF.

But I have to say that even if there was a loss, if it was very marginal to the point of not really being able to tell the difference, well then I'll be all up for it, and I believe that with DNG vs. RAW I personally have not experience any issues with any one I have being working with.

 

[Lucas Wilson]

Oooo.... Mike and Mark having a discussion! (getting popcorn and a good seat...)

But in all seriousness...

Mark - the best way to get the information is to join the working group yourself and get the information firsthand! I'd be happy to provide an introduction if you need it.

Best,

Lucas

 

[Frank Cueto]

I don't like Kool Aid. Not the sugary kind, not the organizational kind, not the film kind, not the Red kind, and not the IIF (it's not IFF, that's an old graphics format that was used by New Tek for Lightwave renders....)

Hey, you seem to be very "up to date" with Amiga stuff :cheers2:

Unfortunately the .IFF was Image File Format and was a standard on the WorkBench OS since 1.0. It was eventually take over by A|W but only the .iff, nothing else was the same. so just a friendly heads up, .IFF was enver a Newtek thing (even Dpaint,Turbo Silver, Digipaint(from NT as well) used that format)

Nothing personal, just an ol'skool Amiga GFX dude at heart here ;-)

 

[Mark L. Pederson]

Mark - the best way to get the information is to join the working group yourself and get the information firsthand! I'd be happy to provide an introduction if you need it.

Best,

Lucas

Yes. Lucas, I'd love an introduction and I'd more than happy to seriously participate in a working group.

As you know first hand, I have ZERO issues laying out all my ideas and opinions - right, wrong or indifferent. But I guess this is why every year when I get asked to join a half dozen or so committees, groups and boards ... I always get "not voted in" by other members. Isn't there a Woody Allen line that goes something like "I would never wanna belong to any club that would have someone like me for member." ...?

 

[David Mullen ASC]

My impression of the Academy project was that they wanted to find a way of describing the intended color / color-correction for a movie in a digital way so that fifty years from now, we weren't trying to find a dye transfer print as a color reference for how the movie is supposed to look, which is the way old color movies are currently restored because you can't tell by looking at a negative (or a RAW file) what the final color effect was supposed to be. Even printer lights don't tell you that because they are unique to the individual lab and only work on the negative in its original state, so if it faded or color-shifted over time, the same set of printer lights at the same lab for the same print stock would not work. So the Academy is working on creating a set of values that describe the color, gamma, density, etc. that is independent of the equipment at particular post houses, etc.

 

[Mark L. Pederson]

I like to judge things on merit, come up with pros and cons, and consider appropriate usage based on those pros and cons.

As do I. You know Mike, I may poke a little fun at you now and then, but it's only because I feel we know each other well enough to do that. You have a LOT of experience (more than me) in the post world, and you have made many great contributing posts on this forum. So .. to be very clear ... I'm NOT trying to debate YOU and I'm not trying to say the IIF is a bad thing. I am clearly admitting I don't know enough.

You clearly know more than me about it (the IIF) - so ... by starting this thread ... I hope to learn more and maybe some other REDUSER's who are interested will learn more as well. You know, just like you, I know lots of people. I do know people who are directly involved and advising on a technical level, and I know people who are very excited about it and I know people who think it's doomed to fail. I don't know enough to give a hard core opinion ... but I have a hundred questions and the red flags are waiving. So ... I volunteer to play Devils Advocate as I don't have any Academy Awards hanging in the balance to worry about.

You're confusing bit depth per channel with overall image processing capabilities. While the half float format (basically invented by ILM, which also came up with the Open EXR file format to contain it) does use 10 bits of basic data, the exponent and sign bits allow this to describe 30 stops of visual information, and an additional 10 stops beyond that with reduced precision in the lower values.

I'm not saying that half-float is not a useful format, I think it is. But tell me ... in the current spec, is the mid grey point fixed? My understanding is that it is. And that alone, I strongly suspect, creates precision issues - specifically with various RAW formats.

It essentially allows a description of a vast range of exposures in order to allow for true HDR imaging that can be manipulated in post. This may sound a bit like RAW to you, but it actually goes considerably beyond that in the sense that it can contain values far beyond what a single sensor exposed once can achieve. So it goes far beyond the practical limits of any current individual camera. This allows for all of the super white and super black values one could ever practically have.

I've worked in OpenEXR before. There are lots of advantages to that file format, specifically multiple channels. But ... you are still talking about spreading the bits over a larger range. And you are still talking about "baking it in", right? It's totally possible that I'm just not wrapping my head around this the right way Mike, but it's very hard to lay down an opinion without the actual spec - which ... isn't done. And I am pretty sure Open EXR can also do FULL float in 32 or 24 bit, no?

Not to mention that the proposal is only that - a proposal, a work in progress. You can rest assured that many of the more influential members of the Council are well aware of the need for more bits, and are working to incorporate that into the final proposal, although the logic behind the 16 bit half float is pretty sound, not to mention that it just happens to be identical to Nvidia's half float format in their Cg graphics language. Not that this fact should be a determining factor, especially with the pace of technological innovation these days...

I have zero influence on the Council ... and I am very sure they need more bits.

No, we don't. At least I don't. RAW is a format that cannot directly be used for anything as it doesn't represent a visible, RGB image. Only after it is processed can it be useful. Now, I do agree that it does contain all of the original information that was captured. But if you've got a 12 bit linear sensor, and you use a proper debayering algrorithm, and keep the values in linear light, and use 16 bit precision math - all of which Red's software does - you wind up with an RGB image that is a direct representation of the RAW information. So there is no difference between the "compressed" version (i.e., the RAW record) and the "decompressed" version (i.e., the 16 bit RGB image) in terms of useful information. Now, once you get past that, there are choices as to how to practically use that RGB image depending on your post path, some of which compromise more of the data than others.

Yes, but ... tell me ... a "decompressed" version (i.e., the 16 bit RGB image) ... can you translate 100% of that into an half-float Open EXR file container?? With zero loss of information? Half-float is not linear and has one bit less precision everywhere, right? Again, not saying half-float is bad - just saying it's not RAW. Period.

Now .. with respect to "proper debayering algrorithm" that's a slippery, subjective slope. Isn't it interesting that RED can update the debayering algrorithm and suddenly we can get more from the same RAW image? I think your "direct representation" may be "direct" but it ain't RAW.

The idea behind the proposal is to allow for a common container that does not compromise the ability of different capture systems to imprint a "look" specific to that capture system, but does eliminate the need to treat each device in a completely different manner in order to achieve proper results.

This sounds 100% antithetical to working in RAW, specifically with different camera systems. More below ...

The fact is that more and more, different capture systems and devices are being used, often on the same project, sometimes in the same scene. Many projects will be shot on a combination of devices - film, a Red camera, a Phantom, various video formats - and all of these formats currently need to be treated differently in order to be used together. The IIF eliminates the need for "secret sauce", and it eliminates the need for color correctors to operate differently for different formats, as well as the need to tailor the DI work for a specific primary output target. It allows for a highly expanded range of possible values and possible colors, regardless of the output device. And it allows for standardized transforms, provided by the manufacturers, to ensure that all input devices and all output devices work in a predictable manner.

Huh? You don't need the "secret sauce" but the camera manufactures provide the "standardized transforms" (secret sauce) "to ensure that all input devices and all output devices work in a predictable manner. And the "predictable manner" is locking the mid grey?

I understand that you currently live in a very Red-centric world. But even on your projects, you're probably using things like the Phantom for certain situations, and you're sometimes buying stock footage that might have been shot on film or another digital camera.

Yes, OFFHOLLYWOOD is very RED-centric. But for the record, we just finished a shot-on-film feature DI and right at this very moment there are lab techs working in NYC on SI-2K footage and D21 footage - working directly from the RAW files. I'm no stranger to multi-format productions .. most are today but I fail to understand the advantage to taking RAW formats to a larger file size RBG container and loosing some information when I can just work directly from RAW files.

The fact is, though, that the film industry in general is not and likely will not be nearly as Red-centric as your company is.

You are correct. The film industry will not be. But the digital cinema industry will be.

I've seen things Mike ... things I would have never have believed possible.

The problem isn't how to best handle one camera's footage, it's how to best handle all footage, regardless of the source, in a way that allows everyone to be on an even playing field, and in a way that provides more transparency to the original photography.

The problem I have with "even playing fields" is that the lowest common denominator always brings the rockstars down a notch.

Tell me about color space and WHY IIF did not choose XYZ?

I long ago gave up trying to convince you of things, but hopefully I've provided a bit of food for thought.

Always listening, even when I roll my eyes.

 

[Sanjin Jukic]

I understand that you currently live in a very Red-centric world. But even on your projects,
you're probably using things like the Phantom for certain situations,
and you're sometimes buying stock footage that might have been shot on film or another digital camera.
The fact is, though, that the film industry in general is not and likely will not be nearly as Red-centric as your company is.
The problem isn't how to best handle one camera's footage, it's how to best handle all footage, regardless of the source,
in a way that allows everyone to be on an even playing field, and in a way that provides more transparency to the original photography.
That's what IIF is all about.



I long ago gave up trying to convince you of things, but hopefully I've provided a bit of food for thought.

All these things are going pretty hot now with this kind of notion at the time when we are preparing for New Year's Eve or Sylvester Night!!!!

Best New Year's wishes.

Sanjin

 

[Carsten Fenstermacher]

You're confusing bit depth per channel with overall image processing capabilities. While the half float format (basically invented by ILM, which also came up with the Open EXR file format to contain it) does use 10 bits of basic data, the exponent and sign bits allow this to describe 30 stops of visual information, and an additional 10 stops beyond that with reduced precision in the lower values. It essentially allows a description of a vast range of exposures in order to allow for true HDR imaging that can be manipulated in post. This may sound a bit like RAW to you, but it actually goes considerably beyond that in the sense that it can contain values far beyond what a single sensor exposed once can achieve. So it goes far beyond the practical limits of any current individual camera. This allows for all of the super white and super black values one could ever practically have. Not to mention that the proposal is only that - a proposal, a work in progress. You can rest assured that many of the more influential members of the Council are well aware of the need for more bits, and are working to incorporate that into the final proposal, although the logic behind the 16 bit half float is pretty sound, not to mention that it just happens to be identical to Nvidia's half float format in their Cg graphics language. Not that this fact should be a determining factor, especially with the pace of technological innovation these days...

Possible dynamic range in a storage format is not the same as the precision it can hold. Just like you can put 30 stops in a 10bit integer file, the precision will suffer. The total "precision" of half float is only 11bits at the end of the day. You can spread that out over many many stops nicely but it's still limited in precision. It's a bit like the difference between negative stock and chromes where chromes have more precision for fewer stops while neg has more stops and less tonal precision.

Nvidia is moving away from half to full float very quickly. It was a hack they did to support float without going to the full expense of full float. What's worse is that it's not even a CPU native format so everything has to
converted.

10bit cineon suffered because it was itself a hack to save space and half float in IIF will be similar.

 

[M Most]

So .. to be very clear ... I'm NOT trying to debate YOU and I'm not trying to say the IIF is a bad thing. I am clearly admitting I don't know enough.

Not a debate. A discussion. An exchange of opinions and information. The way more things here should be.

I'm not saying that half-float is not a useful format, I think it is. But tell me ... in the current spec, is the mid grey point fixed? My understanding is that it is. And that alone, I strongly suspect, creates precision issues - specifically with various RAW formats....But ... you are still talking about spreading the bits over a larger range. And you are still talking about "baking it in", right? It's totally possible that I'm just not wrapping my head around this the right way Mike, but it's very hard to lay down an opinion without the actual spec - which ... isn't done. And I am pretty sure Open EXR can also do FULL float in 32 or 24 bit, no?

Yes, it can. It also supports various compression schemes. And lots and lots of metadata. The problem isn't what can be put into the file, the problem is what can be put into the file and still allow real time playback at 24fps or better on common hardware. Now, hardware is always improving, so that is a good sign. And storage is also getting faster, so that's good, too. But for anything to be effective enough to provide real benefit to the industry, it must be useful now, not 5 years from now. So compromises are often made. The key is to craft a specification that is open ended with regard to file formats and color spaces, thus allowing a smooth evolutionary path as hardware becomes more capable. That is part of what the Council is trying to do.

Yes, but ... tell me ... a "decompressed" version (i.e., the 16 bit RGB image) ... can you translate 100% of that into an half-float Open EXR file container?? With zero loss of information? Half-float is not linear and has one bit less precision everywhere, right? Again, not saying half-float is bad - just saying it's not RAW. Period.

I think you're getting a bit too hung up on the half float thing. The file format and the encoding space are not the same thing. ACES stands for Academy Color Encoding Space. That is a description of a coding standard, not a file format. The current proposal for the file format is based on an existing format that works given the current state of hardware and allows for real time playback, an absolute necessity for the work it's going to be used for. By the time it is finalized, it could easily go to a deeper bit depth (and probably will, given those who are writing the spec) without affecting the underlying ACES encoding space, and without requiring any real changes in the Color Transform Language that is being developed to support it.

Now .. with respect to "proper debayering algrorithm" that's a slippery, subjective slope. Isn't it interesting that RED can update the debayering algrorithm and suddenly we can get more from the same RAW image?

I would say that you're not getting "more," you're just getting a better interpretation of what's there because they're applying more effective noise reduction and a better color matrix. You can't get "more" than what is there originally. At any rate, once you get into post production, you're working with what you have at that point in time. The chances that you're going to go back and do it all over again for minor gains in the future are pretty slim. So if you're getting essentially the maximum clean information that is possible given the state of the demosaic software at the time of post production, I would say that you're not really losing anything. And since Red is not permitting anyone else to use their own demosaic routines, there is only one yardstick to judge by.

I think your "direct representation" may be "direct" but it ain't RAW.

I think it's semantics. But we can agree to disagree on that.

Huh? You don't need the "secret sauce" but the camera manufactures provide the "standardized transforms" (secret sauce) "to ensure that all input devices and all output devices work in a predictable manner. And the "predictable manner" is locking the mid grey?

Once again, I would caution against getting hung up on the current state of the proposal and any specific file formats that currently exist. Look at the big picture. Don't completely ignore the details, but understand that the big picture is what's significant here. The details can evolve and can be ironed out by the time this is finalized.

Yes, OFFHOLLYWOOD is very RED-centric. But for the record, we just finished a shot-on-film feature DI and right at this very moment there are lab techs working in NYC on SI-2K footage and D21 footage - working directly from the RAW files. I'm no stranger to multi-format productions .. most are today but I fail to understand the advantage to taking RAW formats to a larger file size RBG container and loosing some information when I can just work directly from RAW files.

I might agree with you if there was only one RAW format and one compression scheme. But that's not the case, and likely will never be the case unless everyone just adopts something like DNG and some common compression codec. RAW formats are dependent on the sensors that are being used. As long as there are different sensors, with different sizes and shapes, there will be multiple RAW formats. What the Academy is trying to do, in part, is eliminate the need for dealing with endless formats, yet not lose significant information from any of them while going through a post path.

The problem I have with "even playing fields" is that the lowest common denominator always brings the rockstars down a notch.

Not if it's done right.

Tell me about color space and WHY IIF did not choose XYZ?

What makes you think they didn't? In the current proposal, ACES pixel values are defined as CIE 1931 XYZ tristimulus values. Headers in the metadata define the actual chromaticities being used, the specific white point, and thus the encoding space. The ODCT, or Output Device Calibration Transform, converts these to RGB representations that are specific to the output device being used. If you haven't read it already, you might want to look at this paper:

www.openexr.com/UsingOpenEXRandCTL.pdf

It was written in 2007, but it's reasonably detailed and directly refers to the proposal - albeit, the proposal as it was 2 years ago.

 

[M Most]

10bit cineon suffered because it was itself a hack to save space and half float in IIF will be similar.

One could just as easily say that Redcode is a hack to save space and support 4K without going to the expense of actually writing 4K. Just because something is intended to use less space doesn't mean it isn't useful, and it doesn't mean the results it achieves are unusable or unacceptable, especially when it brings other things to the table, such as the ability to write and store HDR.

 

[D Fuller]

Isn't there a Woody Allen line that goes something like "I would never wanna belong to any club that would have someone like me for member." ...?

It was Groucho Marx who said that. (or something pretty close to that). Woody may have quoted him.

 

[Blair S. Paulsen]

IIF sure looks like a major step forward

IIF sure looks like a major step forward

There is nothing wrong with questioning the motivations that drive the setting of standards. I think a case could be made that the Cineon log encoding scheme inside the DPX container was developed by Kodak to give film scans a good showing in Digital Intermediate environments. FWIW the fact that the Academy is making an effort to create a standard that might actually favor linear light source material over log seems like a good sign. Yes, I know, the devil maybe lurking in the details.

In any case, the IIF appears to be a huge improvement over 10 bit DPX and much closer to a level playing field. I am a big fan of the core concept of creating a benchmark (ACES) that all parties can aim at. While top post facilities can and do manage to cut together disparate sources quite nicely, IIF ought to make that much easier.

While IIF may not be perfect (what is), I think the positives outweigh the negatives by a wide margin. Just my .02

Cheers - #19

 

[Mark L. Pederson]

I think you're getting a bit too hung up on the half float thing. The file format and the encoding space are not the same thing. ACES stands for Academy Color Encoding Space. That is a description of a coding standard, not a file format. The current proposal for the file format is based on an existing format that works given the current state of hardware and allows for real time playback, an absolute necessity for the work it's going to be used for.

And to my knowledge - this is still half-float.

Can you clarify for me if IIF will use OpenEXR as a container or a file format? These are two very different thing as you know. For example - S.Two records ARRI RAW into a .DPX file container. Sure, you can open that .DPX file on any application that supports DPX but you won't see the image displayed correctly. So ... with IIF - can I open the file in any application that has OpenEXR support and the image will display correctly?

By the time it is finalized, it could easily go to a deeper bit depth (and probably will, given those who are writing the spec) without affecting the underlying ACES encoding space, and without requiring any real changes in the Color Transform Language that is being developed to support it.

Okay ... so "probably" I shouldn't be concerned about bit-depth and precision of processing?

Once again, I would caution against getting hung up on the current state of the proposal and any specific file formats that currently exist. Look at the big picture. Don't completely ignore the details, but understand that the big picture is what's significant here. The details can evolve and can be ironed out by the time this is finalized.

Mike, it's all about the details. Which is why, this thread is most likely doomed. There are not enough details in the proposal and it's not done.

When does this get finalized? Do industry professionals outside the Academy get to test this prior to it being published and/or "finalized"?

I might agree with you if there was only one RAW format and one compression scheme. But that's not the case, and likely will never be the case unless everyone just adopts something like DNG and some common compression codec. RAW formats are dependent on the sensors that are being used. As long as there are different sensors, with different sizes and shapes, there will be multiple RAW formats. What the Academy is trying to do, in part, is eliminate the need for dealing with endless formats, yet not lose significant information from any of them while going through a post path.

But if we have the tools to work with all these different RAW files - color grading directly from the RAW files, it's just not logical to me to convert them all to a large RGB file format first.

And the white paper says:

Digital Projector and DCDM Output DCDM-encoded images for a digital projector are created by starting with ACES values and applying an
RT, followed by an ODT that converts from OCES to the DCDM encoding.


So ... we are all ending up in 16-bit TIFF in the end, right? At least for digital cinema, right?

So I'm thinking ...

OPTION A:

1) Original images from digital cameras, scanned film negatives, and 3D computer graphics are imported into the ACES.
2) All digital image files contain ACES pixel data. Image processing under user control, such as compositing, color correction or geometric transformation, operates on ACES images and generates new ACES image files.
3) Displaying an ACES image on an output device requires color rendering to achieve the desired look.
4) ACES files render to DCDM (16-bit TIFF)

vs. OPTION B.

1) original RAW camera media is image processing under user control, such as compositing, color correction or geometric transformation operates in 32-bit float.

2) Convert to XYZ color space on render to 16-bit TIFF for DCDM.

What makes you think they didn't? In the current proposal, ACES pixel values are defined as CIE 1931 XYZ tristimulus values. Headers in the metadata define the actual chromaticities being used, the specific white point, and thus the encoding space. The ODCT, or Output Device Calibration Transform, converts these to RGB representations that are specific to the output device being used.

Well ... ACES uses CIE 1931 XYZ tristimulus values but the way they describe it - I don't think it's "XYZ color space". I need to get some clarification there.