IIF discussion thread

[JanneJansson]

This is why I love the place.
..keep goin, don't mind me

 

[M Most]

Can you clarify for me if IIF will use OpenEXR as a container or a file format? These are two very different thing as you know. For example - S.Two records ARRI RAW into a .DPX file container. Sure, you can open that .DPX file on any application that supports DPX but you won't see the image displayed correctly. So ... with IIF - can I open the file in any application that has OpenEXR support and the image will display correctly?

The file format is based on a subset of Open EXR, with some of the additional metadata slots and some of the compression options removed to allow for real time playback. ILM has already designed such a subset and had made available a player to test the real time playback capabilities. Since ACES is its own color space, one would presumably need ACES enabled software to view it properly. However, unlike Arri Raw, it does not require any sort of debayering as it's already RGB image data, so the problem you're talking about with S.two's DPX wrapped Arri files won't exist.

Okay ... so "probably" I shouldn't be concerned about bit-depth and precision of processing?

I would say to keep an open mind at the moment.

Mike, it's all about the details. Which is why, this thread is most likely doomed. There are not enough details in the proposal and it's not done.

One could say the same thing about most of the upcoming products from Red. That doesn't seem to stop people from talking about it, though.

When does this get finalized? Do industry professionals outside the Academy get to test this prior to it being published and/or "finalized"?

The proposal is intended to present a framework to the industry and to SMPTE for eventual specification and standardization. The common method of doing this has always involved testing and comments from any interested parties. I'm sure this will be the case with IIF.

But if we have the tools to work with all these different RAW files - color grading directly from the RAW files, it's just not logical to me to convert them all to a large RGB file format first.

This isn't about any particular individual or facility. It's about the industry as a whole. Interchange between facilities is important, and it's becoming more important as facilities seem to be contracting and more players are sitting at the table. Common standards are the key to real progress that everyone can benefit from, and they are the key to being able to spread work across various facilities - possibly around the globe - and ensure that everyone is seeing the same thing and turning in the same thing. Not to mention the ability to put multiply sourced images through a common pipeline for both efficiency and general sanity. But in the end, nobody is going to stop you or anyone else from deviating from the common standard if it's adopted. If you want to go on dealing with multiple RAW formats and the constant need to debayer them - often with different debayering methods and routines - and combining those with non-Bayer material in a given project, then go right ahead and do that. But I believe that the industry will ultimately come together and "stop the madness," so if one wants to be able to exchange with others, they will likely need to support the common format and methods. However, as I said, it's up to you.

And the white paper says:

Digital Projector and DCDM Output DCDM-encoded images for a digital projector are created by starting with ACES values and applying an
RT, followed by an ODT that converts from OCES to the DCDM encoding.


So ... we are all ending up in 16-bit TIFF in the end, right? At least for digital cinema, right?

Yes. But if you look at this, it's really no different than what most facilities are doing right now for film DI's. You come in with film, convert it to a 10 bit log representation (an "input transform"), you color correct and preview that color correction (a "render transform") and you go through a print preview LUT to view it properly (an "output device transform"). The steps are basically the same, with the only real exception being that in IIF, the render happens in a much wider color space.

So I'm thinking ...

OPTION A:

.....

vs. OPTION B.

As on a previous post, I would say again that you seem to be living in a more Red-centric world than most of the players in the DI game. If one is going to come up with methodologies that can be shared and widely supported, one cannot base it on one technology. Not every camera is going to be based on Bayer pattern chips. That may be a nice technology for the moment for some camera companies, but there is still going to be a lot of imagery that is captured on things like film, three chip designs, and different matrix patterns. Even if one were to assume that RAW records would be available for all of the electronic devices, they will not be the same and will not be treated the same. And even if they are, you still have film to contend with. Fact is, the IIF is not being designed for a digital only world. It is being designed for anything that might already exist or might come along in the future. To base a future standard on one specific technical approach is to ignore the larger world. For some - you, for example - this might be less of an issue because you've already made your technology choice. But for others - many others - it's just not logical to boil the entire world down to one capture method. Who knows, in the future we might not even be capturing images. Perhaps we'll be capturing light field records instead, and deriving images from that. I for one don't believe that industry standards should be based on one methodology, especially when the pace of technology almost dictates there will be new and better approaches in the future. 20 years from now we might be reminiscing about Bayer pattern chips, and what a goofy idea they were. Who knows?

Well ... ACES uses CIE 1931 XYZ tristimulus values but the way they describe it - I don't think it's "XYZ color space". I need to get some clarification there.

The ACES color space is a theoretical unlimited color space that encompasses the entire XYZ spectrum. It goes beyond even what is described by P3, and is intended to be a representation that goes beyond all current and future display technology.

 

[Gavin Greenwalt]

Everything goes to a universal linear gamma [Check]
Half Float [Check]
Super Whites, Super Blacks [Check]
Multiple compression schemes [Check]

Sign me up.

I've already locked horns with Sanjin over this but everything going Linear I think is the correct direction moving forward. It takes out all the guess work over whether the image is REDLog, Filmstream, PDLog, S-Log and every other new specialty gamma curve created. Just open the file and it's right.

 

[Lucas Wilson]

Mike, it's all about the details. Which is why, this thread is most likely doomed. There are not enough details in the proposal and it's not done.

Mark, I respect your passion but since you're not part of the committee working on the format, there's no real way for you to know what the current details are. Like I said in an earlier thread, I'd be happy to get you set up with the Committee...

When does this get finalized? Do industry professionals outside the Academy get to test this prior to it being published and/or "finalized"?

The committee that has been working on this is comprised of working technical professionals and executives at the Facility, Manufacturer, and Studio level, as well as camera companies, high-level working DPs, and data professionals.

The Sci-Tech Council is a who's who of actual working professionals... who happen to be Academy members. Daryn Okada... Phil Feiner... Garrett Smith... George Joblove... these folks know just a little about the guts of production and post-production and what needs to go into this format and why.

So the short answer is, yes - lots of industry professionals at all levels of the industry are deeply involved in the specs, testing, and implementation of the format.

Lucas

 

[Mark L. Pederson]

The committee that has been working on this is comprised of working technical professionals and executives at the Facility, Manufacturer, and Studio level, as well as camera companies, high-level working DPs, and data professionals.

The Sci-Tech Council is a who's who of actual working professionals... who happen to be Academy members. Daryn Okada... Phil Feiner... Garrett Smith... George Joblove... these folks know just a little about the guts of production and post-production and what needs to go into this format and why.

So the short answer is, yes - lots of industry professionals at all levels of the industry are deeply involved in the specs, testing, and implementation of the format.

Lucas

Lucas -

I'm NOT against IIF - that's for sure. I just don't know enough to get excited yet and have lots of concerns and confusion about some of the approach. Like I said - I'm sure I'll regret starting the thread - but excuse me ... I'd like to know more.

Obviously, little old me does not have any where near the track record or credibility of folks like Mr. Okada. I just want to understand more - and I do that by asking questions. I'm just not the kind of guy that is going to say to myself "they have the best team in the world - of course they will win". I guess my question really should have been - will this get to be tested by industry professionals OUTSIDE the committee? Will they publish any updates to the public prior to being "done"?

What is the theoretical timeline for this being implemented?

 

[JanneJansson]

Everything goes to a universal linear gamma [Check]
Half Float [Check]
Super Whites, Super Blacks [Check]
Multiple compression schemes [Check]

Sign me up.

Yep, me too. Been playing with the Open EXR for many years and I have not under stood why everyone used DPX, and with a framework for display and transfer, it will be a backbone for the industry.

Hate gamma curves, linear rules. :crazy:

..but "IFF" is bad name. There are are too many ".iff" file formats already. Amiga IFF, NewTek IFF, Maya IFF ...

 

[Lucas Wilson]

Lucas -

I'm NOT against IIF - that's for sure. I just don't know enough to get excited yet and have lots of concerns and confusion about some of the approach. Like I said - I'm sure I'll regret starting the thread - but excuse me ... I'd like to know more.

Obviously, little old me does not have any where near the track record or credibility of folks like Mr. Okada. I just want to understand more - and I do that by asking questions. I'm just not the kind of guy that is going to say to myself "they have the best team in the world - of course they will win". I guess my question really should have been - will this get to be tested by industry professionals OUTSIDE the committee? Will they publish any updates to the public prior to being "done"?

What is the theoretical timeline for this being implemented?

All of these are good questions and are answered... by being part of the group of people working on it! : ) I'm working on it, and I'm not a member of the Sci-Tech Committee. There are a lot of people involved who are not members of the Academy, or Sci-Tech members.

Reduser is not the right place to discuss most of the relevant details for this particular issue.

Lucas

 

[Mark L. Pederson]

Yep, me too. Been playing with the Open EXR for many years and I have not under stood why everyone used DPX, and with a framework for display and transfer, it will be a backbone for the industry.

Hate gamma curves, linear rules. :crazy:

Well ... I'm 100% for OpenEXR over DPX anyday - that is a no-brainer for sure - and as you know nothing stops any of us from using it right now. But, this seems to be a "constrained" flavor of the OpenEXR file format ... so ... I'm just concerned about HOW it will be constrained, bit-depth and precision - how ACES will encode and write to it.

 

[Mark L. Pederson]

Reduser is not the right place to discuss most of the relevant details for this particular issue.

Lucas

Well ... I did post in "off topic". :emote_headwall:

 

[Dan Hudgins]

Print Stocks

Print Stocks

My impression of the Academy project was that they wanted to find a way of describing the intended color / color-correction for a movie in a digital way so that fifty years from now, we weren't trying to find a dye transfer print as a color reference for how the movie is supposed to look, which is the way old color movies are currently restored because you can't tell by looking at a negative (or a RAW file) what the final color effect was supposed to be. Even printer lights don't tell you that because they are unique to the individual lab and only work on the negative in its original state, so if it faded or color-shifted over time, the same set of printer lights at the same lab for the same print stock would not work. So the Academy is working on creating a set of values that describe the color, gamma, density, etc. that is independent of the equipment at particular post houses, etc.

I would add that you cannot make a print today from older negatives even if they have not faded that will match an original print (even at the same lab on the same printer with the same lights numbers tape), todays print stocks have different curves and dyes from older print stocks are higher resolution but have less sliver and are higher contrast.

It is imposable to match any original grading on todays print stocks they do not "mate" with the curves used on older print stocks, so any new prints need to be re-graded. Yes it is good to have a stable positive use as a guide, but a guide is all it can be since you cannot film-to-film re-print older negatives on newer print stocks and match the look of the older print stocks, and CD-1 is no longer used etc.

 

[Mark L. Pederson]

If you want to go on dealing with multiple RAW formats and the constant need to debayer them - often with different debayering methods and routines - and combining those with non-Bayer material in a given project, then go right ahead and do that. But I believe that the industry will ultimately come together and "stop the madness," so if one wants to be able to exchange with others, they will likely need to support the common format and methods. However, as I said, it's up to you.

Hmmmm ... "the constant need to debayer RAW" .... that's one way to put it.

How about "debayer when I want to see it, or need to see it, in real time, at a resolution and quality I choose" as opposed to "debayer it all first - to a much larger RGB file format".

It just seem a bit "anti-RAW" - especially at a time when many high end DI systems are supporting additional RAW formats natively and we are seeing amazing things being done on GPUs, etc. But like you said - we all have a choice.

And again - I'm all for the end of 10-bit LOG DPX and HDCAM SR tape.

 

[Gavin Greenwalt]

I don't feel it's Anti-Raw to propose an image format which actually contains a defined image.

The trouble with RAW is that it requires a camera manufacturer's 'magic sauce' to produce a 'correct' image. And while many grading applications are adding debayer algorithms at some point it has to become RGB in order to be manipulated anyway. If consistency is the goal then it makes sense to stay in RGB through the entire process and not open up your data to disparate interpretations of RAW. When I'm compositing I'm always a bit ancy about getting RAW. I love it for its flexibility but I also get this nagging feeling that I have more control over the image than I should. "How exactly did the DP envision the color and exposure for this plate?" EXRs give me the security that "This is the way it should look, but here is the rest of the data in case you need it and we made a mistake."

The advantage of RAW is that you can go back to it and run it through new and improved debayer algorithms/color science to extract a more pleasing image. But you would lose that advantage anyway if you kept the image RAW. Every application would need to implement a standardized debayer algorithm and that means it would be relatively frozen in time. A RAW image with a frozen debayer algorithm isn't really any more flexible than a debayered image.

In my experience RAW usually becomes RGB immediately anyway so that we don't have to debayer-on-demand which is extremely processor intensive.

The ultimate goal of this format I hope is that it's note-free. That anyone can simply open it without any external information and get the correct data. You should require no special insight into the nature of the footage when opening it. And it sounds like that's one of their big goals so I look forward to the committee's findings.

 

[M Most]

It just seem a bit "anti-RAW" - especially at a time when many high end DI systems are supporting additional RAW formats natively and we are seeing amazing things being done on GPUs, etc. But like you said - we all have a choice.

Yes, you do have a choice. However, I would also offer the comment that it's easy to sit back and criticize, but it's a lot more useful to try and be part of the solution, which I hope you ultimately decide to do. I think that's what Lucas is trying to say in slightly different words.

Personally, while I'm not a member of the Committee (I'm not a Motion Picture Academy member, although I've been a Television Academy Member for a number of years...), I ask those who are a lot of questions when I see them. But I also feel that people like Phil Feiner, Jim Houston, Ray Feeney, and Andy Maltz know a lot more about the scientific details of all this stuff than I do. Having said that, I have also found them to be pretty open to comments and suggestions.

 

[Deanan]

While the half float format (basically invented by ILM, which also came up with the Open EXR file format to contain it) does use 10 bits of basic data, the exponent and sign bits allow this to describe 30 stops of visual information, and an additional 10 stops beyond that with reduced precision in the lower values.

Can you elaborate on what you mean by reduced precision for the 10 stops with lower values?

Everything goes to a universal linear gamma [Check]

Kindof... It's more like log encoded linear with an auto delog because Half float is essentially
a 15bit log encoding (for positive values, 16 bits including negative values).

If you're storing linear to light values in half float, each stop will give you 10 bits of precision per stop.

Zero to 1.0 will store 13 stops of linear light data encoded as log.

Zero to 65505 (last possible positive value) will store about 32 stops of linear light as log.


0 to 0.0049 = 12 bits of log encoded values (4 stops @ 10bit/stop precision)
0 to 1.0000 = 13-14 bits of log encoded values (13 stops @ 10bit/stop precision)
0 to 2.0000 = 14 bits of log encoded values (14 stops @ 10bit/stop precision)
0 to 65504 = 15 bits of log encoded values (32 stops @ 10bit/stop precision)
65505 = infinity
(and the opposite for the negative sign bit)


It's debatable if this is better than 16bit integer log where you'd have plenty
of room to set black/white points for safety. For CPUs, 16bit int would be
better than half float and for gpu's you're bumping it up to 32bit float for both in most cases.


The attached image shows the plot for 0.0 to 1.0 and from 1.0 to 65505.

 

[Mark L. Pederson]

However, I would also offer the comment that it's easy to sit back and criticize, but it's a lot more useful to try and be part of the solution, which I hope you ultimately decide to do. I think that's what Lucas is trying to say in slightly different words.

I get it. But you know, I don't think I'm "sitting back and criticizing". I'm asking questions and raising some concerns that I don't see anyone else raising.

I re-read the document last night (yes I know it's old) - and there's a LOT of transforms happening - so ... obviously precision is crucial. As you know, when you take a data from a small space, spread it into big space - then back into a smaller space .... well ... it's all about the precision.

I have a very open mind. I'd like to be part of the "solution", even though I don't entirely agree on the scope of the "problem". There are goals in the IIF proposal I would kill for.

I've asked for an introduction to the Committee so I will continue to try to learn and follow the developments here.

 

[JonathanF]

RAW is a format that cannot directly be used for anything as it doesn't represent a visible, RGB image. Only after it is processed can it be useful. Now, I do agree that it does contain all of the original information that was captured. But if you've got a 12 bit linear sensor, and you use a proper debayering algrorithm, and keep the values in linear light, and use 16 bit precision math - all of which Red's software does - you wind up with an RGB image that is a direct representation of the RAW information. So there is no difference between the "compressed" version (i.e., the RAW record) and the "decompressed" version (i.e., the 16 bit RGB image) in terms of useful information. Now, once you get past that, there are choices as to how to practically use that RGB image depending on your post path, some of which compromise more of the data than others.

One of the things about the proposed ACES space is that it is based on what the Council is calling "scene linear" data - in other words, linear light values that are scene referred rather than output referred (the output is handled at the other end of the process). And although you might think that the Council is dominated by film related ideology, the use of scene linear data relates much more directly to digital capture than it does film. In effect, a Red camera already captures what the ACES space wants to see, so in that case, the input data transform applied to convert to ACES would do only very minimal transformations, mostly related to the "unlimited gamut" color space that ACES represents. Any data captured on film and provided via a film scan would need to undergo a "reverse engineering" via a more complex input data transform to yield information that is in the expected format, since film does not capture in linear light. The idea behind the proposal is to allow for a common container that does not compromise the ability of different capture systems to imprint a "look" specific to that capture system, but does eliminate the need to treat each device in a completely different manner in order to achieve proper results. The fact is that more and more, different capture systems and devices are being used, often on the same project, sometimes in the same scene. Many projects will be shot on a combination of devices - film, a Red camera, a Phantom, various video formats - and all of these formats currently need to be treated differently in order to be used together. The IIF eliminates the need for "secret sauce", and it eliminates the need for color correctors to operate differently for different formats, as well as the need to tailor the DI work for a specific primary output target. It allows for a highly expanded range of possible values and possible colors, regardless of the output device. And it allows for standardized transforms, provided by the manufacturers, to ensure that all input devices and all output devices work in a predictable manner.

The one thing that this ignores is that debayering algorithms are continuing to improve and just as we have seen Graham's 'color science' evolve, we are certain to see this continue as vendors develop more and more expertise in working with raw sensor data.

If you debayer the data will you be able to take advantage of subsequent algorithmic advances in debayering technology?

Food for thought,

J

 

[JonathanF]

But in the end, nobody is going to stop you or anyone else from deviating from the common standard if it's adopted. If you want to go on dealing with multiple RAW formats and the constant need to debayer them - often with different debayering methods and routines - and combining those with non-Bayer material in a given project, then go right ahead and do that. But I believe that the industry will ultimately come together and "stop the madness," so if one wants to be able to exchange with others, they will likely need to support the common format and methods. However, as I said, it's up to you.

I'm not taking pot shots at you mike, I'm just making a point here by observing that you're making debayering sound like a difficult and/or undesirable thing.

Consider that with CUDA you can decode and debayer raw datastreams in real time quite easily. It was shown at IBC with multiple streams of 4k being read and decoded to graphics in real time on current graphics hardware.

Because the compressed and undebayered datastream is quite small you reduce the load on networked filesystems and networks dramatically over using something like OpenEXR.

Having worked on and designed workflows (and facilities) for a number of features and I have to say that the increasing storage and bandwidth requirements are staggering. The last feature I did with OpenEXR as the principle data format had 80TB of network attached storage with 48 gigabit pipes fed directly into the network core for that one film (that was a relatively small ~$40M budget film). There were something like 288 spindles in the filesystem and it simply wasn't enough even with all the caching being done.

Now I will admit, had CUDA been an option then we might have explored higher levels of compression on the OpenEXR files like PXR24 which is lossless at 1/2 float precision, but we did not have that available to us then. For that reason this is in all fairness an apples to oranges comparison.

But, I for one am happy to be working with RAW formats that only consume 30-50MB/sec. I know Arri is working on something similar for the new cameras and I suspect it will also have a fairly narrow bandwidth requirement so card based memory can be used in the cameras.

I'm not saying that an interchange format isn't necessary at all, I see the obvious advantages of having such a format even from altitude.

As I mentioned earlier though, I like having the ability to "upgrade" footage as the color and debayering algorithms improve over time.

One concession I should make is that once footage is out to post houses for VFX all this lovely raw workflow through to grading goes out the door.

Keep the comments coming, I'm enjoying the conversation.

J

 

[M Most]

The one thing that this ignores is that debayering algorithms are continuing to improve and just as we have seen Graham's 'color science' evolve, we are certain to see this continue as vendors develop more and more expertise in working with raw sensor data.

If you debayer the data will you be able to take advantage of subsequent algorithmic advances in debayering technology?

While that sounds good and important, the truth is that when a project is done, it's done, in almost every case. The only time what you're saying might be significant is in the case of a restoration, but restoration is only necessary because today's older elements are either physically damaged, significantly faded, or lost. Those are things that will not happen with digitally stored files (except for the "lost" part). You won't need to improve anything if it isn't degraded in the first place, which of course is the ultimate promise of digital finishing. Not to mention the cost of redoing a project from scratch for some minor improvement in a product the original creators were happy with in the first place.

So while what you're saying isn't completely untrue, I don't think it's particularly significant in the market we're talking about.

 

[M Most]

But, I for one am happy to be working with RAW formats that only consume 30-50MB/sec. I know Arri is working on something similar for the new cameras and I suspect it will also have a fairly narrow bandwidth requirement so card based memory can be used in the cameras.

You (and Mark) are assuming that the IIF proposal is an either/or, a complete replacement for everything you know and now do. I don't see that as the case. My guess is that there will be developments in color correction systems that will allow a live debayered output from a RAW file to be piped through an ACES transform and dealt with directly, rather than do a pre-conversion to an ACES file. This will yield a system much like what we have now, except that right now we deal with a somewhat restricted RGB source space. If you consider what is happening in Scratch, or Baselight, or any other system that uses "native" files, what they are doing is a series of steps - debayer to RGB, pipe that through an input transform if necessary, and then pipe that into the color correction engine. The only difference between this and an ACES world is the intermediate format that the color correction engine sees, and that is really just a matter of using different transforms, probably using GPU acceleration. So what you want and what I'm talking about could and likely would be the same thing. The actual Open EXR based file format would be used for exchange purposes, not necessarily for color correction done from a conform of the original RAW files. If you're not sending files out to some other facility or VFX house, you might never even touch the file format itself.

I'm not saying that an interchange format isn't necessary at all, I see the obvious advantages of having such a format even from altitude.

As I mentioned earlier though, I like having the ability to "upgrade" footage as the color and debayering algorithms improve over time.

Once again, I don't see any real world practical, commercial applications for that (other than on your own personal projects), but if you do, that's fine.

One concession I should make is that once footage is out to post houses for VFX all this lovely raw workflow through to grading goes out the door.

That's the main purpose for the file format, common interchange between facilities. Hence the name "Image Interchange Framework". As I outlined above, it's not necessarily the way DI work would have to be done, except when you're sourcing VFX or other files that are not original camera footage. This is one reason I've said to Mark that one shouldn't be as concerned with the specific file format as they are with the big picture of what the proposal is trying to accomplish. It's attempting to eliminate the problems of exchanging material between different facilities in a way that the material is self descriptive, carries all critical information (including color correction, BTW - the Color Transformation Language is an integral part of the proposal), displays properly when viewed with any supporting software on any supported monitor setup, and is capable of real time playback when needed. If you want a basis for comparison, this is not very different from the situation we've had with Red. Some manufacturers (and facilities) adopted the working methodology of converting to DPX files prior to DI grading. Others went the route of doing a "live debayer" into the color correction engine from the R3d files. The IIF proposal allows for both, but with the added advantages of a standard working color space that has an essentially unlimited gamut, a standard method of rendering that will ensure proper results (something Red post methods have struggled with), and a way of passing images between users without the need for supplemental information or specialized software. I think that's well worth supporting and developing.

 

[JonathanF]

You (and Mark) are assuming that the IIF proposal is an either/or, a complete replacement for everything you know and now do. I don't see that as the case. My guess is that there will be developments in color correction systems that will allow a live debayered output from a RAW file to be piped through an ACES transform and dealt with directly, rather than do a pre-conversion to an ACES file. This will yield a system much like what we have now, except that right now we deal with a somewhat restricted RGB source space. If you consider what is happening in Scratch, or Baselight, or any other system that uses "native" files, what they are doing is a series of steps - debayer to RGB, pipe that through an input transform if necessary, and then pipe that into the color correction engine. The only difference between this and an ACES world is the intermediate format that the color correction engine sees, and that is really just a matter of using different transforms, probably using GPU acceleration. So what you want and what I'm talking about could and likely would be the same thing. The actual Open EXR based file format would be used for exchange purposes, not necessarily for color correction done from a conform of the original RAW files. If you're not sending files out to some other facility or VFX house, you might never even touch the file format itself.

For people like me this would be the ideal.

It's a very interesting and compelling point and I'm glad you made it. So glad in fact I think I'm going to pass this comment along to a couple developers I know who work on grading systems and get their thoughts on this. I can't imagine why they wouldn't agree with you.

Perhaps Lucas Wilson might like to make a comment on this.

J