DPX to post house for grade

[Daniel Katz]

I have frequented this site for a while but have only just begun to have the Red projects that I am shooting go for high end grading. Please forgive me if this has been answered somewhere else on this forum. I recently shot a couple of 30 second spots for a production company on the Red. The editor cut the spots with the Red M-proxies.
The producers are keen to do a high end grade probably at Company 3 in NY. I am pretty sure Company 3's preference for material is in the DPX file format. I have never organized a color correct from Red footage ( especially proxies) , through Final Cut and then Color for a DPX output before to be handed off for a color grade. Has anyone overseen a similar workflow that could enlighten me on the pitfalls that one can face.
I'm assuming that once the timeline is locked I can send to Color direct from Final Cut Studio 3, from Color I can render and output the necessary DPX files and use the gather media option to contain these in one single folder? Moreover....
Is it customary for the house to perform a marriage with the Wav sound to provide us with a mov file for final output flexibility? Or more fundamentally; what format should we come away from the grade with to provide the production company with the most flexibility for broadcast and web distribution;bearing in mind they operate with Final Cut Studio and Color primarily and don't really have experience generating or working with DPX files or third party software like Gluetools.
I apologize for such a basic question but I am having trouble pinning down all the elements required and the correspondence from the post house has been less than clear.
Many thanks in advance

 

[Tim Whitcomb]

?? First one should try to avoid shooting RED without knowing exactly what your deliverable is... BEFORE you shoot. RAW workflow is always best reverse engineered.

What is the deliverable? DVD? Broadcast Safe TV? Web? SOunds like a PRORES4444 output will suffice, even with a "high end grade" as you call it.

DPX workflow is one of the least favored workflows for RED RAW (although industry standard for other formats) because most hgh end finishing systems Scratch, pablo, baselight, etc will work natively with R3d's now and simply need an EDL from FCP.
and will get a FULL 4K debayer and better images. (unless you transcode via Red Rocket 4K full debayer to DPX)

lots to learn on these boards. But rendering out from Color to DPX to a high end workflow sounds like a waste of time and will not give you the best results.

If they can afford a high end grade, then perhaps they can hire a post supervisor well versed in RED workflow? That's my advice.

Dont mean to bust your chops on the first post... but you need to sit down with the high end post house and work through the deliverables. Or just keep it in color and hire a Red Rocket to transcode the selects in FCP to PRORES4444 and then Export to Color. Its a simple workflow good enough for many LA high end post houses delivering to TV, DVD and web.

good luck

 

[Dan Hudgins]

DPX vs. TIF

DPX vs. TIF

I am working on DPX support now and have found that DPX is one of the most variable formats:

1) No "fixed" black level

2) No "fixed" white level

3) Can be about gamma 1.7 or use "soft sholder"

4) Can range over 10 bits or have BOTH white and black clip closer to 9 bits within the 10 bits (part of range in not used on screen in many files)

5) Can have many other formats and linear gamma 1.0

Anyone who wants endless headaches getting things to match can feel free to use DPX.

I am working on a way to "fix" DPX to get a better match between DPX processed in many programs that need to come together, but even when I get that done, I would still recommend video gamma 2.2 TIF (48bpp) files (not REC 609/709 but film curve for display on gamma 2.2 monitor) over DPX because you can be much more sure of how the TIF will look because it can be viewed without a LUT on "any" good but "un-calibrated" monitor.

To view a DPX file you more or less need a LUT or at least gamma correction because DPX gamma do not match any kind of monitor's native gamma!

 

[Nick Shaw]

To view a DPX file you more or less need a LUT or at least gamma correction because DPX gamma do not match any kind of monitor's native gamma!

DPX does not have a gamma. It is just a container for image data, and depending on the format of that image data you may or may not need a LUT.

I would say there are three common image data formats used in DPX files:

1. Log data from film scans (or from Digital Cinema cameras formatted so it can be treated as if it was a film scan – REDfilm falls into this category)

2. 'Video range' data with black at 64 and white at 940, and Rec709 gamma between these.

3. Full range video

I suppose REDlog is now a fourth common variant.

Software designed to work with DPX should be able to handle all these formats, and interpret (possibly with user ipnput) and display the data correctly.

The 'problem' as I see it with DPX is not the image data, but rather the lack of consistency in the meta-data in terms of flagging what format the image data is in.

Using DPX with data in a known format, within a known workflow, is not a problem at all. It is what high-end post facilities have been doing for years.

 

[Dan Hudgins]

IIF?

IIF?

DPX does not have a gamma. It is just a container for image data, and depending on the format of that image data you may or may not need a LUT.

The data has a gamma that the monitor is to be set to to view the image data, so each DPX file is MADE for and end use gamma to be viewed at, or in the case of the more complex soft sholder encoded DPX, the right formula needs to be used to view the DPX file.

When you mix DPX files from several sources, the film recorder may not have LUT for each source type or the header data may not have enough information about how each DPX was graded to pass the data through a LUT to show it as it was intended.

I would say there are three common image data formats used in DPX files:

1. Log data from film scans (or from Digital Cinema cameras formatted so it can be treated as if it was a film scan – REDfilm falls into this category)

As far as I can make out film scans are stored in so called log 10 bit DPX at gamma 1.7, can you quote the DPX spec otherwise?

When the log curve is clipped with a black and white clip point, the part of the curve between those clip points HAS a gamma value for the mid-tone.

Any other source image encoded into a 10bit Log DPX must have that curve or it will not match the film scan standard. In other words mid tone must fall at the right value for an 18% gray card between the black and white clip points.

2. 'Video range' data with black at 64 and white at 940, and Rec709 gamma between these.

Using Rec709 intercut with film scans would cause problems since the gamma and clip points are not the same, Rec709 is not film log or linear, one might think that the header has enough information to tell the file reader to adjust for that, but it is possable that the reader does not, or the DPX was passed through CC as a 10 bit log (gamma 1.7) file and now has data that will not match other shots processed from film log DPX files.

The way the highlights are processed is not the same.

3. Full range video

Again the header may or may not cause the reader to interpret the image data tones as intended, any variation from the "10bit log" gamma 1.7 curve would increase the mismatch probability.

The use of soft sholder increases the variance from the Rec709 curve.

I suppose REDlog is now a fourth common variant.

How does REDlog relate to gamma 1.7 and what are the clip points? What numeric value is mid tone in REDlog?

Software designed to work with DPX should be able to handle all these formats, and interpret (possibly with user ipnput) and display the data correctly.

I don't think the DPX spec defines "all the formats" well enough to insure that there is a simple and obvious reader curve that can be selected for each type, the number of combinations is very high and not strictly defined so as to eliminate all ambiguity. If you know of a more detailed spec other than the original DPX spec that covers each possible option in detail please include a link.

The 'problem' as I see it with DPX is not the image data, but rather the lack of consistency in the meta-data in terms of flagging what format the image data is in.

I think the header data issues are do in part to the vagueness of the spec documents, that may seem useful to the people who wrote them, but do not contain enough data for programers to ALL draw the same conclusion about what they mean.

Using DPX with data in a known format, within a known workflow, is not a problem at all. It is what high-end post facilities have been doing for years.

Yes, the best way to avoid odd DPX files is not to use additional programs to make or read them, but to restrict workflow options to those I/O paths for the DPX that have been tested and had work arounds developed to deal with any image shift issues. But that does not change the issue that DPX can vary, and have complex headers that may not be right, and that gamma 2.2 viewable TIF do not have those issues.

ILM developed OpenEXR and IIF is being developed because DPX has issues and something better might help make for fewer issues as "standard" workflows become a thing of the past.

 

[Nick Shaw]

I never said all DPXs were the same, or that DPXs from different sources with different image formats could be freely mixed and treated in exactly the same way.

You say that "gamma 2.2 TIFs do not have these issues", but you are then specifying that your image data has 2.2 gamma (is intended to be viewed on a monitor with a gamma of 2.2 to be more technically accurate.) If you specify the gamma, you could equally put that same image data in a DPX container as a TIF, and DPX has the great advantage that it can store timecode meta-data in the header, which TIF cannot. Like TIF, DPX also supports 16 bit depth, although admittedly not all software supports it. For a workflow based around REDcineX, Nuke and Baselight, I have found 16 bit DPX very useful.

You speak of "work arounds to deal with image shift issues". I would say that an established workflow where you know what image data you are putting into the DPX, and it is formatted in the way that the software you are reading it with expects it to be is not a workaround.

I do agree that OpenEXR based formats are vastly preferable. 10 bit log DPX dates back to a time when computers were unable to deal easily with floating point image data.

Anyway, I apologise for somewhat derailing this thread with a technical discussion of DPX.

The real answer to the original poster's question is that nobody can tell you in abstract what the "best workflow" for a job is. The right workflow for a project needs to be developed in collaboration with the facility doing the grade. Company 3 use DaVinci Resolve, which can read R3D natively. So talk to them about the project.

One thing I would say is that I would not personally recommend using Color as part of a workflow to produce DPXs to go into a grade on another system.

 

[Lucas Wilson]

Hey Daniel,

I've got opinions on DPX files too... : ) ... but more directly to your questions...

I'd recommend one of the following options:

OPTION A

1) Work hard on generating a good, clean EDL for a finishing system
2) Give Co3 an EDL and all the R3D source files that correspond to that list.

Let them do the conform, and then grade to R3D source. By exporting to *any* other format, you drastically increase the chance for error by inadvertently clipping or compressing image ranges. It is more than possible to correctly export DPX from R3D, but it requires a lot of care.

OPTION B

If you absolutely need to give them DPX files...

1) Work hard on cleaning up your timeline and making sure your layers are condensed, and no extraneous material is on your final sequence. Export an EDL of the finished sequence.

2) Anywhere you have opticals, put those A/B sides (for dissolves) or the entire clip (for motion effects) in a separate sequence

3) Put black slugs into the places that had the opticals.

4) Expand all of your clips by 5 frames on the head and tail side.

5) On the opticals sequence, expand all opticals by enough to completely recreate those opticals in a finishing system, i.e.: If you have a 24-fr centered dissolve, expand the A-side by 12 frames on the tail end and the B-side by 12 frames on the head end. That will allow the finish system to recreate the dissolve themselves.

6) Export both of those sequences as a sequential DPX sequence, being careful to maintain correct header metadata for the assemble

7) Give the finishing facility the DPX files and the list.

OPTION C

Create a clean list and export a sequential DPX sequence with no handles.

----

Ok, so....

<mount soapbox/>

THIS is why Avid is still by far the dominant tool of choice in the high-end feature editorial space. In Avid-land, option B above can be shortened to, "Consolidate, Export Pull List and Optical Pull List with appropriate handles."

This exact workflow has been done in feature editorial with 35mm film as a source for many years. There is no wheel to reinvent. It's a tried and proven workflow.

But, in FCP land... doing stuff like this is just... hard.

They are both amazing tools, and both have their justifiably fanatical user bases. But the Avid toolset was built from the ground up with a complete workflow in mind that involves several different tools and skillets (EDIT: SKILLSETS. Thank you spellcheck... argh.) The Apple toolset - like much of what Apple does - is monolithic and is very bad at working with anything outside of the Apple universe.

Not that Avid is without its flaws!!! But... this stuff just bugs me...

</dismount soapbox>

And I am ... of course... writing this from an iPad.

(sigh)

Lucas

 

[Dan Hudgins]

timecode meta-data in the header

timecode meta-data in the header

DPX has the great advantage that it can store timecode meta-data in the header, which TIF cannot.

I'm working on DPX support for my NLE and de-Bayer program, and the issue of what to do about the time code for composite shots, like a fade or desolve where you have two or more source shots being made into a single result shot (optical).

Do you take the DPX header time code from just one of the source shots?

Do you take DPX header time code from one until the fade is 50/50 then switch to the second shot's DPX header time code?

Do you make new time code unrelated to the original time code(s) of the shots being composited and save that in the result DPX file?

 

[MichaelP]

The "optical" is usuallly treated as its own element with naming scheme and timecode.

Michael

 

[Mark Wilkinson]

I have frequented this site for a while but have only just begun to have the Red projects that I am shooting go for high end grading. Please forgive me if this has been answered somewhere else on this forum. I recently shot a couple of 30 second spots for a production company on the Red. The editor cut the spots with the Red M-proxies.
The producers are keen to do a high end grade probably at Company 3 in NY. I am pretty sure Company 3's preference for material is in the DPX file format. I have never organized a color correct from Red footage ( especially proxies) , through Final Cut and then Color for a DPX output before to be handed off for a color grade. Has anyone overseen a similar workflow that could enlighten me on the pitfalls that one can face.
I'm assuming that once the timeline is locked I can send to Color direct from Final Cut Studio 3, from Color I can render and output the necessary DPX files and use the gather media option to contain these in one single folder? Moreover....
Is it customary for the house to perform a marriage with the Wav sound to provide us with a mov file for final output flexibility? Or more fundamentally; what format should we come away from the grade with to provide the production company with the most flexibility for broadcast and web distribution;bearing in mind they operate with Final Cut Studio and Color primarily and don't really have experience generating or working with DPX files or third party software like Gluetools.
I apologize for such a basic question but I am having trouble pinning down all the elements required and the correspondence from the post house has been less than clear.
Many thanks in advance

Hi Daniel,

I've done this a bunch of times. Lots of post houses prefer DPX (merits and demerits aside) because it has been the defacto standard for years. Additionally, Clipster is used by many of these post houses to feed a DaVinci your footage - and Clipster (in the bays I've worked in) feeds the DaVinci 10 bit DPX, sp why bother giving it something different to transcode. Clipster CAN do a simple assemble of R3D ftg, but with post being what it is today, every shot gets painted, pushed, pulled, spun, whatever so this has never been an option for us.

The most important thing to remember when doing your prelight in Color is to protect you black and whites - and get your midrange close to where you want to be in the final without pushing too hard. Ideally, you want to give your colorist evenly prepped material - color temp balanced shot to shot, black levels off zero, white levels down - but all roughly the same shot to shot. Your colorist will then have the best data to create the look in the bay.

Though as a side note, I deliver 16bit tifs to my post house and let the Clipster change to DPX because I do all my online in AE 16bit linear, and it just gives me hives to clip it myself. But with that much data properly handled, and after the VFX have been completed, it hardly matters.

But here's a question for the boards: Apple Color let's you work with non-safe white levels up to 110% - can these levels be handled in DPX or will they be clipped on the way out? I.E. is that a good way to get some more highlight data to the DaVinci or will it clip somewhere? Dan H?

Also, regarding audio - very often they not be able to combine the audio back with picture, so I take the DPX files on a drive, then replace the audio myself and deliver and archive a prores4444 file.

If you have any more questions, feel free to PM.

 

[Greg M]

Hey Daniel,

I've got opinions on DPX files too... : ) ... but more directly to your questions...

I'd recommend one of the following options:

OPTION A

cut
OPTION B

cut

Lucas

This is excellent advise, this post should be a sticky by itself.

The biggest problem we face when conforming is when client do not heed the advise above.

 

[Dan Hudgins]

110%

110%

But here's a question for the boards: Apple Color let's you work with non-safe white levels up to 110% - can these levels be handled in DPX or will they be clipped on the way out? I.E. is that a good way to get some more highlight data to the DaVinci or will it clip somewhere? Dan H?

Mark has some good points in his post.

Without looking at the source code for Apple Color it would be hard to tell exactly what is being done sinde the order of operations for clip and gamma adjustment or display LUT change the end result.

If you can take the Kodak Digital LAD (comes in a free CIN file) or some other un-clipped DPX through and bring back detail outisde of the 95-685 range then it would have to store the full 0-1023 range, maybe.

Clipping is something that was done for video signals to keep them within the range of FM video recording and for transmitter limits.

For film work in my NLE I use 48bpp TIF files, and recommend NEVER doing any "clipping" of the blacks or whites, rather you use an S-curve like film printing would, and with the extra data 16bits per color TIFs can hold, you can pull back the detail you need later without Posterization.

If you use DPX within the normal 95-685, or whatever range you define in the header, there is not enough data bits there to do much in the way of S-curve or contrast change up and down without scaling errors and seeing banding and Posterization since you are for the most part starting with about 9 bits and you need to end up with 9 bits so grading requires starting with more like 12 to 14 bits with a camera that has wide dynamic range and will get a hyper-gamma curve or heavy S-curve.

In my way of doing things (when not using DPX) you NEVER need 110% (over 685) or -10% (under 95) or any silly values like that, 100% is near clear film base and 0% is near D-max, and all values fall between those on an S-curve with enough extra bits to grade to any useful ISO. Older thick emulsion negative/positive film stocks did not clip (when lab work was done right), and Digital Cinema does not need to clip either...

In otherwords common 10bit log DPX have a range of 95-685 or 0-590 after clipping, a 48bit TIF file has the range of 0-65535 or

65535/590 = 111x as much data per primary, or 111^3 colors

With the extra data you can compress the tones near the highlight max and shadow minimum without Posterization in the end result if you pull some of that crushed data back, since it has not "gone over the edge. You don't need 110% since you can hold enough tones to make a good image between 100% and 97% or between 0% and 3% (0.03*65535=1966, 0.02*65535=1310, 0.01*65535=655).

Working uncompressed you can dither tones to make small tones near the top and bottom hold up better since even if you are between tones 255 and 254 and 0 and 1 on an 8 bit display at higher resolutions you can still see many tones between those one bit steps on 8 bit display by pixel group average in your eye.

The common 10bit log DPX were developed from CIN film scans, film scans had grain to dither the values, with newer Digital images that are "de-noised" (and so lack dithering "noise" around the tone steps) the 9 or so useful bits are not enough to avoid Posterization (histogram gaps in the display data) and tone banding after heavy grading.

Converting graded TIF to DPX for sending to the lab for printing maybe be a good way to go as storage is much lower cost than it was in the past.

==

Thanks for the info on new time code for each new optical.

==

It is possable to request a new TIF tag, if you did that you could include SMPTE time code and whatever other meta-data modern editing requires.

You can also put such data between the header and image data in the "dead" space as a note-pad while editing.

==

 

[Peter Dimov]

In otherwords common 10bit log DPX have a range of 95-685 or 0-590 after clipping, a 48bit TIF file has the range of 0-65535 or

65535/590 = 111x as much data per primary, or 111^3 colors

Not really the correct way of looking at it.

The Cineon log encoding does put the black point - 0.0 - at 95, and the white point (1.0) at 685, but this doesn't mean that it doesn't use the values that fall outside of this range. 0 in this encoding represents -0.005651, and 1023 represents 13.521695. The reason it puts the black point at 95 is because most values below 95 are scanner noise, so it "pre-crushes" blacks by ~0.006. You'll still have this noise in your 16 bit TIFF, so it's not fair to claim that these 95 values are useless.

In addition, if you put the white point at 65535, you won't be able to represent values above 1.0. To match the whole Cineon range of a film scan, you have to map 65535 to 13.5, which puts your 1.0 whitepoint somewhere at 4850.

Firther, the Cineon encoding is logarithmic, which means that it allocates more precision to the smaller values. 96, for example, is 0.000084, which is approximately 1/12000 - a smaller step than a 16 bit linear TIFF with a whitepoint at 4850 can represent.

All in all, for a mere 10 bits, the Cineon encoding is really not that bad.

 

[Dan Hudgins]

Midtone value used for 10 bit log DPX files

Midtone value used for 10 bit log DPX files

The reason it puts the black point at 95 is because most values below 95 are scanner noise, so it "pre-crushes" blacks by ~0.006. You'll still have this noise in your 16 bit TIFF, so it's not fair to claim that these 95 values are useless.

The values under 95 are not noise, they are thin parts of the negative if the scanner is working the way it should and the exposure was adjusted, there should be no sensor noise in the signal above the 10bit of the log adjusted data (exposure fusion and dark field subtraction can reduce sensor noise).

Putting those into a gamma 2.2 (displayable) TIF using a adjusted S-curve would make values under about 105-120 in the Cineon type scan DPX file fall in the bottom 3% or so of the gamma 2.2 TIF range so from about 0 to 1966 and nothing would be "lost".

In addition, if you put the white point at 65535, you won't be able to represent values above 1.0. To match the whole Cineon range of a film scan, you have to map 65535 to 13.5, which puts your 1.0 whitepoint somewhere at 4850.

Again values above about 670-680 in the DPX would be in the top 3% or so of the TIF file at gamma 2.2 (with sholder of S-curve used) about values 63569 to 65535, not 4850.

There cannot be brightness values over D-max on the negative scanned, and so when the DPX are made D-Max should be under value 1023.

Since most display devices do not do well in the top 1% of their brightness range you want the "white" for useful subjects under a few precent of the display maximum brightness. Using a curve holds all the useful data in the top few percent of the gamma 2.2 (displayable) TIF, with enough bit depth that you can pull some of the highlights down to make them more visable if you need to later.

In the gamma 2.2 TIF file 18% gray is at about 0.458 signal or,

0.458 * 65535 = 30015

So "white" subject would be between 30015 and 63569, as values over about 60000 should be for highlights and specular reflections and not just subjects that are white in color.

Firther, the Cineon encoding is logarithmic, which means that it allocates more precision to the smaller values. 96, for example, is 0.000084, which is approximately 1/12000 - a smaller step than a 16 bit linear TIFF with a whitepoint at 4850 can represent.

The size of the step in the TIF I was talking about in on a curve and does not relate to linear light steps.

The TIF I was talking about are not linear light values, but video gamma 2.2 (plus curves) which may be bringing up the shadow areas more than the gamma 1.7 of the so called log DPX.

All in all, for a mere 10 bits, the Cineon encoding is really not that bad.

Because BMP 24bpp files are encoded for gamma 2.2 rather than the gamma 1.7 of the log 10bit DPX the shadow areas above black clip are pulled up more and when you display a 10bit log DPX you need to adjust the display gamma which causes scaling errors and makes it not much better than a BMP file.

To make a comparison, the step size and bits resolution in each part of the tonal scale needs to be taken into account. The point of what I was talking about was that the extra bit depth of the TIF can be used to archive tonal details that are too small to see on a monitor, but can be pulled back out of the highlights and shadows if needed, whereas the limited bit depth of the log 10 bit DPX restrict grading range and will show histogram gaps due to scaling errors under heavy grading and contrast increase. In the top and bottom few percent of the TIF you have more range than ALL of the DPX data, and it can have "log" like curve on it. And nothing needs to be clipped off or lost.

To figure the gamma of the so called log 10bit DPX you take the black clip point as 0 and the white clip point as 590 the find the mid tone value, and figure the gamma value from that, which should come out at about 1.7.

What mid tone value are people here using for 18% gray card readings on a 10bit log DPX?

(if you have a value please also state if it is counted from DPX 0 to 1023 scale or from 95 to 685 scale, that is, is it absolute from 0 or has 95 been subtracted.)

 

[Nigel Stanford]

Hey Daniel,


OPTION B

If you absolutely need to give them DPX files...

1) Work hard on cleaning up your timeline and making sure your layers are condensed, and no extraneous material is on your final sequence. Export an EDL of the finished sequence.

2) Anywhere you have opticals, put those A/B sides (for dissolves) or the entire clip (for motion effects) in a separate sequence

3) Put black slugs into the places that had the opticals.

4) Expand all of your clips by 5 frames on the head and tail side.

5) On the opticals sequence, expand all opticals by enough to completely recreate those opticals in a finishing system, i.e.: If you have a 24-fr centered dissolve, expand the A-side by 12 frames on the tail end and the B-side by 12 frames on the head end. That will allow the finish system to recreate the dissolve themselves.

6) Export both of those sequences as a sequential DPX sequence, being careful to maintain correct header metadata for the assemble

7) Give the finishing facility the DPX files and the list.

Lucas

or you could just load an edl into monkey extract, and export all the dpx...

 

[Mark Wilkinson]

Since most of us here aren't working with film scans, but red footage - does it make more sense (when we need to output DPX files for post house color correction) to use full range settings and linear gamma?

 

[Dan Hudgins]

FFFF hex rather than 8000 hex?

FFFF hex rather than 8000 hex?

Where do they get that range 0-32768, (0 to 8000 hex)

16bits is 0 to 65535 for using all the bits (0 to FFFF hex)?

15bits is 0 to 32767, or 0 hex to 7FFF hex.

==

You would need to test the results from using 0 to 1023 to see what the other software does with the clipping or roll off.

FFFF hex not 8000?

Although you can say 8000 is a 16bit value, its really 15bit for the most part since you are only using one of the values as 16bit and all the others as 15 bit.

==

9 or 10 bits, you still want log or gamma 1.7 or 2.2 (shadow detail pull up) in the data, since linear light of the camera's 12bits would not look as good if you drop the bottom two bits. Its not clear what "linear gamma" means because when you de-Bayer at a give ISO a curve is put on the end result, there is no such thing as "linear" once the ISO curve is added, the only true "linear" is gamma 1.0 linear light 12bit sensor data, or 14bit for the M-X sensors. And linear light values RAW do not have an ISO adjustment, the placement of mid-tone 18% gray card in the shot depends on exposure.

==

Its odd that they are dropping almost half the data it seems in their internal processing by clipping at 32768 rather than 65535 after interpolation since the extra data does hold some of the interpolated values...

==

See this link:

http://forums.creativecow.net/thread/223/11644

Quote: [The source files are linear rec709 dpx frames 0-1023, no lut.
I kept the session the same linear 0-1023. Did not override header settings.
I rendered a mov pregrade so I could work in fcp.It did not give me a gamma choice when rendering out the quick time. I then put that material back into color and everything looks fine.(as the dpx time line did) Then i trimmed with the client and rendered to dpx 0-1023 and everything now looks completely different.]

 

[Mark Wilkinson]

Hm, I wonder how they arrived at those settings...

Couple more thoughts and questions:

I just got back DPX files from Technicolor (DaVinci via Clipster) and they gave me full range 0-1023.

I originally delivered 16bit lin tifs for the Clipster - and everything seems good.

Regarding the linear gamma and deBayer pass vs. ISO curve issue - that is a question for Graeme N. it seems to me. My blind guess is that REDCINE-X is adding an ISO curve to the initial "ripping" of the R3D file and putting that in a 16bit tif bucket.

When AE imports the REDCINE-X (ISO gamma'd) tifs into a 16bit linear project, it will toss that ISO gamma out and start from the agnostic (no-gamma applied) image data. Then (assuming you have sRGB IEC61966-2.1 applied as your working space) it will use that as a LUT for your preview.

Does that sound right or am I missing something?

I'm confirming this because it seems to me important to the question of workflow for a post house.

 

[Dan Hudgins]

Linear light vs. ISO adjusted TIF or DPX etc.

Linear light vs. ISO adjusted TIF or DPX etc.

Hm, I wonder how they arrived at those settings...

Yes, it would see it should be 0 to 7FFF hex and not 8000 hex, or 0 to FFFF hex...

Couple more thoughts and questions:

I just got back DPX files from Technicolor (DaVinci via Clipster) and they gave me full range 0-1023.

I originally delivered 16bit lin tifs for the Clipster - and everything seems good.

Regarding the linear gamma and deBayer pass vs. ISO curve issue - that is a question for Graeme N. it seems to me. My blind guess is that REDCINE-X is adding an ISO curve to the initial "ripping" of the R3D file and putting that in a 16bit tif bucket.

When AE imports the REDCINE-X (ISO gamma'd) tifs into a 16bit linear project, it will toss that ISO gamma out and start from the agnostic (no-gamma applied) image data. Then (assuming you have sRGB IEC61966-2.1 applied as your working space) it will use that as a LUT for your preview.

Does that sound right or am I missing something?

16bit TIF files are only "linear" if you save the sensor data or at least the de-Bayer image at gamma 1.0. Many TIF are monitor viewable and so are at gamma 2.2 or 2.4 or if an old mac was used gamma 1.8.

Once you apply ISO curve, the data in the file is no longer "linear" in the sence that it relates to the sensor being linear, the curve used to make the sensor data non-linear and to pull up the mid-tone 18% gray card to the target mid-tone value is mostly unknown and is part of the camera circuits for the Sony hyper-gamma or some curve in the RED de-Bayer program.

Since the ISO meta-data may be lost when the TIF is made by some program the post house as no way of knowing the ISO used or the curve used to convert the linear light sensor data to the "gamma/log/curve/iso adjusted" image they see when they display the TIF on their monitor (at some gamma between 1.8 and 2.4 maybe).

You cannot save 12bit linear light data in a 10bit DPX file, that is why kodak came up with their so called LOG file encoding, what that is is that they encode the brightness values as density steps on a linear scale between 0 and 1023 can call that encoding Log, but it has a bottom and top, and those are not balck or white but near black and near white since they relate to film density which as a S-curve.

I do not think the Cineon scanner linearized the top and bottom of the film curve since it would then be too long to fit into their density range, in otherwords they recorded a log value of a non-linear film density to light ratio, so the relationship was from film density to file value and not from light in front of the camera to file value.

With RED ONE the linear light save should save in the file a direct transfer from the light in front of the camera to the values in the file, I think they have said that you can get that if you save to TIF. DPX cannot hold all 12bits in linear light mode as linear steps of equal size. In DPX log the steps are of un-equal size relative to the 12 sensor bits.

I do not see how any program can remove the ISO curve from a saved TIF or DPX file, those curves are un-known and non-linear.

You can remove the file gamma adjustments and make the ISO curved data back to gamma 1.0 if you know the gamma used and the black and white points. As for the soft sholder, that's another issue as to how it was done and how it can be un-done and if there is then enough head-room to store the high end once you "linearize" the highlights again.

I'm confirming this because it seems to me important to the question of workflow for a post house.

The confusion comes from calling two different things "linear". A ISO adjusted image is not a "linear light" image and probably cannot be turned into one without errors.

Many "linear" images, TIF or BMP, are not at gamma 1.0, some may be at other gammas like 2.2 and therefor are not "linear" but more like the DPX Log which are more like a gamma 1.7 file.

People seem to use Log to describe the gamma 1.7 system Kodak used in the Cineon files, and linear for most other things, sometimes gamma 2.2 files. A 8 bit file using CCIR standard 601-2 says the data shoud range between black at 16 and white at 235, do we call that 8 bit linear, even though the data between 16 and 235 may be near gamma 2.2 which is a curve much like the curve of log DPX in that it compresses the highlights and expands the shadow areas?

I noticed that two Kodak pdf give different values for 18% gray mid-tone, one says it should be 445 and the other 470? So what gives with that? You cannot have two values for mid-tone in DPX files both 10 bit log? That is why I was asking what mid-tone value is being used by people here?

When you output a frame using the RED programs the value of 18% gray card in a 10bit log DPX file should be red=445 green=445 blue=445 as far as I can make out, even if the file is set to 0-1023 or 95-685?

For a TIF gamma 2.2 the 18% gray mid-tone at ANY ISO SETTING should be about 0.458 of signal or 30015 in 0-65535 scale "linear" values, (the values in a DPX log file are always linear even if it is a log file, just as the values in a TIF are a linear scale even if it is a gamma adjusted image).

I have been converting R3D to RedLog TIF since that seems the best to insure that the shadow detail is expanded before color correction, and the TIF have enough counts to insure than all of the 12 camera sensor bits are not scaling errored out of the final image.

When you convert the 12bit sensor data to 10bit log DPX you loose many of the sensor counts before you start color correction through the down scale. After heavy grading you get even more histogram gaps and bumps in the sensor data tone steps from scaling errors, which add up with monitor LUT and such later.

The ISO adjustments themselves cause scaling errors when you save to lower bit depth file types since you are changing the size of each sensor step, making the highlight steps smaller and the shadow steps larger. The higher the ISO speed the more scaling errors that get into the images, each stop of ISO gain causes about one step size loss because to make the mid tones twice the contrast the mid-tone steps are made twice as large (by way of the gamma or hyper-gamma curve used to convert linear light to ISO adjusted data).

==

As for working with a post house, you can use a ref image like the old film LAD image I have on my web site which has a near white, mid-tone, and near black patch, you measure those values in a graphics program that has a "probe" (my CC has a probe to measure "digital" density values as a % of full scale) and write those numbers down. You then make sure your "Kodak Gray Card Plus" shot for each setup will match using the ISO in the de-Bayer program.

For example if you save the DPX log 10 bit and load them into a graphics program with a probe you shold measure the 3% black at about 95 (or what?), the 18% gray at about 445 (or 470 as given in another pdf?), and the 90% white at about 685 (or what?), if not the ISO used in the de-Bayer program was probably off the real exposure used in the camera?

Here is a link to the gray card plus,

http://www.kodak.com/country/US/en/motion/products/tools/card.shtml

When you get the graded images back, they should still match or at least be close to those three points of tone to be usable.

One issue with the RED ONE shots is that if you output linear light or REDLog the black level may be quite a bit above 3% black level, so mid-tone and 90% white may also be off, thats why you need to shoot the Kodak Gray card plus or a print out of the old LAD so you have something to go by that is consistent, consistent is more important than being "absolute" right since you can make over all adjustments later, but you do not want shot-to-shot inconsistencies.

Quote: [The characteristic curve for the Cineon Digital Negative is shown in the following figure. For a
normally exposed negative film, the 90% white card has a code value of 685, the 18% gray card
has a code value of 470, and the 2% black card has a code value of 180. Dmin (~1% black) has a
code value of 95. With the scanner balanced on film dmin, the three color records of the film can
be represented by a single printing density curve.]

Quote: [Finally note that some facilities may choose to use a digital LAD with code values other than
445, 445, 445. We chose these values to be consistent with a large body of industry practice.
However, other values could be chosen, such as a set of code values that give densities closer
to the publication H-61 densities.]

Note the phrase "some facilities may choose", WTF? That's why you need a TESTED workflow, there is NO standard!

I'm adding a "second" level of color correction to my de-Bayer/batch CC program to help convert what looks good on the monitor into "density" numbers like the 90%=685 18%=470 2%=180 1%=95 given above, or whatever other numbers the lab uses so that you and make a global "match" conversion to DPX output so that you can convert "your" way of doing things to someone elses "way" to "try" to resolve these matching issues, maybe...

 

[Mark Wilkinson]

Thank you, Dan. I appreciate your generosity with information.

I'm going to shoot a DSC chart and do some tif and DPX testing via REDCINE-X and AE linear settings. I'll post those for discussion.

And if I'm going to let the Clipster transcode my tifs, I need to find out what it's doing to the data.