Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

RAW or DPX

P

paulson lang

Member
Joined
Dec 31, 2010
Messages
6
Reaction score
0
Points
0
HI all,
New to this forum. please suggest a workflow for lustre. Is it better to work on RAW or transcoded DPX footage?

paulson
 
output will be in 2k DPX format for recording. So is working in RAW really going make a difference?
 
well, if you save something like a log dpx it's possible you can get what you need. but working with the native raw files from the camera provides the greatest possible latitude.
 
RAW is far from allways the first choice vs DPX...

And "working with the RAW file" results depends heavily on available metadatasettings and other variables.

Possibly well developed EXR's would be first choice if you can transcode them with a Rocket and RCX and lustre supports them.

RedlogFilm DPXs possibly the second best choice, all depending on a lot of variables.

Rather than being "one best option" there are a multitude of "best" options depending on what your deliverable is and how the files are exposed, whether they involve a lot of VFX work, light conditions and sensor.

It's not hard. Quick A/B tests will probably give you your answers.

But RAW is far from neccesarily the best post-option allways...

Cheers

And happy new year.
 
very true, gunleik.

however, with full access to the RAW metadata, it does provide the most image fidelity.

though i agree, that does not necessarily make it the best suitor for every workflow.


also, what about rc2/rg2 vs. redlog?
 
Hmmm.... Raw has some advantages over the DPX.

For one you can change your white balance rather easily.

Also, is nice to alter de raw debayering to suit your needs on a per shot basis and lets not forget that its the best way (at least for us) to deal with tricky shots.

There is also a TON of use for the metadata.
 
Points are many...

here are some:

1. Many softwares don't have access to all the metadata in their development stage.

Curves are quite essential.

2. Many softwares are constructed to work with RGB LOG files and do their things best that way.

3. With rocket support, the rocket actually feeds an RGB image down the pipeline, but if you don't have full metadata-access - that still isn't the best solution.
 
It is all actually depends on what's under the hood of the Lustre you will use. The problem is, that Lustre doesn't support R3D natively, like Baselight or FilmMaster or Scratch etc. RAW files need to be debayered first through Wiretap. Lustre itself then uses debayred material, like DPX for grading. So, as I said, it depend on what's under the hood of Lustre. If you have a powerful wiretap station with RR, then the process is pretty transparent and nearly instantaneous, but if you don't, then DPX will have to do...
 
Same position with Gunleik here.

I work with a Quantel Pablo (with the iQ V.5, I have the ability to use the R3D Raw files soft mounted via Red Rocket), but still prefer a well distilled RGB Log image to work with.

Imho, the raw side of the grading are really useful and helpful in order to setup your first light. After that (secondary, creative grading, vignette and what else) I'm feeling more comfortable in the "traditional" way.

And this is only my opinion, with my equipment, and my skill. Never say never...
 
jake blackstone said:
It is all actually depends on what's under the hood of the Lustre you will use. The problem is, that Lustre doesn't support R3D natively, like Baselight or FilmMaster or Scratch etc. RAW files need to be debayered first through Wiretap. Lustre itself then uses debayred material, like DPX for grading. So, as I said, it depend on what's under the hood of Lustre. If you have a powerful wiretap station with RR, then the process is pretty transparent and nearly instantaneous, but if you don't, then DPX will have to do...
Click to expand...

Jake, didn't Mike Most specifically say that it's no longer like that on the 2011 series of systems from Discreet? I could check on our smokes, but I am done working this year :-)
 
Frank Cueto said:
Jake, didn't Mike Most specifically say that it's no longer like that on the 2011 series of systems from Discreet? I could check on our smokes, but I am done working this year :-)
Click to expand...

Lustre 2011 has the same soft import abilities as Smoke 2011. Jake is not completely wrong in his statement that it uses a Wiretap Gateway to do the debayering, the difference between 2011 and past versions is that the gateway runs on the local host, and doesn't require an external computer. Internally, the gateway does the debayer and sends a live stream into the application. But it's all transparent to the user, and it does allow access to the metadata at the gateway level. Still not nearly as integrated or as flexible as Baselight, Scratch, or Film Master, but much better than it used to be, and doesn't require a separate transcode step.
 
M Most said:
Lustre 2011 has the same soft import abilities as Smoke 2011. Jake is not completely wrong in his statement that it uses a Wiretap Gateway to do the debayering, the difference between 2011 and past versions is that the gateway runs on the local host, and doesn't require an external computer. Internally, the gateway does the debayer and sends a live stream into the application. But it's all transparent to the user, and it does allow access to the metadata at the gateway level. Still not nearly as integrated or as flexible as Baselight, Scratch, or Film Master, but much better than it used to be, and doesn't require a separate transcode step.
Click to expand...

It's good to hear, that Lustre finally integrated Wiretap on a local host. I haven't had a chance to play with 2011 version. Nevertheless, I still find Wiretap unnecessary complicated and user unfriendly, unless they had managed to hide it under the hood, but it is a step in a right direction.
 
jake blackstone said:
It's good to hear, that Lustre finally integrated Wiretap on a local host. I haven't had a chance to play with 2011 version. Nevertheless, I still find Wiretap unnecessary complicated and user unfriendly, unless they had managed to hide it under the hood, but it is a step in a right direction.
Click to expand...

I can speak from direct experience on Smoke 2011. It is basically hidden under the hood. You don't use the Web interface at all. You initiate a gateway right in the Library module, by accessing the network and selecting any available directory as a read-only directory. It then appears just like any other library on your local storage, with the only caveat being that it's read only. You have access to foreign file formats like R3D directly within that interface, so even though it's doing a live translation, to the user it just seems like you're accessing a library clip, except with the ability to access and change certain metadata as part of the bargain. You can drag and drop into local libraries, load directly into a source area from the gateway, and do all of that either by making local copies or not. You can also choose whether or not to make proxies. On Smoke, conforming is accomplished the same way, by dragging and dropping either an EDL, an AAF, or an XML file from the gateway (or wherever it lives) into a local library and having it conform from wherever it comes from. It's all quite an improvement over the rather clunky system it used to be.
 
DPX 16bpc vs. 10bpc

DPX 16bpc vs. 10bpc

Although common DPX are said to be 10 bpc (log is about gamma 1.7), the Cineon type clip points of 95 and 685 reduce the range down from 0 to 1023, so you get about one bit more over all than a 8bpc BMP file at gamma 2.2 after the DPX is run though a monitor LUT to convert gamma 1.7 to gamma 2.2.

With RAW data going to a 16bpc TIF, even though the sensor may be 12bit, you get less Posterization near visable range after heavy grading than you would adjusting a 10bpx DPX file that is clipped to near 9 bits to start with.

That brings up the less supported 16bpc DPX files, they should be as useful as 16bpc TIF files, but also have date plus time, and time code and user bits fields in the DPX header, for editing. So if you have to go DPX for some reason, look into using 16bpc rather than 10bpc if you are going to do any major adjustments to the way the images look in grading, ad the 10bpc DPX if low noise could show banding after heavy grading (heavy grading was NOT in the original Kodak design of the Cineon system that morphed into 10bpc DPX, and film scans had grain which in high resolution images gives more bits of tonal range off the projected print through dithering and the mixing of pixels from the film, film recorder lens, and projector lens).

One advantage of RAW over DPX is that you can use DRX since the data has not been white balanced yet, the extra highlight detail in the red and blue sensors can be converted to monochrome and added to the Luma to get better highlight detail. DPX would be white balanced in normal use, and so the "extra" highlight detail used in DRX would have been clipped off before the DPX are saved, unless you have the DRX adjusted right before making the DPX, but even so, you cannot adjust the DRX once its burned into the DPX, whereas you can make as many changes as you want if you go back to the RAW data for the final render.

=

I was wondering if anyone has some 16bpc DPX files they could email me, HD, or 2K size would be fine, I just want to check the data alingment and endian issues, I have support for 10bpc DPX in the new de-Bayer/batch CC program I am working on, and would like to add 16bpc DPX support, but need some sample files from various programs in "real world" use to make sure I get all the details right for Interoperability between my "freeish" DI system and other software in industry use. You can email sample frames smaller than about 20MB to tempnulbox [at ] yahoo [dot] com if you zip or rar the file it should fit at 2K or less, thanks for any DPX files you can send that are 16bit (48bpp) uncompressed. Please put "16bpc DPX" in the email subject line, thank-you!
 
Dan Hudgins said:
Although common DPX are said to be 10 bpc (log is about gamma 1.7), the Cineon type clip points of 95 and 685 reduce the range down from 0 to 1023, so you get about one bit more over all than a 8bpc BMP file at gamma 2.2 after the DPX is run though a monitor LUT to convert gamma 1.7 to gamma 2.2.
Click to expand...

That is not quite accurate. 685 is not a clip point in a log DPX, it is the reference white level. There is normally a lot of highlight information significantly higher than 685, and this can be 'dug into' in grading.
 
Softclip or hard clip

Softclip or hard clip

Nick Shaw said:
That is not quite accurate. 685 is not a clip point in a log DPX, it is the reference white level. There is normally a lot of highlight information significantly higher than 685, and this can be 'dug into' in grading.
Click to expand...

It depends what program made the DPX file, and if it was from a film scanner or HD camera etc.

Kodak says in their Cineon documents that "Soft Clip" should ONLY be done once, and once you softclip the maximum lighlights to the 685 level there is no extra detail above that, or it is reduced what is above 685 is clipped off by the monitor LUT so you do not see it.

In a Digital Cinema camera you should apply the needed curves BEFORE saving the image, in that case sensor clip is set to 685 and there is no "extra data" since you want the "film look" which means that there is a "S" curve before the data is scaled to fit within the 95-685 range.

In normal display the monitor LUT clips the data to 95-685 range, in Kodaks Cineon vision, there was a vertical adjustment to slide the data up and down over tha window, with everything under and over NOT being in the display or on the film print, unless you apply softclip ONCE in the grading chain at the first stage after the scanner (such as when VFX are going to give you a second generation DPX to use in the film recorder.)

You could softclip from the original scanner files in the film recorder, but that would need to be setup for doing it shot-by-shot. In the Kodak Cineon documents they recomend not using softclip and only doing hard clip in most cases since the softclip makes the highlights non-linear.

If there is or not, any data of use above 95-685 range depends on the source, if its a digital camera or a graded image, there may not be, you can tell by setting the load range to 0-1023 and look at the histogram, a clipped DPX will show a band with blank areas at the top and bottom.

There are other ranges within the 0-1023 used for gamma 2.2 HD DPX files, that would have even less range above the upper clip point, so using softclip on those files would give less highlight detail increase.

If you have a specific 10bpc DPX file, I can look at the full range histogram to see if it has any useful data above 685. If it does, then the question is why was that extra highlight detail not moved lower by the in camera or in de-Bayer curves that are used for ISO adjustments? When film-to-film printing was used, there was no highlight clip as the high sliver content film stocks were all long range and had S-curves, so the were all doing "softclip" all the time. What happend in grading in the bell and howell additve light house, was that the highlights were compressed or expanded onto the master positive, but not clipped, which is what gave film-to-film printing better highlight detail than video to clipped range FM videotape recording. All of the older stocks that had a bit of silver in them had S-curves, it was not easy to totaly lose any of the highlight or shadow detail, as they did not clip if processed right (with the bromide set right), with Camera_Negative-Master_Positve-Dupe_Negative-Print_Positive printing each step added "softclip" S-curve to both the highlights and shadows. When Digital Cinema images are processed they should also be on S-curves that simulate the 4-generation film printing, before the data gets to a DPX file, or you will not see the full 11+ stop range the camera can capture.

The HD camera, or de-Bayer program, needs to compress the contrast in the highlights in order to change the ISO numbers up past the sensor having 90% white level for the green sensors near ADC max. Although the "video" way of having the output linear was a "standard" that is obsolete, modern digital cinema cameras like Cinealta use "hyper-gamma" or "S-log" curves to "softclip" before the DPX are made (or you could not adjust the ISO), so the range over 95-685 would not be like a film scan where the 95-685 window only shows part of the data range saved, which by the way would not extend to 1023 if the color negative was pull processed enough.

If you make a 10bpc LogC (~ gamma 1.7) non-white-balance DPX of a Digital Cinema Camera's 12bit+ sensor data, you cannot hold ALL of the data within the 95-685 range, unless you use softclip, and if you don't use softclip then you get reduced tonal bits range on the end use screen, and the results will have clipped highlights like a "cheap video camera" that does not use internal "softclip" curves like "hyper-gamma" etc. Why would you want to not just flatten down the highlight detail before saving the DPX file, what highlight detail do you want to "through away"?

At any rate, the end use device, like a film recorder or projector, only shows the tones between the hard-clip points (unless it is doing softclip and scales in 32bit data space), so even if highlight has been pulled down, the end results AS FAR AS TONE BANDING and Posterization goes, depends on the 95-685 sub range that is going through the display LUT to convert it to gamma 2.2 etc. for projection or display, and the going through the monitoring LUT causes scaling errors that reduce the accurate bit depth of the tones you see so that in the end you get about 9bits not 10bits on the screen, so what you see is just about one bit better than a 8bpc BMP file.

==

If anyone has uncompressed 16bpc DPX files they can share, as I mentioned above, it would be a big help to my fininshing my batch CC program, for processing EPIC HDR images, I should have 16bpc DPX support.

Also if anyone has uncompressed 10bpc or 16bpc DPX files that have 50, 59.94, or 60fps SMPTE time code + user bits fields in the DPX header, that would be a big help as well, since I a need to look into adding support for SMPTE time code for fps above 30fps, thanks in advance, you can send the sample files to tempnulbox [at] yahoo [dot] com if they are under 20MB zipped, or I can download from your ftp etc, thanks again...
 
Dan, you are missing an important point. This discussion started asking for a comparison of the merits of DPX vs RAW for grading from. While it makes sense to constrain your final deliverable to sit between the black and white points of your display medium, it makes no sense to apply the same restriction to an intermediate working format.

Log film scans, REDlogFilm, ARRI LogC from ALEXA and D21, PanaLog from Genesis, and S-Log from F35 all contain a lot of useful information significantly above the reference white level. This is part of the design of all of these formats, and not needing to constrain the values to "legal" at an early stage of the workflow is one of the big advantages all of them have over "video". IMHO the "legalising" is best done at the last possible stage.

The early PDlog685 curve from RED did not give any values above 685, which is why it has been superseded by the much superior REDlogFilm. Take any of the ISO800 R3D samples available on this forum, and render them to REDlogFilm DPX (10 or 16 bit) in REDcineX, and you will see what I mean.
 
derivatives

derivatives

Nick Shaw said:
Dan, you are missing an important point. ...

The early PDlog685 curve from RED did not give any values above 685, which is why it has been superseded by the much superior REDlogFilm. Take any of the ISO800 R3D samples available on this forum, and render them to REDlogFilm DPX (10 or 16 bit) in REDcineX, and you will see what I mean.
Click to expand...

I have been working with True RAW sensor data for some time, and almost always do not white clip the green sensors since that would lose highlight detail. The red and blue sensors need to be clipped to match the "natural" ADC max of the green sensors (under most light green clips first, otherwise you don't clip the color that does at ADC max) for the chroma balance. The left over highlight detail from the red and blue is used to add additional highlight detail to the Luma by various means, as the DRX feature in RED products does.

Even working with the sensor at minimum gain, and using the full 12bits from the ADC, I do not see any reason to clip off any of the highlight detail from what the end user sees on the movie screen, in none of the images I have worked with would I say that there would be an improvement by losing highlight detail.

If people are going to use DPX as an intermediate (derivative) step between two operations I would recommend using the 16bpc files, as each time you scale the 10bit data you get more "histogram gaps".

R3D files cannot be used for intermediate (derivative) steps, to simulate intermediate (derivative) steps you need to go back to the original R3D, find the right frames, de-Bayer them into a high bpc format, make additions and changes, then output some format that is RGB or YUV etc.

In the case of RED data, there is no actual "RAW" data outside the camera, you have the wavelet compressed REDCODE that is a type of RGB file with un-equal resolution between the RED/BLUE and GREEN parts, it seems.

So for working "RAW" you always have to go back to the R3D files, when working with DPX you are working with data that has already had scaling errors introduced through conversion to one or the other curve. If you base the end format on de-Bayered DPX rather then the R3D, you add a second stage of scaling errors when any grading adjustments are made to the already ISO adjusted data.

If the 10bpc DPX data were not ISO adjusted, at higher ISO there would be too few bits since most of the image would be in the lower 4 to 6 bits resolution (plus 95 count), and that would show too much banding.

Even with a 14 bit ADC or more, I can see nothing superior about clipping highlights, if it was superior then why would anyone use EPIC HDR mode, just to fill unused bits above some clip point and not use them. Film has a soft sholder in the negative, or did when Kodak was putting enough silver in the film, and that is what give film its smooth highlights that are not "video like".

Do you have examples of images that are superior because someone clipped off the highlight detail?

As for the choice between DPX and R3D, sometimes you need an intermediate format for steps that R3D would not be the best choice for the workflow, so in that case 16bpc DPX would hold the time code, or 16bpc TIF can be used where you don't need the time code or the software does not support 16bpc DPX.
 
You must log in or register to reply here.
Back
Top