Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

External Recorder - RAW out?

A

Adam Howden

Well-known member
Joined
Dec 20, 2007
Messages
298
Reaction score
0
Points
0
Location
Australia
Website
adamhowden.com
Hi All,

Am just wondering if it's possible to record a flat image ie the RAW as an image stream to something like the Sound Devices Pix240 or Atomos Samurai.

Essentially, on the HD-SDI out of the Epic (or Scarlett) can you set the menu to send a dry non LUT or non REC709 image?
 
Adam Howden said:
Hi All,

Am just wondering if it's possible to record a flat image ie the RAW as an image stream to something like the Sound Devices Pix240 or Atomos Samurai.

Essentially, on the HD-SDI out of the Epic (or Scarlett) can you set the menu to send a dry non LUT or non REC709 image?
Click to expand...

If you want "RAW" sensor data without white balance and linear rather than gamma corrected you would need to record it with 12 to 14 bits minimum since if you record linear data without gamma and ISO correction curves you will get histogram gaps with fewer bits depth after the 3D-LUT curves are applied for grading.

In general recording the monitoring output would require gamma and ISO curves as well as white balance in order to reduce the number of bits needed to 8 to 10.

Check the monitoring bit depth as that would limit the type of curve that could be used for saving images that way.
 
Hi Dan,

Thanks for your response. I understand most of what you've said but honestly I'm more of a creative than a math geek. Could you please go into a little more depth if you have a moment to bring me up to speed. I understand the sensors A/D are 16 bit but I'm not sure about the linear and histogram stuff you mentioned.

Do you mean essentially that the HD-SDI out of the Epic/Scarlett is 8 or 10 bit and that by recording that signal you've essentially lost well 6-8 bits potentially

If you could record the RAW image out of the HD-SDI then I understand there would be no ISO, white balance or gamma applied but I suppose that could be easily corrected with a simple LUT in FCP or Premier/AVID etc and then finally corrected in a grade of some kind?
 
Hi Doug,

Just to clarify, I'm not talking about recording the RAW stream as a proprietary REDCODE stream but rather as a flat (low contrast) Pro Res Quicktime stream with the RAW look, so therefore increased flexibility in post for indie medium to small budget projects.
 
If i remember correctly, the Epic (and so assumedly Scarlet) lets you monitor in RedLogFilm. Running this signal out to an external recorder would probably be the closest to your idea as you can get now. ISO and white balance are baked in, but the log gamma would give you more flexibility than RedGamma2.
 
What is RAW?

What is RAW?

Since REDCODE (tm) is much like Cineform (tm) and both talk about being "RAW" you need to ask what is it about the signal that is the single point that makes it so called "RAW" and that point is the lack of White Balance (white balance being when two of the three colors are clipped to match the third color).

With a sensor that is close to daylight balance between the red and blue sensors, the red and blue have lower signals than the green, and that is generally a good thing since it gives lower luma noise in the shadow areas and you can use the above white clip data from the red and blue to extend the highlight detail.

To record so called "RAW" data compressed but still linear and not white balanced you need at least 12 bits because you can see banding and histogram gaps if the end graded image has less than 9 bits. The ISO and gamma correction use about 2 to 3 bits at the lowest ISO and several more at higher ISO ratings.

You can record "RAW" data with a log curve in 10 bits, but it reduces the grading range and only at the lower ISO range from the sensor, if you grade up 10 bit "RAW" log data you can get histogram gaps above the bottom bit in the 9 or 8 upper bits displayed.

The main feature of REDCODE (tm) is the ability to adjust the ISO and K after recording, in that regard it shares some of the qualitys of True RAW recording where the actual ADC output is recorded as is.

You cannot adjust the ISO and K value after recording if you only save 8 bit data since you need 9 bits not to see banding and histogram gaps, and any adjustments to the image will produce scaling errors that will increase the size of the histogram gaps.

If you record the "RAW" non white balanced sensor data 10bit LOG using a 10 bit monitoring path without white balance, you can only use the lowest ISO rating(s) of the sensor without getting histogram gaps, so you lose much of what recording "RAW" is about, the wide range of adjustment that makes it better than white balanced compressed recording with small bit depths.

As was mentioned if you record 10bit LOG that has been white balanced and ISO adjusted, you do have "some" limited grading range before the histogram gaps enlarge enough to be noticable if there is some noise in the signal since your eyes fuse tones in consecutive frames and sum adjacent pixels if the image is high resolution and viewed at a distance.

I don't see the advantage to using an external recorder for "RAW" non-white-balanced recording, if the in-camera R3D is going to give you a higher bit depth.

Recording less than 10 bits does not give you "any" grading range before banding, histogram gaps and scaling errors can show up in grading even if the recording is ISO, K and monitor gamma adjusted before being recorded.

In noiseless images banding can be seen in 8 bit images. Cineon (tm) 10 bit log did not use all the bits range and so was near or past the point where banding and posterazation can be seen, but because film scans had high grain noise that grain acted as dithering when the film was projected, also the high resolution allowed pixel summing due to blur from the film recorder lens, the blur in the negative stock shot in the film recorder, blur from the high speed conact printer, and blur in projector lens not being in focus. All that tended to give an extra bit or two of apparent bit depth. I have noticed some people saying that 10 bit Cineon (tm) log is enough for today post production because the films look "great" so there is "no problem", but if you see side by side results from noiseless cameras, there will be some losses with 10bpc DPX over what you get from doing a heavy grade of 16bpc TIF or DPX.

The point of REDCODE (tm) is that you can grade from the wide bit depth recorded direct to the finished image. If you first make a monitor level render at 10bits, then re-grade that, you are introducing scaling errors TWICE and so componding them, so the results will be not as smooth in the tonal rendering because of the irregular stairsteps introduced in the transfer curve between the original sensor data and the light your eyes see.
 
Hi Dan,

Thankyou for taking the time to write a detailed response.

Ok I now pretty much understand what you are saying from a technical stand point. So really if you were going to use an external recorder the best way would be to bake in the ISO and Kelvin balance, so that you're not introducing too many artefacts, whilst possibly dialling in a a flatter look in the LUT. I understand that the RAW image is superior to grade with, no doubt! But for say some corporate work and web work a ProRes recorder could speed up post and relieve you of the need to process and transcode RAW data?

Final question: What do Waveform gaps present themselves as in a final image? I understand banding and have seen it myself numerous times.

Thanks :-)
 
Adam Howden said:
Hi Dan,

Thankyou for taking the time to write a detailed response.

Ok I now pretty much understand what you are saying from a technical stand point. So really if you were going to use an external recorder the best way would be to bake in the ISO and Kelvin balance, so that you're not introducing too many artefacts, whilst possibly dialling in a a flatter look in the LUT. I understand that the RAW image is superior to grade with, no doubt! But for say some corporate work and web work a ProRes recorder could speed up post and relieve you of the need to process and transcode RAW data?

Final question: What do Waveform gaps present themselves as in a final image? I understand banding and have seen it myself numerous times.

Thanks :-)
Click to expand...

If you record ProRes try to use 10bit format (assuming the monitoring output can be 10bit). If you are forced into 8bit monitoring output, you should "grade in the camera" and save as monitor gamma such as Rec709 (Rec709 may have mid-tone a bit off, you will need to learn your workflow and compensate with + or - about 1/4 stop on the exposure). For 10bit monitoring the Cineon (tm) type film log curve can be graded a bit in post but you should try to have the exposure within +/- 2/3 stop in the camera.

What exactly the histogram gaps look like depends on:

1) Will the images be reduced in size, if so you gain tones if all pixels are averaged on the re-sample, OR you gain noise and sometimes tone gaps if the resize is just sampling one source pixel for the one result pixel (that can also cause aliasing and moire).

2) Will the images be sharpened, sharpening can increase the noticeablity of histogram gaps because sharpen's making the difference between tone area's edges greater, so if you get a patch leveled by histogram gaps or noise reduction, or sometimes compression filters, the the sharpen will make a lighter or darker ring around that patch of pixels of the same value.

You get three effects:

1) Bands on areas of low detail and noise, or that are out of focus.

2) Roughness to noise since it takes larger steps, like jumping between tones five apart rather than just one tone apart.

3) Posterization

http://en.wikipedia.org/wiki/Posterization

In the link above you can see that in the image of the snake, the out of focus and the shadow parts show more artifacts than the in focus parts of the image, one reason not to judge a camera or workflow by using a black and white resolution test chart which works fine and looks sharper with just two tones, but real film like images require more than thousands of tones to be smooth.

What happens in grading is in general that you increase the contrast in some parts of the image and when you do that you spread the tones appart on the transfer curves. If you have 16 bits to start with, then you can spread them maybe 128x in contrast before you see gaps artifacts, but if you start with 10 bits you can only use 1.5 to 2x contrast change on the luma or chorma, and if you have 8 bit data, you are already one bit short so should not do any grading but get it right in the camera before recording.
 
Hi Doug,

Just to clarify, I'm not talking about recording the RAW stream as a proprietary REDCODE stream but rather as a flat (low contrast) Pro Res Quicktime stream with the RAW look, so therefore increased flexibility in post for indie medium to small budget projects.
Click to expand...

Have you try Exporting Images and Sequence using these setting out of RCXP, this is what we send out to Post Houses.

Color Space = REDcolor2
Gamma Space = REDlogfilm

You will get a lot more data this way vs what you are talking about using SDI out. SDI is a lower bandwidth pipe by design compare to a R3D 5k Image file out of RCXP. R3D files are technically Compressed Raw files.
 
Hi Dan,

Thankyou once again for your detailed response. I get it now and understand that the waveform gaps thing or the expanding of the tonal range of an image that was captured at 16 bit but is now getting pulled or pushed around at 8 or 10 bit could be bad. I want to upgrade from my 7D kit which has totally paid it self off in under 6 months - excellent. But am just trying to navigate a serious upgrade to a Scarlett-X or eh hem.. a C300 (Ssh...!) so lots to consider kit wise and what not. Thankyou again :-)
 
Hey Michael,

Thanks for your response, I am looking at not buying RedMags initially just to get me started for a month or two. But yes understand that a compressed RAW would be much superior. But a straight to ProRes workflow is a good one to explore as well :-)
 
We got pretty good results with an Atomos Ninja with S-Log output from the Sony F3, and IMHO RED's REDlogFilm is even better. Should get you going until you earned REDmags – far superior to a DSLR anyway.

Regarding the C300: AFAIK you don't get more than 8 bit out of it…
 
One other consideration should be that Epic outputs a 10 bit 422 SDI signal and the Pix 240 records only in 422 as well. Sony S log can be recorded, if i am not mistaken, in 444 10 bit as well the Alexa output can be recorded 444 10 bit. A 422 recording will have less range in grading than 444 so using Red Log Film out to Pix 240 will not have the grading range available to Alexa recording to a Cinedeck at 444. I am using a Pix 240 with my Epic and add some contrast and saturation to reduce the needed range to grade in post. It looks half way between Rec 709 and Red Log Film. I'm still testing in various situation but so far the results are very very nice.

I haven't yet tried making looks from Red Cine X Pro yet. Does anyone know if increasing sharpness in Red Cine X Pro and creating a look, then loading that look into the Epic will result in increased sharpness in the Epic SDI output?
 
conrad gaunt said:
.. its called jpeg2000..
Click to expand...

Plus some colorspace Juju. It's stored in something almost like LAB color space where the HSV channels appear to be split out and then they compress the channel differences or some such. Or at least that's how it used to be a couple years ago. To RED's credit I would also add *Jp2k compatible* not jp2k. The encoder of a codec is often more unique than the decoder. Divx and xvid are different codecs but both are MPEG 4 and compatible on the decode end of things.
 
Mark Crabtree said:
One other consideration should be that Epic outputs a 10 bit 422 SDI signal and the Pix 240 records only in 422 as well. Sony S log can be recorded, if i am not mistaken, in 444 10 bit as well the Alexa output can be recorded 444 10 bit. A 422 recording will have less range in grading than 444 so using Red Log Film out to Pix 240 will not have the grading range available to Alexa recording to a Cinedeck at 444. I am using a Pix 240 with my Epic and add some contrast and saturation to reduce the needed range to grade in post. It looks half way between Rec 709 and Red Log Film. I'm still testing in various situation but so far the results are very very nice.

I haven't yet tried making looks from Red Cine X Pro yet. Does anyone know if increasing sharpness in Red Cine X Pro and creating a look, then loading that look into the Epic will result in increased sharpness in the Epic SDI output?
Click to expand...

Hey Mark
I - and others I think! - would LOVE to hear more about your exploits with the PIX240. I intend to get one and ANY insight into how you set up the camera to feed it would be fabulous! May want to start another Thread on this but do hope you will post your findings and workflow.

regards
Stu Aull
Alaska
 
While there is no disputing Jim's position that 1080 has no future, at this time a lot of my work has no future beyond 1080. TV commercials are the lions share of my work and few play more than a few months. Having a 1080 recording on the spot greatly increases our productivity. We have, since 2008, converted this work from 4k raw to 1080 CineForm 444 and then graded this with first light and color finesse. Comparing the graded CineForm 444 with the graded DNXHD 422 10 bit shows almost identical results for us. We do need to get a good white balance on every setup because we are of course recording 422 and not raw. For now I am double recording with both Red Mag and Pix SSD because the Pix 240 is still early firmware and we have had some intermittent corruption issues. Sound Devices is a great company and they are on top of the problem if there even is a problem. After reloading the latest 1.05 drivers using the loading utility program instead of the copy to SSD method, I have had no further corruptions. They are also recommending completly booting the Epic before booting the Pix, then genlocking the Epic to the Pix. I just ordered my Epic sync cable so I have yet to try it. I'm not sure I really like this genlock idea, just another link in the chain to cause potential problems, but, whatever works for now. Regarding sharpness, CineForm converter includes a sharpening tool, so if there is any difference between the CineForm and DNX it would be that tool. We have some great sharpening tools for post and can match just fine, but I would love to be able to dial in a little sharpening on Epic's SDI output. That would great for those times the producer just wants the DNX or Prores dumped to a drive so they can hop a plane.
 
You must log in or register to reply here.
Back
Top