RED Workflow, SDK & Access to RAW data

[Kyle Mallory]

I'm not sure why I'm writing this, but I've seen a number of comments, complaints, etc (perhaps even one or two of my own) about the lack of access to the RAW image in the RED SDK. And I want to share my opinion on the subject, for what it's worth.

Before I dig into my thoughts, let me pre-empt myself by saying that prior to B16, I had successfully decoded the R3D data, including image and audio. I had a fully functional, if slow, R3D workflow for IFX's Piranha that included 16bit audio, and 16b/component images. I had also decoded approximately 90% of the available metadata. The shortcomings of my decoder, was unable to process/color match the image using the metadata, and I had implemented only a few, very simple debayer algorithms.

I say this, so you'll understand that I've successfully faced the internals of the R3D file format, like few others outside of RED have. I have also had access to the RED SDK for OSX for a number of weeks now.

So, when I hear comments and criticism with statements like "RED should allow access to the RAW data via the SDK." I find myself perplexed. While I appreciate that some want to offload the debayer process to the GPU, the reality is, for 95% of what anyone and everyone wants to do with their RED footage, can be done using the SDK (minus audio) in its current form.

Additionally, there is as much, if not more processing occurring within the Wavelet/J2C decoding than in the debayer process. In my efforts, using an open source, reference J2C codec, processing a single R3D frame, it took almost 10x longer to decode the frame than it did to debayer it. Understanding that the library is not heavily optimized for performance (unlike RED's SDK), neither was my debayer algorithm.

The biggest problems I faced while implementing my decoder was calculating/populating the respective color transformation matrices necessary to apply the metadata to the "RAW" image to give me the "one light" result.

There are countless algorithms and articles on how to calculate and apply color, tint, white balance, exposure, etc, but unless you have the same "magic numbers" that RED has (ISO 250 = XyZ transform, you will never match the final output of RedCine with that of your own home-grown solution. If you can't match the image characteristics of RED, aside from Timecode and the upcoming lens i/data, then there is absolutely not point to having the metadata.

Understand this: If the metadata is meaningless, then so is the image. Without it, there is little more than your subjective eye that can be used to quantify the results that you get from the RAW, bayered data. Without the SDK, and its internal processing of the RAW image/metadata, any image you might get is essentially worthless. In other words, if you want access to the RAW/debayered data, you might as well throw away the SDK, because the beauty and value of what the SDK offers is its ability to do that magic processing of the image data into something that is quantifiable.

Now, there is an argument that can be made for having "unprocessed frames of 'RAW' RGB data., leaving the work of color matching, exposure compensation, etc. to your favorite application, and bypassing all the metadata and leaving it to your subjective eye. You can still do this with the RED SDK (since it returns an image of RGB data). Simply set the gain values to 1.0, the curve flat, the white balance to 5000 Kelvin, ASA/ISO to 320, color space to Linear or CameraRAW, and everything else to 0.0 or 1.0 (whatever the default is), and you should be getting an unprocessed image.

So, please don't bash RED for not giving you access to the RAW image. Chances are good, if you are like me, you wouldn't know what to do with it anyway. :blink:

Now, having defended RED in this regard, I do want to voice my frustration at the very RED notion of trying to make the R3D format proprietary, and not publishing the specs for the format and the magic numbers of the transformation matrices. I get that RED has intellectual property here, but, having gone through the R3D format myself once already, let me be the first to say, there is little proprietary about it (aside from those magic numbers). RED has stated, at the discontinuation of some of their products such as the matte box, etc. that RED is a CAMERA company. Not a camera accessory company.

I am disappointed that RED has chosen to selectively ignore this position when it comes to their software (Sorry Deanan, Rob, Greame). If RED was truly a Camera company, they would have made life simple for everyone and simply adopted the DCI standard of JPEG2k and MXF as the "data standard", and focused solely on making a camera that "Just works". Pick a standard, or *publish* your own, and let all the software companies do what they do best, write software; while RED, as a camera company does what they do best, build kick-ass cameras. Sure, go ahead, give Apple and Scratch the heads up, and let them build their software. MiniDV and MPEG-2 fought with differences in codec implementations, and occasionally Sony or Pana or Canon would argue "Its the codec..." and someone would do a little research, and blog about it, and it would get discussed to death on DvxUser or DVINFO, but then a general consensus would be reached about a new, better software codec from MainConcepts, or Cineform, or Sorensen, or JimBob's custom SSE2-optimized codec, and everyone would move on, better for it-- and Panasonic/Sony/JVC/Canon would go back to what they do best, making overpriced bricks of plastic and silicone and copper, and calling them Cameras.

So, to finish my general rant... RED: Fix your post workflow, fast, or give it to the community, and let natural selection do what it does best. Everyone else, in the meantime, pipe-down and be glad that the SDK gives you RGB processed data, instead of RAW bayered data. You'd be pissing and moaning even more if the didn't process it for you.

You're comments, criticism, and corrections are welcome. It's late, I've got a headache, and need to go to bed.

 

[Jay A. Kelley]

Good post, makes lots of sense.

Jay

 

[David Newman]

Many users have assumed, incorrectly, that RAW access was necessary to preserve the camera dynamic range. In our new converter we show that is not the case. However, we strongly request for RAW for performance reasons, your 10x decode to cheap debayer tests doesn't apply to a quality debayer within the SDK, which I believe is currently more compute intensive than on the decode. There is a third party opportunity to do faster and potentially higher quality demosaic in different parts of the processing chain, including GPU acceleration.

 

[Mike Zinner]

A couple of points I see from a different point of view.

1) The Red camera would not exist at this price point if they wouldn't have figured out the in-camera-compression, REDCODE(TM) RAW, to make the amount of data manageable. So this *is* an essential part of the camera. And it is their right to protect that, if we like that or not.

2) Direct file access - like you have done before - has been prohibited by Red and starting from Build 16 the bitstream of RAW data is encrypted (according to my sources) so even if you would disobey their rules, you could still not access it anymore.

3) The main reason I want to see RAW data access in the SDK is that people skilled enough to use it (I am not) can create even better workflow solutions to create competition. And this competition is beneficial for everybody since it will drive development and result in better tools for everybody.

Sure, the fact that the SDK offers de-bayered image data helps me write my own simple tool much easier. But for people much more skilled in this area - like the CineForm guys - not having direct access to the RAW data slows down their conversion tool and makes their workflow much less attractive.

4) That the SDK does not include some features as of now (like sound) is clearly caused by limited resources on Red side (Rob and team, we appreciate your dedication and workload a lot! ). That it does not include other features like RAW data access even in the first version is beyond me - since that is what the currently exported functions use and what should be much easier to export.

One thing I notice was that Jim modified one of his SDK discussion post http://www.reduser.net/forum/showpost.php?p=262061&postcount=87 from "Full access to RAW is coming, ..." to "Full access to RAW is on the roadmap, ...". Depending on how you see it, that could put the availability of direct RAW access months away.

5) Anyway - now that CineForm has found a way to work around this by back-converting the RGB image data to CineForm RAW, and by Jim's statement that direct RAW access will be available in the future - the only thing that keeps me from updating my cam from Build 13 (where the old CineForm converter still works) is the missing support of Sound in the SDK. And this is coming as soon as Rob gets to it.

So I consider everything resolved. Why are we have this discussion again?

 

[Kyle Mallory]

Many users have assumed, incorrectly, that RAW access was necessary to preserve the camera dynamic range. In our new converter we show that is not the case.

Exactly. You know as well as I do, that, as long as the RGB data coming from the SDK is 16bit/comp, and no other transformations are being applied (or the equiv of an identity transform), and you're using a simple linear debayer algo, we *should* be getting every bit of data that was in the original Bayer pattern image, albeit demosaiced, with at most a few floating point precision errors (which would likely get lost in the conversion back to 16bit). Only RED really knows if that is true, but I trust their passion for quality.

However, we strongly request for RAW for performance reasons, your 10x decode to cheap debayer tests doesn't apply to a quality debayer within the SDK, which I believe is currently more compute intensive than on the decode. There is a third party opportunity to do faster and potentially higher quality demosaic in different parts of the processing chain, including GPU acceleration.

I agree. Which was why I said "95% of everything anyone wants to do". I thought it went without saying that Cineform would be one of those other 5%. Myself included, I would imagine most SDK users are looking for one of two things: A) to programatically access/do what RedCine does-- process R3D and export images; and B) to read R3D metadata.

 

[Dan Hudgins]

Artifacts...

Artifacts...

Exactly. You know as well as I do, that, as long as the RGB data coming from the SDK is 16bit/comp, and no other transformations are being applied (or the equiv of an identity transform), we *should* be getting every bit of data that was in the original Bayer pattern image, albeit demosaiced, with at most a few floating point precision errors (which would likely get lost in the conversion back to 16bit).

Many of the sample TIF files I have been using for compatibility testing show heavy artifacts (the artifacts look to most people as an "improvement" on first glace) with the chroma noise filtering and chroma edge sharpening, these do not look film-like and scream "I came from a Digital Cinema camera". Since I do not know exactly what was done to these sample files, it is hard to say what settings in the SDK might be able to dampen the artifacts, if they can be.

I do not like these artifacts and would rather be able to do a de-Bayer that does not use any of these methods that sharpen the image or the chroma or take the chroma detail out of the images.

Since I am interested in going out to film the "resolution" needed in the 4K frames is only about 1280x720p about what you get on a movie screen with the projector a little out of focus at 80%MTF.

Putting a sharpened image out of focus or doing a blur on it will not result in the same artifact free image that you would get if the artifacts were not put into the image in the first place.

It would be a simple thing to add a RAW port to the RED ONE (tm) camera and record 24fps directly to 48 compact flash cards in parallel sets. I am only interested in 24fps shooting, for the most part. The reason given for not doing this is that others want to shoot slow motion. Well, why not offer a 24fps solution to those who want it, too few sold to make the effort pay for itself? Filmmakers have been and will be too small of a group to get much, the Videographers have been those who made the RED ONE (tm) sales volume high enough for the camera to be a financial success, and they do not seem to notice the video/digital like artifacts or at least want to avoid them.

 

[Kyle Mallory]

A couple of points I see from a different point of view.

1) The Red camera would not exist at this price point if they wouldn't have figured out the in-camera-compression, REDCODE(TM) RAW, to make the amount of data manageable. So this *is* an essential part of the camera. And it is their right to protect that, if we like that or not.

Not to offend anyone at RED, but... They didn't figure out the in-camera-compression. The JPEG consortium did. What RED did was implement an FPGA version of that codec for use in the camera. They spent a lot of time, energy, and money on that implementation, and there is IP in that. But at the end of the day, REDCODE is nothing more than a repackaged, specialized JPEG2000 implementation. No amount of fancy names or marketing hype will change that. At least Sony, the masters of all things proprietary, owns up to the fact that HDV is still just MPEG-2.

2) Direct file access - like you have done before - has been prohibited by Red and starting from Build 16 the bitstream of RAW data is encrypted (according to my sources) so even if you would disobey their rules, you could still not access it anymore.

Trust me, I know. Don't get me started... Suffice it to say, encrypting an open standard, and wrapping it in a proprietary container and calling it your own is, well...

3) The main reason I want to see RAW data access in the SDK is that people skilled enough to use it (I am not) can create even better workflow solutions to create competition. And this competition is beneficial for everybody since it will drive development and result in better tools for everybody.

If this is your ultimate goal, don't ask for access to RAW data, that is like asking for the pie, instead of the recipe. Ask RED to release the R3D specs, and stop encrypting the data, so people like the David's can do what they do best, and create even better workflow solutions to create REAL competition.

4) That the SDK does not include some features as of now (like sound) is clearly caused by limited resources on Red side (Rob and team, we appreciate your dedication and workload a lot! ). That it does not include other features like RAW data access even in the first version is beyond me - since that is what the currently exported functions use and what should be much easier to export.

In my efforts, audio was the first thing I decoded. It took me all of about 3 hours to extract WAV files from R3D files. It is trivial. Perhaps too trivial. I'm not knocking Deanan or Rob for that they haven't done. I am totally on board with their priorities and with what they have done so far. First priority is to get images out of the camera. After all, it is a camera, not a portable audio recorder. Again, my guess is, 95% of all SDK users want a usable image- RAW would be meaningless to them, so Bayer-RAW takes a back seat to getting a processed RGB image. For me, audio is a far higher priority that RAW.

So I consider everything resolved. Why are we have this discussion again?

Because I was on one last night, after people were bitching about how workflow sucks because RED didn't provide access to the RAW data via the SDK. In fact, the reason workflow sucks is because RED didn't just release the R3D specs (or stick to an industry standard) and let the industry do that competition thing that RED is perfectly okay to do when it comes to "hard" accessories (hardware), but not when it comes to "soft" accessories that support the camera. And, breath...


Kirk Out.

 

[Kyle Mallory]

Many of the sample TIF files I have been using for compatibility testing show heavy artifacts (the artifacts look to most people as an "improvement" on first glace) with the chroma noise filtering and chroma edge sharpening, these do not look film-like and scream "I came from a Digital Cinema camera". Since I do not know exactly what was done to these sample files, it is hard to say what settings in the SDK might be able to dampen the artifacts, if they can be.

I'd suggest then bringing this up with Rob/Deanan (if you haven't). I don't want artifacts in my images anymore than you do. If they are a result of the SDK, then RED needs to fix them.

If you can, I'd be interested in seeing the images (or regions of the images) showing the artifacts, so I know what to watch for in my own stuff.

 

[laguun]

Direct access to the RAW format is crucial.

Especially GPU-basing realtime debayer (as iridas speedgrade), custom realtime 4K hardware (as DVS Clipster), clustered & distributed realtime systems (as filmlight baselight), linux basing top-end systems (as discreet smoke, fire, flame, inferno, burn etc), realtime server-basing systems (as Quantel Q-Series) wont be able to really transmit its horspower to the road when going through an SDK.

To gain an excellent reputation in the etablished posthouses, red needs these manufacturers behind them. Their combined manpower, expertise and sales intention can advance reds post to an excellent level within a short amount of time, something red on its own will only be able to realize with dozens of manyears R&D.

The existing excellent solutions for red which so far run only behind close doors and under strict NDAs are all held back by the respective manufacturers - and many advanced technologies simply cant be utilized by the SDK, or an SDK in general.

In my humble opinion, RED should allow native access to their JPEG2000 raw files for the top players in the post market. The lack of native support is *the* weak link in reds production chain.

Decoding is an option for houses as ours, who can throw 50+ processors at a single decode job - but this is, today, still untypical for the avergae post boutique.

Dizens of productions, who *really* do like the camera, have switched back to the wait-and-see.

If the closed format policy stays in place for everyone than 2 secondary players, as nice their products are, red wont be able to really set free the avalanche as the post-pioneers will be the bottleneck and critics will have a clear and valid argumentation against the quality/cost benefit.

Just my 2eurocent.

 

[Dan Hudgins]

To each his own...

To each his own...

then RED needs to fix them.

I think these artifacts would not be viewed as "bugs" by the RED Team, but rather be their "best" compromise in reducing chroma moire and holding the resolution measurement up, with the OLPF used.

If the raw (uncompressed) data was available then one could pick a different balance between color quality, resolution, and sharpness. With the compressed data they may need to filter the color to reduce some compression related artifacts, while introducing others.

The open source DCRAW (tm) de-Bayer program may show some similar artifacts in some images, so I think it has to do with the de-Bayer method and filters used in and around the de-Bayer, along with the OLPF, sensor, and lens used. The softer the image on the sensor the less some of these issues comes out during the de-Bayer.

Future builds yield different results, but not the same as true RAW data might.