What Type of Compression do Red Cameras Use? Wavelet?

[Desmond Ford]

What is wavelet structure compression and why is it so CPU intensive? Is it motion or temporal compression, or neither?

If I understand correctly, Red cameras use Wavelet Structure Compression. That's why when I set up an Epic, I must chose a compression level in my settings, such as 8:1 all the way up to 16:1 or whatever...

Does it follow that if there were no compression, then transcoding would be significantly faster? For example a 10GB uncompressed Raw file would take significantly less time to transcode than a 10GB 8:1 compressed file...

 

[Brian Merlen]

you gotta ask one of the red folks about that one... i wondered the same thing. been shooting a lot of 3 to 1 low res scarlet stuff....

 

[Nick Pasquariello]

Flipside of that thought: if the images were not being compressed, there would be way more data per frame, and you would have to write all that data to the SSD cards at *CurrentRecordingFramerate* frames per second. Write speeds simply are not fast enough to make this feasible. Nor is there an overall noticable quality improvement from having 0 compression versus 3:1 or even 6:1.

And it would also take way, way longer to unload your footage off the mags. As in 8 times longer for 8:1 footage versus 1:1. This would create a huge bottleneck on set for any digital loader or DIT.

And! Working with uncompressed raw R3D footage would also create a bottleneck of reading off whatever drive/RAID you house the media on.

Basically, any gains you would get in CPU performance are eaten up and made worse by data input/output limitations of whatever magazine/drive/storage you're using.

 

[Evin Grant]

Red rocket FTW! Small investment for the ability to process/edit in realtime.

 

[Les Dittert]

It would be desirable for some of the cropped sizes on Red to shoot uncompressed, and the data rate would be manageable .
The Black Magic camera does this for 2.5 K .

For extra slow frame rates and stills, the Red should not compress at all, obviously. There may not be a data path in current generation cameras to allow this, however.
I hope there is.

Flipside of that thought: if the images were not being compressed, there would be way more data per frame, and you would have to write all that data to the SSD cards at *CurrentRecordingFramerate* frames per second. Write speeds simply are not fast enough to make this feasible. Nor is there an overall noticable quality improvement from having 0 compression versus 3:1 or even 6:1.

And it would also take way, way longer to unload your footage off the mags. As in 8 times longer for 8:1 footage versus 1:1. This would create a huge bottleneck on set for any digital loader or DIT.

And! Working with uncompressed raw R3D footage would also create a bottleneck of reading off whatever drive/RAID you house the media on.

Basically, any gains you would get in CPU performance are eaten up and made worse by data input/output limitations of whatever magazine/drive/storage you're using.

 

[Nick Pasquariello]

It would be desirable for some of the cropped sizes on Red to shoot uncompressed, and the data rate would be manageable .
The Black Magic camera does this for 2.5 K .

For extra slow frame rates and stills, the Red should not compress at all, obviously. There may not be a data path in current generation cameras to allow this, however.
I hope there is.

Fair points. I wasn't considering the stills-workflow.

 

[Dave Blackham]

Threads here mentions jpeg2000

 

[Pawel Achtel]

It would be desirable for some of the cropped sizes on Red to shoot uncompressed, and the data rate would be manageable ...

Why on Earth would you like "uncompressed" when you can get no-lossy compression at 1/3 of the data size?

 

[Les Dittert]

Why on Earth would you like "uncompressed" when you can get no-lossy compression at 1/3 of the data size?

I believe that the minimal image processing that is happening ( J2K and some dark smoothing ) ruins the image detail .
Why else would my $250 craigslist Canon XTi ( similar pixel pitch) have more detail that my Scarlet, when comparing RAW to RAW ?
( carefully shot same scene with same lens, racking focus on each to make sure it was the best, etc )
Especially in the shadow detail. The RED looks to be running some big lowpass filter on the darkest darks. ( good compression cheat )
When I first got my Scarlets, I thought it was defective and called support.

Thats why

edit: If you mean they should implement lossless j2K, like bit for bit lossless, I fully agree and disregard my post

 

[Pawel Achtel]

I don't understand. How do you expect to get a sharper image from "uncompressed" than from no-lossy compression? Please explain.
Also, How can you possibly compare Canon RAW to Red RAW?

 

[Les Dittert]

As I said in my post edit: Sure I'd like lossless j2K compression, that would be fine. They don't implement that , do they . If they can add that for slow frame rates it would be fantastic.

Why should I not compare Canon ( or Nikon or whoever ) RAW image to RAW from Red ? Especially if the sensor has about the same pixel size.
I want to see similar image sharpness coming from about the same sensor size. Sure , the Red has to read out much faster, but the pixels should have similar 'quality' if you will.
Especially if comparing stills.
Of course, The $250 Canon can't shoot movies, but as I said before, I am looking at detail and noise. Raw vs RAW.
I'm looking at what is possible from a similar sensor size and pitch.

The Red may have an extra dose of optical low pass filter to make damned sure there are no aliasing artifacts, but still...

 

[Pawel Achtel]

As I said in my post edit: Sure I'd like lossless j2K compression, that would be fine. They don't implement that , do they . ...

they certainly do. I'm not sure about Scarlet, but Epic has 3:1 compression option, which is lossless.

BTW, RED RAW 6:1 is so close to lossless that I can't tell the difference by pixel peeping 3:1 and 6:1 which one is which.

Why should I not compare Canon ( or Nikon or whoever ) RAW image to RAW from Red ? Especially if the sensor has about the same pixel size....

This wasn't my question. My question was "How?"

AFAIK you can only compare processed immages and the amount and type of processing is not directly comparable between the two.

Again, how do you compare RAW images?

 

[Pawel Achtel]

Low pass filter is required for both motion and stills and, either way, I still do not understand how recording uncompressed images (whether with or without low pass filter) will achieve anything different than recording them lossless compressed.

 

[Les Dittert]

Here is how I compare the RAW images ( Red calls r3d raw, but it really isn't , but for arguments sake Ill go with that)

i shoot with the Red, process r3d in RCX to a 16 bit tiff sequence. Highest quality settings, non clipping color correction.
I then shoot same scene with Canon in CR2 raw format. Process in adobe camera raw to 16 bit tiff at best settings and no clipping in blacks or whites.
Both shoots rack focus slowly to ensure the best focus is captured somewhere in the sequence of frames. On tha canon I would shoot a dozen or so frames. The red gets more frames obviously because it's a movie camera.

I then color grade one to match the other . A curve fit match is enough to match if I'm mainly looking at detail issues. These steps are now in Fusion or nuke, high bit depth.
zoom in 800% of so and look at fine details.
For extra fun, look at the dark shadow details, brighten to see what detail there is visible in each. The red tends to lack detail or even noise at the darkest, almost black areas. The lack of noise is not consistent with the noise in brighter parts of the image.

Have you tried this ?

I plan on repeating this test when I receive my black magic 2.5 K camera. It has no compression and an actively cooled sensor. Should be revealing . I'm guessing Red will up the priority of improving demosaic in RCX , which will be welcomed by many.

 

[Pawel Achtel]

OK, so you don't compare RAW, you compare TIFFs. And, the debayer, sharpenning and a host of other processing occures with both. You are comparing two different processing algorithms.

Having shot countless SFR charts with the Epic and analysing them with Imatest I can safely say that the Epic sensor (together with the low pass filter) resolve right up to the Nyquist frequency limit, which means that, if you are getting "sharper" pictures with the Canon, that extra "sharpness" is quantisation noise due to insufficient low pass filtering of high frequencies that exceed the theoretical resolution limit for the sensor.

Sure, the roll-off may be gentler on the Epic due to more aggressive low pass filter, but this can be easily "rectified" with sharpenning unlike in the case where there is insufficient low pass filtering and you end up with false sharpness (essentially quantization noise).

 

[Les Dittert]

We have to compare processed images, that is what humans will end up seeing.
I've also compared the R3D RAW bit planes as well, RGGB . I have also debayered these with Cineform , after making Cineform RAW's out of them.
I'm no ordinary user either, bro

I urge you to take a DSLR and repeat my test. Don't keep comparing Epic images with Epic images. Look in the darks for detail.
I'm not talking fake sharpened detail here.
The test is a good data point and educational.

I'll repeat my test as well, just to be sure, and post images. My old Canon 400d ( XTi in the USA ) has 5.7 micron pixels, a close match to the Scarlet.

Also, Jpeg 2000 lossless does not get anywhere near 3:1 on typical sensor data.

I just wanted to add that I like my Red cameras, and only want them to improve as a result of our discoveries and experiments and revelations.

 

[Desmond Ford]

Flipside of that thought: if the images were not being compressed, there would be way more data per frame, and you would have to write all that data to the SSD cards at *CurrentRecordingFramerate* frames per second. Write speeds simply are not fast enough to make this feasible. Nor is there an overall noticable quality improvement from having 0 compression versus 3:1 or even 6:1.

And it would also take way, way longer to unload your footage off the mags. As in 8 times longer for 8:1 footage versus 1:1. This would create a huge bottleneck on set for any digital loader or DIT.

And! Working with uncompressed raw R3D footage would also create a bottleneck of reading off whatever drive/RAID you house the media on.

Basically, any gains you would get in CPU performance are eaten up and made worse by data input/output limitations of whatever magazine/drive/storage you're using.

Thanks for the response.

However, won't Thunderbolt SSDs obviate that issue? I mean, once we can have 2TB SSD Thunderbolt drives?

I just bought an Elgato Thunderbolt 240GB SSD. THis thing is unbelievablly fast. I can do 10GBs in less than a minute. So once SSDs come down in price and they start creating larger Terabyte size SSDs, shooting in uncompressed Raw wouldn't create any bottlenecks, would it?

 

[Desmond Ford]

But what about Wavelet... what is it exactly and why would Red choose this compression over others?

Is it the industry standard? Do most or all cameras use Wavelet Structure Compression?

 

[Les Dittert]

My suggestion for uncompressed was for stills and slow frame rates only, where the SSD bandwidth is no limit.
I will be shooting my DSLR RAW vs RED RAW test this evening.

 

[Phil Holland]

But what about Wavelet... what is it exactly and why would Red choose this compression over others?

Is it the industry standard? Do most or all cameras use Wavelet Structure Compression?

There are many methods of compression (and many recording formats) and there isn't an industry standard overall. I imagine along the way during the initial R&D on what we know as RedCode Raw today that Red discovered that wavelet was a good way to go to keep file sizes small and maintain good results at various ratios without introducing any visual compression artifacts like "blocks".