Compression, Math & the Internet

[Steve White]

Obviously RED RAY sounds amazing on paper:
2 hours of 4K on a ~9 GB disk.

People are crunching numbers and pushing out fascinating results, like 8 hours of 1080p and upteen hours of 720p. This math is wrong, and here's why:

The RED RAY codec is going to be an algorithm that compresses real world images. The amount of available detail in a given frame does not scale linearly with the number of pixels. There's only a finite amount of observable detail in any given real-world image, plus noise.

Even at 1080p, focusing errors and softness appear in most current films on Blu-Ray. Going to 4K is not going to make these films sharper - the information simply isn't there. Going to 4K won't make the picture worse, but it can't make it better either.

Moreover, there are no details out about the sampling of the RED RAY 4K. Your average Blu-Ray disc is 8-bit 1920x1080 4:2:0, giving a 3.11 MB/frame (uncompressed). 540 GB for a 2 hour film.

Suppose RED RAY uses a Bayer pattern encode at 4096x2304 and 12-bit, which gives 14.2 MB/frame. That's 2.45 TB for a 2 hour film.

Obviously neither Blu-Ray nor RED RAY can sustain these data rates with their 25-50 GB and 9 GB capacities. It therefore follows that you can break any of these codecs by feeding material that actually has bandwidth approaching the theoretical data rates of the uncompressed image stream.

Anyway - we'll just have to wait until the RED RAY codec is available to see under which scenarios it behaves. By that time, I don't think many people will care that RED RAY fits on a 9 GB disk - because that kind of bandwidth is almost feasible over the internet.

If the RED RAY codec becomes a viable internet distribution codec for HD and 4K video, I think that's how it will kill Blu-Ray, but I don't expect the images coming out of it to be significantly better than Blu-Ray currently provides. If they are, I will be pleasantly surprised.

 

[Dj Joofa]

Obviously RED RAY sounds amazing on paper:
2 hours of 4K on a ~9 GB disk.


Your average Blu-Ray disc is 8-bit 1920x1080 4:2:0, giving a 3.11 MB/frame (uncompressed). 540 GB for a 2 hour film.

Suppose RED RAY uses a Bayer pattern encode at 4096x2304 and 12-bit, which gives 14.2 MB/frame. That's 2.45 TB for a 2 hour film.

Spiff, actually if you look at the numbers for Red Ray, it appears they are manageable to some extent. Using your numbers for HD 540 GB for 2 hour film crammed into a 9 GB disk gives a compression ration of 9/540 = 1:60. Firstly, 1:60 is decent sized compression, not a whole lot, considering that many HD video conferencing systems do a decent image at around 1:300 compression ratio. Of course, for a video conferencing environment the image is assumed to have little motion. However, if you take a ratio of 300/60 = 5, that tells us that our HD movie can have about 5 times more motion on average than a typical video conferencing scenario, which is ok with me.

Now look at Red Ray's data. 9 GB / 2.45 TB ~= 1:270, which is approaching the figure for 1:300 for HD video conferencing. However, as happens frequently in image compression, the data rate does not scale the same way as the image size increases. I would estimate that compression ratio for Red Ray to be around 1:150. Hence, compared to HD video conferencing it is about 300/150 = 2.

 

[Steve White]

Keep in mind that I limited the data rates by chroma subsampling for the Blu Ray, and assumed a "RAW" compression for the RED RAY. This 270:1 image compression we're talking about is pure data reduction based upon redundant information within successive images.

Perhaps I was being unfair with my 12-bit resolution. So instead lets give RED RAY 8-bit at 180:1 compression. Your average mathematically lossless compression ranks between at 4:1 and 2:1. And I don't know about you, but I can sure see the artifacts in Blu-ray at 22:1 (assuming 2 hours in 25 GB).

My main point however is that this 180:1 data reduction raises eyebrows... but is not the relevant number. The relevant number is 9 GB, and whether or not enough information can be fit into 9 GB that the average person can't tell anything is missing in 2 hours.

 

[Dj Joofa]

Keep in mind that I limited the data rates by chroma subsampling for the Blu Ray, and assumed a "RAW" compression for the RED RAY. This 270:1 image compression we're talking about is pure data reduction based upon redundant information within successive images.

Of course, even my figure of 1:300 for video conferencing is for 4:2:0 sampled data.

Your average mathematically lossless compression ranks between at 4:1 and 2:1. And I don't know about you, but I can sure see the artifacts in Blu-ray at 22:1 (assuming 2 hours in 25 GB).

4:1, or 2:1 is for lossless compressions such as zip/bzip, etc. as you already point out. Lossless figures don't concern us here. As far as seeing artifacts in Blu-ray, may be that is because you are seeing it on an HD TV at home. For theatrical projection, the discerning power for signal variation reduces a lot because of many factors, with the contrast reduction due to the presence of flare in a theater not being the least of them. Hence, Red Ray can take advantage of that fact and smooth and blur the image to some extent as some detail is not going to be noticeable in a theater, which is otherwise visible on a computer monitor. That can help Red Ray fit more data in on a 9 GB disk.

 

[Steve White]

That can help Red Ray fit more data in on a 9 GB disk.

Just to be picky, I don't like this phrase. You can fit 9 GB of data on a 9 GB disk. What we're concerned about is how much data you can to represent with 9 GB.

It's obvious that you can stick a whole movie onto 9 GB - it's been done for years. What RED RAY offers is an opportunity to make that 9 GB yield a prettier image than the DVD MPEG-2 codec.

I'm awfully tired of picking out compression and edge enhancement artifacts from video. However, I'm not interested in blaming monitors for showing me compression artifacts. I want the display makers to try their best to accurately display their source, and if that source is loaded with errors, I'd like to know.

I hope RED RAY doesn't come with a whole slew of compression artifacts. I assume we won't see macro-blocking and mosquito noise. I assume softening and blur are "better" since they are less noticeable.

Ultimately however, I'd like my eyes to be the limiting factor with my motion picture displays. Right now, that doesn't appear to be the case. As a result, I still haven't purchased an HDTV.

 

[Dj Joofa]

I hope RED RAY doesn't come with a whole slew of compression artifacts. I assume we won't see macro-blocking and mosquito noise. I assume softening and blur are "better" since they are less noticeable.

A whole lot of noise artifacts creep into "video" because of block-based nature of compression. H.264 is typically applied to blocks in an image. I think blue-ray is also H.264 based and perhaps block based.

Red Ray has an opportunity to do full frame compression such as wavelets and hopefully such noise should be more limited.

 

[Bing Bailey]

the DCI Master spec is based on wavlets and jpeg 2000 codec

 

[Zach Nelson]

I'm awfully tired of picking out compression and edge enhancement artifacts from video. However, I'm not interested in blaming monitors for showing me compression artifacts. I want the display makers to try their best to accurately display their source, and if that source is loaded with errors, I'd like to know.

amen.

 

[Steve White]

Some of the Blu-ray discs out there are MPEG-2, and those ones look like crud. The better disks are H.264. Both of these algorithms are DCT and motion estimation based, and hence both feature mosquito noise and macro-blocking.

From what I understand of wavelet encoding, images are usually full-frame encodes, which does eliminate mosquito noise from the artifact list. Instead, overcompression causes softening in the picture. I'm not familiar enough with bad wavelet encoding to know what other artifacts one can expect.

Either way, in order to obtain the 22:1 (let alone 140:1) compression ratios we're talking about, temporal compression methods (usually group-of-pictures (GOP)) must be used. For low motion recording motion estimation and macroblocks do a great job, since the difference between successive pictures is minimal. However, since noise differentiates any two otherwise identical successive (real-world) frames, some amount of noise rejection is required for GOP to be effective. The difference between noise and the image is contextual however, so if your GOP codec is intelligent, noise rejection doesn't have to be a brutish attenuation of high frequencies.

I recall that Cineform uses something like a 2-frame GOP in their wavelet codec to boost compression efficiency. Their range is usually somewhere around 6:1 and 10:1 to achieve their "visually lossless" moniker. RedCODE RAW seems to be batting between 10:1 and 12:1 using an I-frame only approach.

 

[Bing Bailey]

what RED are about to attempt is amazing and the work of a mad genius lunatic. they are taking over acquisition with the RED ONE and RED CODE, They are taking over the Lens Market with their Lenses. They are doing most of their own accessories at lower cost. now they want to take over the post production and playback side of things with REDRAY, 4k Displays and Projectors.

I hope jim has bodyguards and armor because he's going to ruffle feathers in about 6 different industries. he may need a RED army and Im not talking about the one in CHINA

 

[Dj Joofa]

Some of the Blu-ray discs out there are MPEG-2, and those ones look like crud. The better disks are H.264. Both of these algorithms are DCT and motion estimation based, and hence both feature mosquito noise and macro-blocking.

A small correction. H.264 is based upon an (integer) transform derived from DCT. It is not exactly DCT, however, it does have the property of being perfectly invertible. In addition, H.264 standard allows for a de-blocking filter to reduce some of the blocking artifacts.

Either way, in order to obtain the 22:1 (let alone 140:1) compression ratios we're talking about, temporal compression methods (usually group-of-pictures (GOP)) must be used.

Yes, Red Ray has to be spatio-temporal to get compression ratios like that. In addition, there are only so many known ways of doing compression, and notwithstanding the mystery/hype surrounding Red Ray, it might be a variation on the known ways to mix spatial wavelets with temporal compression.

 

[Radoslav Karapetkov]

It'd be super cool if Jim resurrects HD-DVD for RED RAY and focks Sony good.

 

[Dj Joofa]

It'd be super cool if Jim resurrects HD-DVD for RED RAY and focks Sony good.

It is not easy to shatter a known standard such as H.264 based coding even if Red Ray comes up with its own known notion of spatio-temporal based compression comprised of wavelets. I think such compression will be designed from the point of view of projector/displays only, but it will suffer when internet streaming is considered. Wavelets do have some advantage over block-based coding, however, on the other hand, block-based coding does have certain advantages when it comes to internet and streaming. Can't have your cake and eat it too always.

 

[Steve White]

A small correction. H.264 is based upon an (integer) transform derived from DCT.

Thanks for that! I haven't read enough literature about H.264 to accurately reference the standard. Likewise, I'm not sure Blu-ray is technically H.264, though it certainly belongs to the same family as H.264, WMV-HD, and AVC-HD.

block-based coding does have certain advantages when it comes to internet and streaming.

Can you list or refer to some of these advantages? I was under the impression that the block-based encoding schemes won out previously due to their ability to be realized in hardware. Most of the wavelet stuff I've heard about has been CPU intensive calculation made possible by the likes of modern x86 and 64-bit architectures, and I expect RED RAY to have steep decode calculation requirements.

It doesn't seem like bitrate would be a problem though, which I thought would be the dominant issue with internet and streaming distribution.

 

[Radoslav Karapetkov]

No, what I meant was to just use the media [and its capacity], not the codec.

HD-DVD discs + RED RAY Codec.

 

[Dj Joofa]

Can you list or refer to some of these advantages? I was under the impression that the block-based encoding schemes won out previously due to their ability to be realized in hardware. Most of the wavelet stuff I've heard about has been CPU intensive calculation made possible by the likes of modern x86 and 64-bit architectures, and I expect RED RAY to have steep decode calculation requirements.

It doesn't seem like bitrate would be a problem though, which I thought would be the dominant issue with internet and streaming distribution.

Without going into the issues of fine/coarse grained parallelism offered by data partitioning and data locality for block-based vs. (full frame) wavelets; for purely streaming applications, the loss of data packets on the Internet, or even delayed receipt of a packet can cause severe problems because of time constraints maintained between the live streaming applications such as video conferencing. These issues are not faced with a dvd player or a projector, because data is never lost. However, for online streaming with image partitioning even if certain partitions are lost, portions of images can still be reconstructed. The H.264 standard went to great length to even suggest a Flexible Macroblock Ordering (FMO), where even if half of the data blocks making an image are lost, the error can be concealed! I am currently not aware of FMO being implemented in an online system due to its complexity, though. However, such options are there.

Additionally, internet streaming imposes a certain size on optimal MTU (~1500 bytes) for packets (Final Cut Pro / Quicktime users must have seen this number when adding a hints tracks to their movies). It is much easier to partition an image into blocks to fit into such MTU-sized packets as opposed to breaking the set of compression co-efficients for full frame image compression techniques.

H.264 (and to some extent H.263 and H.261; (H.262 is MPEG-2)), were designed with such issues in mind and even uses a Network Abstraction Layer (NAL) methodology for easier manipulation on networked environment.

 

[Steve White]

Ah. So would it be reasonable to say that a full frame encode scheme would be okay for internet downloads, but not so much internet streaming?

Moreover, can you see some kind of similar (in result, different in implementation) packet management for a wavelet image stream?

 

[Dj Joofa]

Ah. So would it be reasonable to say that a full frame encode scheme would be okay for internet downloads, but not so much internet streaming?

If temporal compression is also used for a full frame encoding scheme (which it will have to to reach those compression ratios), after the first frame, successive frames will have to offset the data content in the first frame (which block based do by differencing). What type of frame differencing may be used here is something that I have not thought about before and even if I did that would mean I would be doing the work for Red free of cost So can packets from the 2nd frame onwards be partitioned into optimal MTU sized units with the stipulation of good error concealment even if they are lost is something that will require deeper analysis of this issue than I can think in a few minutes here.

But it appears something has to be devised to get to the cleaner approach of block-based schemes for such issues.

Moreover, can you see some kind of similar (in result, different in implementation) packet management for a wavelet image stream?

Again, I shall have to think about it. Wavelets co-efficients are spread scale-space. How to use them in a temporal scheme in addition to their usual spatial domain mechanism will require some thinking at least on my part.

 

[Dj Joofa]

Spiff, you have had the honor of receiving the 500th message on this forum from me

 

[Steve White]

Woot! Now I'm at 50.

It's always encouraging for folks in R&D when the "future work" slide comes up.

A scholar.google.com search for "temporal compression wavelet" produces some 13,800 hits... so clearly some folks have been thinking about such things. I wonder if RED RAY will be the first iteration to percolate to the market.