Red and 4:4:4

[KStephens]

In order to try and clear up some confusion and help people understand why "4:4:4" notation (chroma subsampling notation) is inappropriate when talking about the Red, I've written this. Thanks to Graeme for his comments and suggestions. Hope it helps.

29.Oct.2008: I've updated it with a few changes based on people's responses.

Why “4:4:4” has nothing to do with the Red One camera

History

First, a bit of history. Originally, there was black & white television. Then came colour. To accommodate black & white televisions, colour was added to the original luma (brightness) signal through a new chroma (colour) signal, thus keeping colour separate and black & white televisions viable. In addition to making sure that black & white televisions were still useful, this was also an intelligent division of the visual data because less bandwidth was used for the chroma signal, which the human eye is much less sensitive to compared with luma.

Y’CbCr

In the digital world, the standard for separating luma and chroma is the Y'CbCr colour space – Y’ represents luma (gamma-corrected), Cb and Cr are used for blue and red channels. The green colour information is derived mathematically. As stated above, the human eye is less sensitive to colour information than brightness information. Naturally, digital video could also use this fact to save bandwidth. In digital video, the process of reducing chroma information translated into subsampling – that is, decreasing the resolution for the chroma channels.

Subsampling

Analogue television used much less bandwidth for the chroma signal than the luma – approximately one quarter. So the effective ratio of bandwidth would be 4:1, with the luma signal using four times the bandwidth of the chroma. The same effect is achieved in digital video through subsampling – reducing the resolution. Due to the original, analogue numbers, subsampling uses a “4” to describe a full-resolution (luma) channel. To achieve subsampling the chroma resolution is cut to one half or one quarter. Chroma subsampling notation describes this as 4:2:2 and 4:2:0 (or 4:1:1), respectively. The first number is the luma channel and the second and third numbers are chroma channels (Cb and Cr). Normally, this describes the horizontal sampling per scan line. However “4:2:0” is a special case that indicates the chroma channels are halved both horizontally and vertically. So, a subsampled 4:2:2 or 4:2:0 image will either have lost half or three quarters of the original chroma data.

RGB

Y’CbCr is a way of storing colour visual information – that is, a colour space. RGB is another colour space. RGB, of course, stores the information in a completely different way. Instead of separating luma and chroma, all the information is derived by mixing red, blue and green channels. As RGB is not designed to separate the chroma information, it is not suited for chroma subsampling.

Bayer Pattern

A CMOS image sensor most often uses a Bayer-pattern filter to produce colour information (like Red's Mysterium). The Bayer-pattern filter divides the light into red, green and blue (like RGB), so that each photosite captures one of the three colour channels. However, since a photosite only captures one channel and yet correlates to one pixel in the output, the RAW data saved is not RGB. It is Bayer-patterned RGB, which can be called "GRGB". The two G's in GRGB refer to the fact that there are twice as many green photosites as red or blue. Green is favoured because the human eye is more sensitive to green light. To get an RGB image from GRGB data, RGB data has to be re-constructed through a process called “demosaicing”. There are several approaches to this and getting the best result is both a science and an art.

Conclusion

Demosaicing and subsampling are performed in completely different colour spaces, GRGB and Y’CbCr, respectively. They are also two very different processes. Subsampling reduces bandwidth by reducing resolution. Information is lost in that process. Demosaicing, on the other hand, is a method of interpolation in which none of the existing data is compromised.

The Red One does not perform chroma subsampling, nor does it operate in Y'CbCr colour space. The RAW data captured by the camera and stored as Redcode RAW is Bayer-patterned RGB (GRGB) and this data is demosaiced into RGB data. It is therefore meaningless to use subsampling notation when discussing the quality of the Mysterium sensor, the quality of native Recode RAW footage or demosaiced Redcode footage. It would only be appropriate if the image data were later converted to Y’CbCr colour space, in which case, the user has complete control of the quality.

 

[Rolf Heiler]

Great explanation. This is the first time to me, to understand the difference between RGB and YCbCr. But what impact have different red codecs for the debayering quality?

RHeiler

 

[Radoslav Karapetkov]

Great post indeed.

 

[Birns and Sawyer]

Nice consice description.

 

[M Most]

The Red One does not perform chroma subsampling and it operates in RGB colour space. The RAW data captured by the camera and stored as Redcode RAW is Bayer-patterned RGB and this data is demosaiced into full RGB data. It is therefore meaningless to use subsampling notation when discussing the quality of the Mysterium sensor, the quality of native Recode RAW footage or demosaiced Redcode footage. It would only be appropriate if the image data is later converted to Y’CbCr colour space, in which case, the user has complete control of the quality.

While what you say is technically correct, the fact is that in a Bayer pattern sensor, there are half as many photosites devoted to blue and red as there are to green. Since most of the detail information captured by cameras and by human eyes is in the green spectrum, this basically means that the actual capture is, in effect, subsampled in red and blue. The RGB image that is the product of debayer and reconstruction is valid, but it is interpolated, not captured - which is why the debayer and reconstruction algorithms used are so critical. I'm not saying there's anything wrong with this, but it is simply a fact that the sensor itself does not capture all three components at the same sampling rate.

The basic premise of your post that 4:4:4 and 4:2:2 are basically video terms that describe YCbCr color space is true, but in many peoples minds, the numbers basically mean that all three color components are either sampled at the same rate or they're not. A simplistic view, and not necessarily technically correct, but a commonly held view nonetheless. And the use of a Bayer sensor does imply that two of the color components are not sampled at the same rate as the third.

 

[Thommes Ulfeng]

Short sweet and to the point

Short sweet and to the point

Thank you for "turning on the light". This was not clear to me until now.
Thommes.

 

[number6]

Very enlightening man. Thanks for posting this in such a reader-friendly way.

 

[KStephens]

While what you say is technically correct, the fact is that in a Bayer pattern sensor, there are half as many photosites devoted to blue and red as there are to green. Since most of the detail information captured by cameras and by human eyes is in the green spectrum, this basically means that the actual capture is, in effect, subsampled in red and blue. The RGB image that is the product of debayer and reconstruction is valid, but it is interpolated, not captured - which is why the debayer and reconstruction algorithms used are so critical. I'm not saying there's anything wrong with this, but it is simply a fact that the sensor itself does not capture all three components at the same sampling rate.

The basic premise of your post that 4:4:4 and 4:2:2 are basically video terms that describe YCbCr color space is true, but in many peoples minds, the numbers basically mean that all three color components are either sampled at the same rate or they're not. A simplistic view, and not necessarily technically correct, but a commonly held view nonetheless. And the use of a Bayer sensor does imply that two of the color components are not sampled at the same rate as the third.

The point of this is to try and help people understand the terminology and (to some small extent) the technology - not to accept a common error. Hopefully that can help prevent misuse of the terminology and subsequent misunderstanding.

Subsampling and demosaicing are two very different processes requiring very different approaches. It is inaccurate to assume that separating light rays into one of three RGB channels for capture in a photosite is effectively the same as reducing the chroma resolution of existing Y'CbCr data. That is a crucial point. Subsampling and the Bayer pattern both have advantages and disadvantages in their own fields. However, a direct comparison between the two is impractical and of little value.

Back to subsampling notation, it is simply the wrong approach when dealing with Bayer-patterned data.

First, chroma subsampling notation does not refer to three colour components, nor should it be used to do so. It refers to luma and chroma. Subsampling is reducing the amount of colour information in the image to save space. Thus, it begins where there is already more information. That is an important distinction: existing digital image information is actually sacrificed to save space. This does not occur in a Bayer-pattern CMOS. So, the Red One never has more information than is provided by the RAW Bayer data.

Secondly, the subsampling process requires the separation of chroma and luma (via Y'CbCr) because only colour information is subsampled. Bayer is, as stated, RGB. The Bayer pattern favours green, but every pixel is still captured as a colour channel (R,G,B). There is no separation of luma and chroma.

So, once again, using chroma subsampling notation does not make sense, regardless of the distribution of colours by the Bayer pattern. Both requisites for using subsample notation are not met: chroma and luma are not separated and data is never subsampled.

 

[KStephens]

Great explanation. This is the first time to me, to understand the difference between RGB and YCbCr. But what impact have different red codecs for the debayering quality?

RHeiler

Thanks!

Do you mean Redcode 28 and Recode 36? That's an interesting question. The compression probably has some impact on the demosaicing process, but I'm not sure what that impact might be. Redcode may be fine-tuned for Bayer data, which would make it hard to compare without being able to compress in it.

However, if you want to get an idea of the compression quality, you can try doing compression tests yourself with another wavelet CoDec (JPEG 2000 comes to mind). That will demonstrate how visually lossless such a CoDec can be. From my tests, the results are rather impressive.

 

[Rocco Schult]

Rgb 4:4:4

Rgb 4:4:4

...but in many peoples minds, the numbers basically mean that all three color components are either sampled at the same rate or they're not.

The point of this is to try and help people understand the terminology and (to some small extent) the technology - not to accept a common error. ...
Back to subsampling notation, it is simply the wrong approach when dealing with Bayer-patterned data...

I have never been in a competent technical discussion about 422/444 regarding a RED shoot...yet, only here. It is usually customer talk, amateurish questions, as color space and subsampling is more for us tech geeks really. And they ask the questions different or know it anyway.
People asking about 444 want to know more if it is like "full quality" - that is what they associate it with.
They just know 444 is the full package, the best, which is quite right, not ?
Is RED like 444 is the question I faced a couple of times. They don't know the difference to 422 and how much the keying is really better in 444 or not.

I always try to deal around with it and say, it is different, more like DSLR, other chip-pattern which doesn't apply to this kind of terminology...
But it is easy: Compared to a true 4K RGB, no, RAW is not the full package. Not like a 444 image in 4K would be. Not that sharp, not the same color resolution. Yes, we might have 4K pixels, but they are RAW, there's not 4K res in. And in this regard, the term might be wrong, but the understanding is right and a question and comparison does make absolutely sense - in its own way.

...First, chroma subsampling notation does not refer to three colour components, nor should it be used to do so. It refers to luma and chroma.
Bayer is, as stated, RGB. The Bayer pattern favours green, but every pixel is still captured as a colour channel (R,G,B). There is no separation of luma and chroma.

So, once again, using chroma subsampling notation does not make sense, regardless of the distribution of colours by the Bayer pattern. Both requisites for using subsample notation are not met: chroma and luma are not separated and data is never subsampled. ...

While 422 usually stands for YUV color space without further prefix, the terminology itself works also in RGB colorspace where 444 stands for full raster and no further reduction via color subsampling.
Your post is very elaborate and well researched and expressed - the statement color subsampling term would only apply to YUV colorspace is not absolutely correct though, at least not for RGB 4:4:4.

 

[OpalFilm]

Referring to 4:4:4 can also tell something about ease of keying.

Do you know how easy is to key RED footage at 4k resolotion?

 

[KStephens]

I have never been in a competent technical discussion about 422/444 regarding a RED shoot...yet, only here. It is usually customer talk, amateurish questions, as color space and subsampling is more for us tech geeks really. And they ask the questions different or know it anyway.
People asking about 444 want to know more if it is like "full quality" - that is what they associate it with.
They just know 444 is the full package, the best, which is quite right, not ?
Is RED like 444 is the question I faced a couple of times. They don't know the difference to 422 and how much the keying is really better in 444 or not.

That is exactly the problem: that "4:4:4" becomes a general, blanket term for quality that really doesn't mean anything. I've talked with cinematographers and directors about this thing and you're right that the basic question is that they want "full" quality. The point here is to understand what compromise has been made in the Red camera. Comparing Bayer and 4:2:2 (Y'CbCr) is comparing apples and oranges, but it's true that people need to understand a compromise has been made.

I always try to deal around with it and say, it is different, more like DSLR, other chip-pattern which doesn't apply to this kind of terminology...

Yes! It's exactly like the vast majority of DSLRs.

But it is easy: Compared to a true 4K RGB, no, RAW is not the full package. Not like a 444 image in 4K would be. Not that sharp, not the same color resolution. Yes, we might have 4K pixels, but they are RAW, there's not 4K res in. And in this regard, the term might be wrong, but the understanding is right and a question and comparison does make absolutely sense - in its own way.

What you are getting is a subset of RGB data (but not from an original set of RGB data).

While 422 usually stands for YUV color space without further prefix, the terminology itself works also in RGB colorspace where 444 stands for full raster and no further reduction via color subsampling.
Your post is very elaborate and well researched and expressed - the statement color subsampling term would only apply to YUV colorspace is not absolutely correct though, at least not for RGB 4:4:4.

Thanks. As I wrote before there is no subsampling in a Bayer-patterned device. And what is the purpose of "RGB 4:4:4"? The matter of demosaicing is quite complex and it's not the same is reducing the colour resolution (as with 4:2:2/4:2:0 in Y'CbCr). So, unless we could have "RGB 4:2:2" (which we cannot) this misnomer is quite misguided.

Fair enough, though. People may need something easier than "Bayer-patterned RGB" to understand that we are talking about a source image that needs to be interpolated into RGB.

Perhaps we could use the already-established "GRGB" term for the Bayer-patterned data? Would "Bayer RGB" help? Again, the point is to try and make people aware (in a simplified and intelligible way) of what this technology means; and with that, what it means for them (quality-wise, etc.).

Cheers

 

[Justin Kirchhoff]

Great post.

 

[Mathew Mackereth]

thanks - very concise post - am teaching Shake at the moment and managed to confuse the heck out of my students by "explaining" Y'CbCr notation and (one of the reasons) why XDCAM and HDV keys so poorly....

this will hopefully restore some sanity....

macka

 

[Dj Joofa]

Lightman, you are mixing up a number of issues here. Since the terms 4:4:4/4:2:2/etc., have lost their meaning (to be legally correct they applied to analogue subsampling and not to digital subsampling in YCbCr), therefore, it is up to the context to figure out what an author is implying.

I think MMost summed it up very well and you might reread his message to get exactly what he is saying.

 

[Raphael Varandas]

Good Job guys...very good thread

 

[Stacey Spears]

Since you are using the prime notation for Y'CbCr, you should probably use it for RGB as well, as in R'G'B' when you are talking about gamma corrected R'G'B' and RGB when you are talking about linear light.

 

[KStephens]

Lightman, you are mixing up a number of issues here. Since the terms 4:4:4/4:2:2/etc., have lost their meaning (to be legally correct they applied to analogue subsampling and not to digital subsampling in YCbCr), therefore, it is up to the context to figure out what an author is implying.

I think MMost summed it up very well and you might reread his message to get exactly what he is saying.

I've re-read it and I'm afraid I don't agree. I appreciate his points: that GRGB needs interpolation and that people use subsampling notation as a shortcut to describe "quality". Otherwise, I've addressed the problems in his post.

If you want to elaborate on what you think I'm mixing up, let me know.

Anyway, thanks to everyone for the positive feedback!
Cheers.

Since you are using the prime notation for Y'CbCr, you should probably use it for RGB as well, as in R'G'B' when you are talking about gamma corrected R'G'B' and RGB when you are talking about linear light.

D'oh! ... it never ends!

thanks - very concise post - am teaching Shake at the moment and managed to confuse the heck out of my students by "explaining" Y'CbCr notation and (one of the reasons) why XDCAM and HDV keys so poorly....

this will hopefully restore some sanity....

macka

Ah, excellent! Should give them something to chew on.

 

[M Most]

I've re-read it and I'm afraid I don't agree. I appreciate his points: that GRGB needs interpolation and that people use subsampling notation as a shortcut to describe "quality". Otherwise, I've addressed the problems in his post.

I don't really see any problems in what I said. I wasn't disputing your explanation in terms of technical correctness, I was just pointing out that you can't educate the world, and you can't stop terms from being adapted for purposes other than their actual, literal meaning as understood by those who have enough experience to understand them. Believe me, I've tried. Many times. "Telecine," "film," "linear," "log," "DNR" (that one is misused by Jim Jannard all the time) - all of these terms have been usurped in the last year or so to mean things they don't actually mean. And I've tried to point this out when it happens, to no avail. You can't fight City Hall, and you can't prevent misuse of terminology, especially in the Internet age when lack of experience and education on a subject is seemingly no longer a hindrance to presenting oneself as an expert.

 

[KStephens]

I don't really see any problems in what I said. I wasn't disputing your explanation in terms of technical correctness, I was just pointing out that you can't educate the world, and you can't stop terms from being adapted for purposes other than their actual, literal meaning as understood by those who have enough experience to understand them. Believe me, I've tried. Many times. "Telecine," "film," "linear," "log," "DNR" (that one is misused by Jim Jannard all the time) - all of these terms have been usurped in the last year or so to mean things they don't actually mean. And I've tried to point this out when it happens, to no avail. You can't fight City Hall, and you can't prevent misuse of terminology, especially in the Internet age when lack of experience and education on a subject is seemingly no longer a hindrance to presenting oneself as an expert.

Fair enough. "Problems" was also a poor choice of words. I believe I explained what I thought needed clarification. I've just wanted to try and separate two very different processes and terminologies.

You're right and there are loads of examples of terms and names that have eroded. I don't expect a post on this board (or anywhere) is going to halt every occurrence of someone asking "is it 4:4:4?" in a completely inappropriate context. However, I'm quite satisfied if only a few people come to a better understanding. I really appreciate the positive responses.

There are people who are interested and do care. Certainly, with the Red One, it is useful for its advocates and users to be able to understand and explain what the technology is and how it can be compared (or cannot be compared) with competing technology. If any place is appropriate to try and clarify this, I'd imagine Reduser is that place.