Is R3D 4:4:4 or 4:2:2?

[Roberto Lequeux]

Love this thread. This stuff is pretty over my head but I can hang enough to be curious and want to learn more. Now I understand what you were talking about the other day Bruce, mostly anyway.

If with a full resolution but slightly more lossy compression applied only to chroma you churn out files exactly the same size as those with far lower chroma resolution -- you have a far better image at no cost!!

Why hasn't this happened yet? (aside from our inability to clone Graeme) Would it be heavier on processing than sub-sampling? What's the deal?

 

[conrad gaunt]

My main issue here is correct use of terminology, because there's been some serious abuses of this terminology in the past from people who should know better.

Graeme

I know you're a stickler.. and I'm guilty

I've never computed accurate numbers myself, but it wouldn't surprise me if some of the RGB->YUV->RGB transformations through out as much or more than 50% (i've heard higher numbers) of the information originally present, so it would be wrong to compare 4:2:2 yuv signals to a raw RGB sensor output .. such as redcode

Until someone invents some kind of multi-layered full resolution RGB sensor, I predict we're going to be using the 4:2:2 RGB "ratio" for a while.. and with ultra-res and downsampling, its hard to imagine a better system right now ..haven't seen one

People use terminology "wrongly" all the time. My pet hate was the guys from fxguide RC, early on, who for a long while (but not for a long time now) would refer to "doing a half/quarter debayer".. more accurate to say, "doing a partial inverse wavelet transformation" (which is of course the reason we can read back only part of a R3D and still get an image out, albeit at some lower resolution, and the debayer resolution incidental).. but I got their points

 

[conrad gaunt]

Love this thread. This stuff is pretty over my head but I can hang enough to be curious and want to learn more. Now I understand what you were talking about the other day Bruce, mostly anyway.

If with a full resolution but slightly more lossy compression applied only to chroma you churn out files exactly the same size as those with far lower chroma resolution -- you have a far better image at no cost!!

Why hasn't this happened yet? (aside from our inability to clone Graeme) Would it be heavier on processing than sub-sampling? What's the deal?

My answer would be long. Here's my short version.

Cine used film, job done. TV only needed to meet the broadcast signal standard..SD.. which tended to use various myriad forms of YUV over time, now we have more widespread use of codecs and higher resolution digital viewing options, and its happened hand in hand with internet speeds and computers, quite quickly compared to the evolution of say motor cars which are surprisingly similar to the first cars produced, as are production techniques in many ways.. and so we find our selves asking "why hasn't this happened yet?" .. right now, what is the resolution standard 720p, 1080p , 4k -> im still on SD for viewing or monitor? stereo? .. and the distribution format? .. I don't know, and everyone else is guessing, including the manufacturers who are at the mercy of "what people are actually going to buy".. so my answer is, all sorts is happening, at different speeds, sometimes in secret, sometime not, probably right now, maybe! Wish I still had my sisters of mercy live at the royal albert hall laser disc..

..difference is, Ford knew he had to make cars his employees could afford to buy, and aspire to buy .. camera manufacturers seem to have a different idea.. which is why i dropped out of "camera owning world" for ten years, and wholeheartedly, and totally have skipped HD.. wish I could skip owning an mx sensor frankly, but you eat what you're given or starve

 

[Les Dittert]

Conrad, what is it that you are trying to do?
For me it's getting the max image quality from the gathered data stored in the r3d.
My pet peeve right now is how the debayer rendering from RCX is looking.
It's a linear debayer as far as I can tell.
We have done our own r3d decode and saved the result in a Cineform RAW mov. When I set the debayer to linear in Firstlight, Cineforms control program, it looks exactly like the RCX debayer. Soft.
I prefer the more detailed debayer preformed by the rocket card. When I set the debayer to other versions, I can get to the detail level seen in rocket card renderings. The slight difference I see may be due to the re-compression of the RGGB once again. Or maybe we have a math error still, in our code.
For the less than tech savy, no, we did not just transcode RCX output as a Cineform RGB. We are taking the bayer data and compressing that, preserving the RAW nature of the image.

 

[Uli Plank]

I'm under the impression that you look at things from two very different perspectives. Conrad takes the information theory POV, where it's very clear that a Bayer pattern sensor can't resolve the same amount of information for all colors. Period. (BTW, I knew that an OLPF is filtering both luma and chroma).

Graeme is taking the POV of the mastermind behind RED's debayering/compression methods, who is achieving the best cinematic images to be looked at in the end. What I'm getting from his explanations is this: as long as you can use the RAW information from a Bayer pattern sensor, you can optimize the process in a way that can't be achieved after any YUV subsampling because you have a whole color space. Take a pure blue vs. a bluish-green for example, the pure blue can't be reconstructed to the same scene detail the mixed color could reveal, since there is more green in it. This would be true for any color other than blue and red. But even then you'd have dyes which are not perfect in filtering plus light scatter from neighboring photo sites and so on. If you take this into consideration, it's far more complex but also more capable in the right hands.
YUV subsampling is a pretty harsh process in comparison.

 

[conrad gaunt]

I'm under the impression that you look at things from two very different perspectives. Conrad takes the information theory POV, where it's very clear that a Bayer pattern sensor can't resolve the same amount of information for all colors. Period. (BTW, I knew that an OLPF is filtering both luma and chroma).

Graeme is taking the POV of the mastermind behind RED's debayering/compression methods, who is achieving the best cinematic images to be looked at in the end. What I'm getting from his explanations is this: as long as you can use the RAW information from a Bayer pattern sensor, you can optimize the process in a way that can't be achieved after any YUV subsampling because you have a whole color space. Take a pure blue vs. a bluish-green for example, the pure blue can't be reconstructed to the same scene detail the mixed color could reveal, since there is more green in it. This would be true for any color other than blue and red. But even then you'd have dyes which are not perfect in filtering plus light scatter from neighboring photo sites and so on. If you take this into consideration, it's far more complex but also more capable in the right hands.
YUV subsampling is a pretty harsh process in comparison.

I think my thinking probably covers a bit more than information theory, it involves practical programming of wavelets, debayering algorithms and many other techniques including fractal compression (my own kind), 3d visualisation/ raytracing, image resampling (my way), and recently audio dsp..along with analysis of r3d files and about 6000 other things. I started programming 30 years ago, 31 soon. How you perceive my POV, is not very accurate..but not offensive

My primary interests are non-realtime visualisation technology, and image pipelines..im a patient man

I bet I could explain a lot of what makes a red frame look so nice, and simulate it ..but its amazing the compression rates you can get if you just stick 16bit raw data thats had wavelet transformations applied, into a png or j2k container.. but when you start doing non-realtime stuff, like using wavelet coefficients to guide super-resolution resampling..debayering and wavelets join forces.. i spent a while on that a couple of years ago

my focus at the moment is lossless D.I compression, something I've explored at several different times over twenty years, but its more relevant than ever for me right now

Raw is great for acquisition, but you need something more 4:4:4 (in nature) to archive masters, and sizes then get funky of course

My problem is, im not a commercial progammer, im an explorer

anyway..is Rob Lohman not the main codec brain, lets not forget the other elves!

 

[Eric Z]

Graeme,
Any chance the new Red Dragon sensor will be a Foveon sensor, like in the Sigma SD1 (15MP x 3 layers)?
That would surely help keep 5K footage at 5K, without any need to debayer it to lower resolutions.
I realize the Foveon sensor introduces its own problems, such as less and less light for the bottom 2 layers, but all in all - its sounds like a very good solution.

Any opinion on that?

 

[conrad gaunt]

Conrad, what is it that you are trying to do?
For me it's getting the max image quality from the gathered data stored in the r3d.
My pet peeve right now is how the debayer rendering from RCX is looking.
It's a linear debayer as far as I can tell.
We have done our own r3d decode and saved the result in a Cineform RAW mov. When I set the debayer to linear in Firstlight, Cineforms control program, it looks exactly like the RCX debayer. Soft.
I prefer the more detailed debayer preformed by the rocket card. When I set the debayer to other versions, I can get to the detail level seen in rocket card renderings. The slight difference I see may be due to the re-compression of the RGGB once again. Or maybe we have a math error still, in our code.
For the less than tech savy, no, we did not just transcode RCX output as a Cineform RGB. We are taking the bayer data and compressing that, preserving the RAW nature of the image.

same thing as you by the look of it

there are some very good linear interpolation based debayering algorithms actually, their secret is they're adaptive. Microsoft wrote a paper about one of theirs which did well against the well known "Variable Number Of Gradients" algorithm, which is great for natural scenery..

I quite like reds debayer, and cineforms.. but I tend to archive downsampled masters, so that vastly reduces aliasing or artifacts and makes all good anyway.. I say that, I haven't for three years because my money is not coming from post production at the moment

The secret of Reds image IMO is the balance they've got between softness and sharpness. I'd rather have high resolution that is a little soft, than a high (or low) resolution image with aliasing, and thats perhaps a lot to do with OLPF, and partly to do with wavelet transformations throwing away small details similar to noise reduction systems (some of which are wavelet based of course), and partly because debayering not only re-creates missing pixels, but often blends adjacent ones.. which is why an unsharp mask filter may be your friend..if thats your thing. A lot of video cameras with baked in looks used to aim for high acutance, because the human visual system treats it like real-resolution, but in reality its a sock pushed down the front of the cameras pants (thats a metaphor, I do know real cameras are always in the nude), but no real meat

I like using my own downsampling software, which is both slow and bullet proof in equal measure, and can do 28k -> SD conversion, of even awkward frames.. its linear interpolation, with gaussian weighting, and comes in 32bit, 64bit and 80bit versions.. I will produce a commercial grade version when I eventually buy a new compiler (don't ask..) ,

 

[conrad gaunt]

Graeme,
Any chance the new Red Dragon sensor will be a Foveon sensor, like in the Sigma SD1 (15MP x 3 layers)?
That would surely help keep 5K footage at 5K, without any need to debayer it to lower resolutions.
I realize the Foveon sensor introduces its own problems, such as less and less light for the bottom 2 layers, but all in all - its sounds like a very good solution.

Any opinion on that?

You don't debayer to a lower resolution, you always debayer at full resolution, then downsample to a lower one..

..or, you can perform a partial inverse wavelet transformation of the r3d, to whatever resolution level you decide (in this example, a lower resolution), followed by a debayer.. which is not as good, but faster than above method

btw, foveon sensors are a bit crap at the moment,.. but i might be tempted by a light field sensor in the future , from a documentary POV

 

[Graeme Nattress]

Long Answer: Foveon has some real colorimetry issues - what you save on not having to demosaic, you spend on having to build in extensive noise reduction as part of the basic decode process (see the Foveon white papers). This is because silicon is a pretty poor colour filter. It also makes it very poor at higher ISO and leads to other imaging issues that various forums have pointed out, especially in the new chip with the smaller pixels.

Most of the Foveon fans like excessive aliasing which adds fake sharpness to their images. We think such levels of aliasing are unacceptable for cinematography use. When used with a proper optical low pass filter the "benefits" of Foveon have diminished so far as to make it pointless. Then you're stuck with a sensor that's not fast enough to do motion at 120fps (I dunno what it's fps is, but it's no-where-near where our sensors are).

Short answer: no.

Graeme

 

[Uli Plank]

What about this rumor? Is it similar to Foveon?

http://www.sonyalpharumors.com/sr4-...-in-2013-100-megapixel-sensors-for-2013-2014/

 

[Graeme Nattress]

No idea Uli. But the picture attached is nothing like Foveon which uses photon penetration through silicon depth as a poor colour discriminator.

Graeme

 

[Les Dittert]

Unsharp mask is only 'my friend' if it is used to recover slight softness. Kodak called that 'aperture correction' back in the Cineon scanner days.
Sharpening the linear looking RCX output does not recover all the detail that the Rocket card has, unfortunately.
If I apply superresolution to the RCX output, it resolves a lot less than superresulution applied to the Rocket output.
Superres 'loves' aliased inputs, because there is a lot of detail to be extracted from that very sampled sample, that is bayered data.

Conrad, you are totally right: The future of this is combining superres into the debayer process. Old debayer research was driven by still image needs.
Let's shift gears and move beyond that! I am not too concerned with processing speed. Time will solve that.

The wonderful thing is that the detail can be 'mined' out of the old r3d's when the above is productized.
Red has a lot of irons in the fire, I understand fully this is lower priority.

same thing as you by the look of it

there are some very good linear interpolation based debayering algorithms actually, their secret is they're adaptive. Microsoft wrote a paper about one of theirs which did well against the well known "Variable Number Of Gradients" algorithm, which is great for natural scenery..

I quite like reds debayer, and cineforms.. but I tend to archive downsampled masters, so that vastly reduces aliasing or artifacts and makes all good anyway.. I say that, I haven't for three years because my money is not coming from post production at the moment

The secret of Reds image IMO is the balance they've got between softness and sharpness. I'd rather have high resolution that is a little soft, than a high (or low) resolution image with aliasing, and thats perhaps a lot to do with OLPF, and partly to do with wavelet transformations throwing away small details similar to noise reduction systems (some of which are wavelet based of course), and partly because debayering not only re-creates missing pixels, but often blends adjacent ones.. which is why an unsharp mask filter may be your friend..if thats your thing. A lot of video cameras with baked in looks used to aim for high acutance, because the human visual system treats it like real-resolution, but in reality its a sock pushed down the front of the cameras pants (thats a metaphor, I do know real cameras are always in the nude), but no real meat

I like using my own downsampling software, which is both slow and bullet proof in equal measure, and can do 28k -> SD conversion, of even awkward frames.. its linear interpolation, with gaussian weighting, and comes in 32bit, 64bit and 80bit versions.. I will produce a commercial grade version when I eventually buy a new compiler (don't ask..) ,

 

[Wayne Morellini]

I think my thinking probably covers a bit more than information theory, it involves practical programming of wavelets, debayering algorithms and many other techniques including fractal compression (my own kind), 3d visualisation/ raytracing, image resampling (my way), and recently audio dsp..along with analysis of r3d files and about 6000 other things. I started programming 30 years ago, 31 soon. How you perceive my POV, is not very accurate..but not offensive

My primary interests are non-realtime visualisation technology, and image pipelines..im a patient man

I bet I could explain a lot of what makes a red frame look so nice, and simulate it ..but its amazing the compression rates you can get if you just stick 16bit raw data thats had wavelet transformations applied, into a png or j2k container.. but when you start doing non-realtime stuff, like using wavelet coefficients to guide super-resolution resampling..debayering and wavelets join forces.. i spent a while on that a couple of years ago

my focus at the moment is lossless D.I compression, something I've explored at several different times over twenty years, but its more relevant than ever for me right now

Raw is great for acquisition, but you need something more 4:4:4 (in nature) to archive masters, and sizes then get funky of course

My problem is, im not a commercial progammer, im an explorer

anyway..is Rob Lohman not the main codec brain, lets not forget the other elves!

At last another explorer.

That is a laugh. I am sure Rob was somebody I much liked, but there is so much to it not even in the books which I came up with (invented). I have received personal professional opinion about Red code compared to the original Cineform Bayer wavelet compression years ago, but I don't know what has been done to it since then. Genius is like the hen's teeth. If you take everything you can learn and learn from what everybody else has done, you may still only have state of the art at most, to get beyond that is the tricky part. Sorry to be such a pessimist, but a number of behind the scenes things have happened and I have seen so many things less then they should be. It is good Rob is a worthy sort of person from memory. However, until somebody can meet my 1000:1 true loss less compression goal, they have not caught up. I know I have not mentioned this before, because people spin out when I say a conservative 10:1, let alone 100:1. Most every abstract forecast I say is understatement, often servere understatement. I am pushing for up to 10,000:1 visually loss less fast routines. The ideas I have pursued to end game, they are so compkex and so many years in advance. People are not good at seeing and manipulating the domain space of problems, many designs are inadequate, if not hopeless, written by smart but never the less half clueless people, that I treat nicely, thank and even praise for what they have actually achieved. That is life unfortunately. I wish that I had not been too sick to work on my own, that these dead spots had not turned up in my head, I could have had my own business, at least my own research department. I can't even remember propperly who Rob was now and it is hard to grasp onto my past ideas and how to best fit them together. I should at least have a consulting job advising research and development departments how to improve things, but I will have trouble advising on how to do things the best way now, but still can swing a mean solution.

One thing I can advise for anybody here engineering, grasp onto the process and mechanism domain, that is where you can make vast improvements rearranging things, because we live in a physical process and mechanism based universe, it is that simple. In programming it is not which object or routine you use, it is which you can make better as well , in process and mechanism. It is good to have people around that can have a clue what to do. Maybe this redray will turn out to be really something, so my faith in humanity is returned.

 

[Wayne Morellini]

No idea Uli. But the picture attached is nothing like Foveon which uses photon penetration through silicon depth as a poor colour discriminator.

Graeme

The crosstalk problem again. Years ago I came up with a solution to this I was going to implement, but some years latter a camera company announced a design that used it (actually this was a sensor pixel technique I was looking at). The issue is, did I talk about it, and did they then patent it. Some of these things are surprisingly simple you think it must have been done by now, but often it seems to be a late discovery. Simple you have pixels, and these can be virtually 100% fill factor easily because of the nature. You use dichroic filter prisms/mirrors to seperate the three, or more, colors out reflecting, or projecting them, onto their individual fill areas, that can be smaller and surrounded by the sensor pad circuites, allowing 100% fill. It is such a simple mechanism that it comes very quickly to me, but disturbingly difficult for others. I came up with a number of dichroic mechanisms. However, I think I came up with a method to clean up that foveon depth crosstalk situation, but the reality is you are still are left with red with increased noise. /However, my sensor ideas have advanced greatly since then, just wish I had funding. I have one that can test for any frequency one at a time, not the infinite frequency range one I mentioned elsewhere, but thinking about it I could apply something like that, but there are penalties. I'm excited about that 8000% QE efficiency one I glanced elsewhere. I can sleep at night, 8000 should be good enough.

 

[J.D. Frey]

What about this rumor? Is it similar to Foveon?

http://www.sonyalpharumors.com/sr4-...-in-2013-100-megapixel-sensors-for-2013-2014/

I don't feel that sensor size/density is an issue for most cameras/users these days. I would take a better processor over sensor size, unless I was building a telescope or some other instrument that needed that kind of resolution. That article makes it sound like everyone is waiting on pins and needles for a 100MP sensor.

 

[Adrian Jebef]

There's some really juicy stuff here. Nicely done boys. But let's be clear and get back to the point. Is the R3D a true 4:4:4 recording?

What we are talking about here is full color sampling at the pixel-by-pixel level. We are not talking about software conversions and color space transforms. The question becomes: "Is each pixel in the intended, recorded resolution (in this case 4K) derived from individual, finite RGB values?" With regards to the Red1, Epic, and Scarlet the answer is no. These are not true 4:4:4 cameras. And, to be honest, I have my doubts that they are even 4:2:2. RED's cameras sacrifice color fidelity in favor of higher resolution. In order for the Epic to record a true 4:4:4 R3D file there would need to be twice the number of pixels on the sensor. Whether the file is recorded as RAW sensor data or 4K video is irrelevant. If full RGB color is not discreetly sampled at the sensor for the given output resolution then it is not 4:4:4. As my father always said, "There is no such thing as free."

Now I'm sure that the Epic could be capable of delivering a very nice 1080p 4:4:4 output but because the system is designed to only output 4K+ there is no ability to recover this lost color information within the R3D file.

 

[Graeme Nattress]

Adrian, you miss an important point.... In any sane camera design the input image is bandwidth limited so that it won't produce excessive aliasing via optical low pass filtering. That means you shouldn't be seeing resolution at the level of the sensor implied by your definition of 4:4:4. There's one type of stills camera that acts like this: Sigma Foveon, and it shows quite strong luma aliasing, so although you do get the colour resolution you desire, you also get corrupting aliases along with it and sampling theory says there's only one real way around this - aim for a much higher resolution and employ proper filtering.

In single chip (non Foveon) cameras, the one that fits the bill is the Sony F35, and guess what? It shows strong luma aliasing vertically and chroma moire horizontally. Again, by your definition it's 4:4:4, but it's not a 4:4:4 that I want due to the corrupting influence of the aliases...

That said, as I say above, the terminology of 4:2:2 etc. implies you're starting off with an RGB image, which you transform to Y'CbCr and decimate the chroma. Applying such terminology elsewhere in the image making process is wrong.

Graeme

 

[Wayne Morellini]

There's some really juicy stuff here. Nicely done boys. But let's be clear and get back to the point. Is the R3D a true 4:4:4 recording?

What we are talking about here is full color sampling at the pixel-by-pixel level. We are not talking about software conversions and color space transforms. The question becomes: "Is each pixel in the intended, recorded resolution (in this case 4K) derived from individual, finite RGB values?" With regards to the Red1, Epic, and Scarlet the answer is no. These are not true 4:4:4 cameras. And, to be honest, I have my doubts that they are even 4:2:2. RED's cameras sacrifice color fidelity in favor of higher resolution. In order for the Epic to record a true 4:4:4 R3D file there would need to be twice the number of pixels on the sensor. Whether the file is recorded as RAW sensor data or 4K video is irrelevant. If full RGB color is not discreetly sampled at the sensor for the given output resolution then it is not 4:4:4. As my father always said, "There is no such thing as free."

Now I'm sure that the Epic could be capable of delivering a very nice 1080p 4:4:4 output but because the system is designed to only output 4K+ there is no ability to recover this lost color information within the R3D file.

I know this probably has been largerly covered before, but it is a bit inbetween. It is about what you want, at the extreme resolution end you either have compromise to get rid of the jaggies, or you have real square pixels, why 8k is important to reduce the appearence of square pixels (notice how I coached the statement guys). At this extreme end, you may want to use software to get rid of jagged edges. Now there are other reasons for jaggies I am leaving out here, but any sensor that has less than 100% fill factor I regard as dated.

Bayer has been suggested as good for 4:2:2, but in reality it is just a different beast that does not even match 4:2:0, as not every pixel is green, but it canbe made to do a good job against 4:2:2, and even 4:4:4 (just an opinion) given suitable conditions of course. But in reality it is close to 4:4:4 (real color gaps sub pixel) when resolution is dropped in half or so from debayered etc. So, some might use this to claim that real color resolution is half etc etc, but let's not go there.

 

[Adrian Jebef]

Graeme:

I am not referring to 4:4:4 or 4:2:2 Y'CbCr and chroma subsampling for bandwidth's sake (compression in order to transmit data after it has already been recorded). I am referring to full resolution color sampling which is usually designated 4:4:4 in the 10-bit RGB color video model. The Epic uses a CMOS Bayer sensor to capture its images. Wikipedia offers a simple explanation: "In a Bayer filter arrangement, green is given twice as many detectors as red and blue (ratio 1:2:1) in order to achieve higher luminance resolution than chrominance resolution." Therefore, a proper 2:1 Bayer pattern will produce a full (4:4:4) luminance resolution image with subsampled color via a full-sampled Green channel, half-sampled Blue channel, and half-sampled Red channel (4:2:2).

The Sony F35 is an excellent example of a camera with the ability to provide fully sampled color images for its defined output (1920x1080 HD). The F35 uses a single CCD imager made up of 1920x1080 triple RGB pixels. By definition the F35 uses a discreet Red, Green, and Blue pixel to sample full chroma (4:4:4) before combining them into a single luma/chroma pixel value. Technically the F35 has three times the resolution of HD in its luminance sample rate. Is it a 3K camera? Sony didn't think so. The Arri Alexa uses a Bayer pattern similar to the Epic and though it comes close in luminance resolution to the Epic it was only marketed as an HD or 2K camera. The Alexa, like the new Sony F65, purposefully over-samples its intended luminance resolution in order to properly sample its chromatic resolution. So the Alexa samples 3.2K luma and HD/2K chroma to derive a proper 4:4:4 HD/2K output. The F65 samples 8K luma and 4K chroma to derive a proper 4:4:4 4K output. Can the Alexa record RAW sensor data at 3.2K? Yes. It it a proper 4:4:4 full-color 3.2K recording? No. Can the F65 record 8K RAW? Yes. Is it a proper 4:4:4 8K recording? No. The same applies to the Epic. And because the Epic is only intended to provide a 4K pipeline via its R3D files it simply does not have enough pixels on the sensor to sample full color 4:4:4 at 4K.

Saying that 4:4:4 color sampling does not apply to RED's R3D files because they are RAW may be technically accurate at that stage in the workflow process but it is mostly an irrelevant definition when discussing the finer details of how a camera works to its intended end user: the artist, not the technician.