Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

General Demosaicing Questions

Graeme Nattress

Graeme Nattress

RED Problem Solver
Joined
Dec 28, 2006
Messages
6,496
Reaction score
16
Points
38
Website
www.nattress.com
I'm happy to talk in general about demosaicing. It's not the mysterious process that some make it out to be, and although you don't really need to know how it works to get the best out of shooting a RED camera, it can be interesting.

Graeme
 
Graeme you are a true gent! I’ll try not to burn you out and I hope plenty of others will not be too shy to ask any sort of questions in this area.

So here’s a very basic one.

In the old days I used to shoot a lot of large plates, 4x5, 8x10, 13cmx18cm, 5x7cm etc. etc. The nice thing about 20 years ago is that Kodak introduced some amazing films that virtually eliminated grain artifacts as well as true reversal films that have no masking layer. When these images were scanned on spatially and radiometerically accurate scanners at 7.5 micron the results were absolutely stunning.

About 20 years ago I got to do some serious documentation with the first Leaf digital camera backs, they were 2k by 2k, (peltier cooled) used a color filter wheel, the pixels sizes were humungous in-terms of dynamic range and light gathering capacity. These were the best 4 million pixels on the planet, (Hassleblad glasss helped). Providing you could register the R G and B channels correctly!

In these situations there is this notional 1:1 type of resolution, but then came CCD and CMOS chips that use the debayer pattern?

So people now have this idea that a device that is 4k using a de-bayer mosaic actually only resolves to a resolution that is approximately 20% lower? (or lower still?)
So the question is. ..

Is there really about 20% “flab” in our images, and for most practical purposes is it possible to down-sample to get rid of this “flab”? Do we get leaner much more efficient images in terms of resolution vs the amount of data required to effect such resolution?

Lets say you have a 4k [RGBA] image, you down sample it to lets say 3.2 K [RGBA], (or whatever it’s combined MTF really is), and then you up-scale it back to 4k and then subtract it from the original. What is left over and what is the difference other than a bit of noise?

(sorry for the long pre-amble).

Ta.

Eric
 
Gunleik Groven said:
Hi and thanks.

For the not so learned, could you suggest some background articles on the web?

Is this a reasonable (but very basic) start?
http://en.wikipedia.org/wiki/Demosaicing
Click to expand...


Gunleik

Looks like a nice and concise link, (well illustrated to me anyway)
I like the quest or goal oriented approach

Quote from link.

“Goal

The aim of a demosaicing algorithm is to reconstruct a full color image (i.e. a full set of color triples) from the spatially undersampled color channels output from the CFA. The algorithm should have the following traits:

• Avoidance of the introduction of false color artifacts, such as chromatic aliases, zippering (abrupt unnatural changes of intensity over a number of neighboring pixels) and purple fringing

• Maximum preservation of the image resolution

• Low computational complexity for fast processing or efficient in-camera hardware implementation

• Amenability to analysis for accurate noise reduction “

I liked particularly “Ameanbility to analysis for accurate noise reduction”

Nice turn of phrase.

I think this is perhaps an aspect that Graeme has a wealth of experience of …

What does that really mean in practical terms, especially for the "bloke" who has to write algorithms for this?

Cheers,

Eric

(I’ll sit out for a bit so I don’t spoil what could be a really good thread…)
 
Graeme, Sony presumes that they will be able to demosaic its new CMOS 17MP sensor with 67%, I presume artifacts free, efficiency.

RED has something like 78% factor so 5K is resolving to 4K.

Does Bayer patter lends itself better for color coincidence than diagonal RGB patter and that is why Sony presumes only 67% efficiency out of this 45deg patter?
 
To talk about 1:1 resolution we need to go to the basics of sensors and sampling theory. Lets simplify things by looking at a monochrome sensor of 4096x2304 pixels.

You'd think you'd get 1:1 resolution (100% resolution efficiency for want of a better term) in that case. But we're forgetting sampling theory, which says that you need at least as many samples as you want line-pairs of resolution. We generally talk lines of resolution, not line-pairs, which now means to accurately sample without aliasing 4k resolution, we need 4k samples. Great, we have 4k samples, but what we don't have is a band-wdith limited signal that as only resolutions up to, but not including 4k in it.

In audio, or digital images it's easy enough to band-width limit a signal via a low pass filter. But optically, it's a bit more tricky. We have the OLPF or optical low pass filter, but they're very "shallow" filters, in that although they reduce the high detail we want removed before the image hits the sensor, but they will also remove resolution we actually want to keep. That's the nature of OLPFs, and they can't be "steeper" due to the absence of negative photons!

This leads us to the situation that we cannot obey sampling theory without making a very blurry image, which implies the need to balance aliasing and resolution carefully! It implies that our monochrome sensor can indeed have 100% of it's rated resolution, but only on the condition we allow for lots of aliasing. The less aliasing we want, the lower the rated resolution % we have to accept.

Graeme
 
Andrew M. said:
Graeme, Sony presumes that they will be able to demosaic its new CMOS 17MP sensor with 67%, I presume artifacts free, efficiency.

RED has something like 78% factor so 5K is resolving to 4K.

Does Bayer patter lends itself better for color coincidence than diagonal RGB patter and that is why Sony presumes only 67% efficiency out of this 45deg patter?
Click to expand...

Actually, they're assuming 100% luma efficiency, which as I show below is impossible unless you accept for bad luma aliasing. Their 67% figure is not related to the 78% linear resolution factor that we get after OLPF and demosaic. Their 67% factor is relative to sensing RGB at each pixel being give 100%. They're only getting 67% of the RGB relative to their 100%, because they're only using 67% of the number of pixels. Their 67% is just a relative pixel count compared to an RGB sensor - nothing more, nothing less. They give 33% to traditional Bayer because quite amazingly they only use 1/3 of the pixels of their RGB example and 1/2 of the pixels of their Q67 example. 1/2 of 67% is 33%. In other words they double the number of pixels they use for Q67 compared to bayer to make it look better. If they'd kept the number of pixels the same, it's look worse!!!

Graeme
 
Graeme Nattress said:
To talk about 1:1 resolution we need to go to the basics of sensors and sampling theory. Lets simplify things by looking at a monochrome sensor of 4096x2304 pixels.

...

...


This leads us to the situation that we cannot obey sampling theory without making a very blurry image, which implies the need to balance aliasing and resolution carefully! It implies that our monochrome sensor can indeed have 100% of it's rated resolution, but only on the condition we allow for lots of aliasing. The less aliasing we want, the lower the rated resolution % we have to accept.

Graeme
Click to expand...

Wow, O.K. I have a sense of where this is going… especially as it pertains to Graeme’s special magic, It may be hard to find the correct language for this.

BTW. If anybody is interested in sampling resolution and MTFs there’s a great book called “Modulation Transfer Function in Optical and Electro-Optical Systems” by Glenn D Boreman SPIE press (ISBN 0-8194-4143-0), it leads you by the hand very simply step by step into some really good stuff.

I have probably not interpreted what you have said correctly, but from what I infer, is that you are saying that you can achieve better resolution based results, theoretically and practically than what normal sampling theory would suggest. I.e. a mathematically derived Modulation Transfer Curve, implications of the Nyquist Limit and various types of natural aliasing that happens (as a result of undersampling).

So by massaging the way the “pixels” decide to organize themselves at the edge of where weird aliasing artifacts start to creep in you have found better ways for how these “pixels” should perhaps re-organize and contribute their tonal information into more “probable” or useful groupings. I don’t remember exactly which thread that was, but I seem to remember some time ago JIM showing resolution/MTF charts that seem to “transend” or go substantially beyond the Nyquist limit and or other theoretical and practical limits.

Is this not the Graeme special sauce? (or at least one ingredient of said special sauce??)

Notional 1:1 monochromatic image is a good place to start…
 
Stephen indeed the stuff on MTF is fine, but the Bayer stuff is rather biassed and in places incorrect. It's funny to read or hear about the supposed "problems" with Bayer, but when you see the image from the RED that is highly detailed, but practically free from aliasing and moire, yet the F35 shows strong aliasing and chroma moire.

Eric, at no point do we break or violate sampling theory and get a result better than basic theory allows. Remember sampling theory says that for a 4k monochrome sensor you can get no more resolution than 4k, and it says if you try to achieve 4k or you don't have a bandwidth limited input you will get aliasing.

Now Bayer is indeed a compromise. With it you'll never achieve full 100% resolution of the sensor, but we know we can't do that anyway without violating sampling theory. We also know we don't want excessive aliasing so we employ a good OLPF.

With the OLPF in place we know we can't measure 4k, so at that point we know there's little benefit to fully populating our monochrome sensor to sense detail we'll never see! And we can now use the gaps (where we removed some of the pixels from the fully populated sensor) to sense red and blue too. Hence the Bayer pattern sensor is born.

Given we want an image without nasty aliasing, it's easy to see why the Bayer pattern is a very efficient use of the pixels you have.

Graeme
 
Now let's have some math fun!!!

We'll take the Q67 as a starting point. It has 4096x2304 (assuming 16:9 5k) greens, and therefore as it's a Bayer pattern, it has 4096x2304 / 4 red pixels and 4096x2304 / 4 blue pixels.

We know from sampling theory the maximum luma resolution is 4k (as we have 4096 pixels across) and we know if we don't OLPF it will alias badly. Let's give Q67 the magical benefit of a perfect OLPF here (which doesn't exist, but biases the numbers in the favour of Q67). Let's give it luma resolution of 4096. We know it can achieve that, but without the magic OLPF it will alias. If we don't want bad aliasing, it will be something like 80%-90% of 5096 = 3276 to 3686. The OLPF will have negligible effect on chroma, so they'll need a good chroma algorithm there, so let's grant them that also. We'll assume that you'll see negligible chroma moire, just like the RED.

continued....
 
Last edited:
Hi Graeme,

First of all let me thank you for opening such a thread. I know it benefits you guys to inform us on the advantages of your preferred sensor technology over other camera makers' choices but this warm, friendly approach is very much appreciated :)

I have a few questions that I have been wondering about for a while and I guess this is the best place to ask them.

1. I understand that the R1/MX OLPF bins 20% of measured resolution to avoid moire artefacts. Is there any reason why a camera cannot be made without the OLPF and a postproduction low pass filter slider/tab could be added to redcine akin to a reverse "sharpen" instead?

I know the drawback of this would be added computation time for debayering and some less skilled users creating less than desirable examples of work done on Epic/Scarlet...but as moores law plows ahead and everybody gets comfortable with the RAW workflow would this be an option? Or is there no point?

I know every CMOS based DSLR has an OLPF and that's a much more mature market than motion RAW photography ...so the fact that this option isnt offered probably means that there is no point. I was just wondering if there would be a point for the user to dial in a sweet spot depending on the nature of the project?

2. You mentioned in the Sony thread that you guys looked into the Foveon sensor technology but found it wanting in terms of speed and artifacts. I was wondering if you looked seriously into CCD's and/or any other type of sensor design? Did you guys ever attempt to create a completely new type of sensor? Were there any pre Mysterium prototype RED sensors built on Foveon/CCD technology?

I doubt it as the whole Red Digital Cinema Company seems to have rallied around the M sensor first and then worked very hard to build a camera around this technology.

Still, any pre Breadbox/Frankie prototypes out there with non CMOS/Bayer Pattern sensors in them?

Thank you for your time and considerable wisdom :)
 
Now let's take those same pixels and re-arrange them into a traditional Bayer pattern. It makes an image which has 4096x2304 * 2 pixels in it (re-arrange that to 4096xroot2*2304*root2 to see how to calculate the image dimensions), which as a 16:9 array gives 5792x3258.

That's a very large Bayer array, but it has the very same number of red, green and blue pixels as the Q67, just arranged differently. They're the same area, and they cover the same area of sensor. Only thing changed is the arrangement.

Now from experience we know how a Bayer array measures with a Real OLPF (not a magic one) and real compression too, and we get around 78% of the rated resolution, so let's do the numbers again...

5792 * 78% = 4518 for luma resolution. That's about 4.5k. Nice.
5792/2 = 2896 for chroma resolution. With a good demosaic we'll get negligible chroma moire too.

continued...
 
End result is that from the same pixels, just re-arranged we've got a theoretical max luma resolution with perfect magical OLPF on Q67 of 4096.

For RED style Traditional Bayer using the same number of pixels and a real life OLPF we get 4518 luma resolution, and we know that's post compression too.

Graeme
 
Thanks!

1) Filtering must occur before sampling or you've just violated sampling theory. Once an image is corrupted by aliases, there's no way for a post processing filter to remove them without removing uncorrupted image data also. That's why we go to such lengths to avoid aliasing occurring in the first place.

Now there are moire removal filters that can in some circumstances work, but they're slow and even the best I've seen are problematic on stills and would be a lot worse on motion.

2) Foveon is too slow a sensor to get the fps we need, and the colour processing is as intensive as demosaicing and you have colour fidelity issues due to silicon being a poor colour filter. As for the CCD approach we went with CMOS because it had the balance of features we needed - DR, noise, fps, etc.

Graeme
 
How does the spectral characteristic of the Bayer color filter interact with the chroma demosaic algorithms? Can the filter colors be developed to enhance the information extraction for better luma and chroma resolution than one would get with a theoretically pure RGB filter?
 
Graeme Nattress said:
Eric, at no point do we break or violate sampling theory and get a result better than basic theory allows. Remember sampling theory says that for a 4k monochrome sensor you can get no more resolution than 4k, and it says if you try to achieve 4k or you don't have a bandwidth limited input you will get aliasing.

Now Bayer is indeed a compromise. With it you'll never achieve full 100% resolution of the sensor, but we know we can't do that anyway without violating sampling theory. We also know we don't want excessive aliasing so we employ a good OLPF.

With the OLPF in place we know we can't measure 4k, so at that point we know there's little benefit to fully populating our monochrome sensor to sense detail we'll never see! And we can now use the gaps (where we removed some of the pixels from the fully populated sensor) to sense red and blue too. Hence the Bayer pattern sensor is born.

Given we want an image without nasty aliasing, it's easy to see why the Bayer pattern is a very efficient use of the pixels you have.

Graeme
Click to expand...


This is what I had in mind, Perhaps you can clarify a little as to how the envelope was pushed, (without giving too much away), i.e. say nice things about what you and RED team are doing to push things to new limits..

Jim wrote the following in

www.reduser.net/forum/showthread.php?t=41121

Apologies for my retarded “neticate/forum skills”
Quoting Jim
“Big Resolution News...

--------------------------------------------------------------------------------

Well... even we were lulled to sleep with conventional thinking. It was the industry's perception (before RED) that the most measured resolution you could get from a Bayer Patter sensor was 75%. We showed that with Graeme's magic, you could get 80%. We believed for the past 2 years that was the theoretical limit.

We were wrong. What new techniques in the new sensor, new OLPF, new debayering technology, better compression and a wicked sharp RED Pro Prime... that "limit" has been blow by.

Here is a test of the new M-X sensor shot 4K 16:9 at REDCODE 42 with a 100mm RPP. It shows that 4K now yields 3.5K measured resolution (as compared to 3.2K in old tests). We also will show (as soon as I post the next chart) that 4.5K from the same sensor and lens yields a true 4.1K measured resolution.

I guess the "RED doesn't really deliver 4K" is old news.

Jim
www.reduser.net/forum/showthread.php?t=41121
 

Attachments

  • 1265951628[1].jpg
    1265951628[1].jpg
    57.4 KB · Views: 0
  • 1265973099[1].jpg
    1265973099[1].jpg
    58.9 KB · Views: 0
Eric... this is where Graeme can't comment, as you fully understand. "Without giving too much away."

No hints about what we do in addition to the principals Graeme is sharing.

Jim
 
Jannard said:
Eric... this is where Graeme can't comment, as you fully understand. "Without giving too much away."

No hints about what we do in addition to the principals Graeme is sharing.

Jim
Click to expand...
Well thanks anyway,

It’s just really cool to see individuals and moreover groups of talented individuals achieve amazing things. i.e. pushing the envelope, this really has to be held up in high praise and set as an example that if you put your mind to things that amazing , beautiful and useful things can be accomplished. …

Not like the SONY presentation (mentioned on the other thread 'SONY" and “the Q67”) that tries to make out that only humungous corporations have the right to dictate what is appropriate technology or not…; hokey algorithms etc. I would expect a more honorable attitude from them. I’m not against SONY but the presentation really needed a re-think…at least for an American audience.

Cheers,

Eric
 
You must log in or register to reply here.
Back
Top