General Demosaicing Questions

[paulherrin]

haha, that's good. look at some nice things that aren't a computer screen for awhile then. but you like to work on the punch cards, right? so i guess you can look at a screen.

 

[Christopher Probst]

Out of curiousity, does anyone know the pixel size the Q67 is working with after upping the number of photosites?

 

[Graeme Nattress]

Keeping things simple, if at our 4k we're 5.4 micron pixels (29.16 area) doubling the number of pixels would change them to 5.4 / root 2 which is 3.8 micron (14.44 area).

 

[Christopher Probst]

Graeme, that's not taking into account fill factor, right?

All my delving into imager history has left me wondering to myself where we are heading with sensor technology. And I still haven't pinpointed in my mind a theoretical finite limit to the number of CMOS photosites a Super35mm-sized sensor can ultimately hold, Not that I have time to break out a calculator and try to figure it out... For arguement's sake let's say the MX sensor had 50% fill factor. Would that make the actual photo-sensitive area also 5.4/root2? Without knowing how much fill factor we're really dealing with, it's hard to know where improvements can occurr.

My point in a previous post was something to the idea of that if Moore's law continues, perhaps we can lock-in at a sweet spot photo-sensitive area size in the pixel and continue miniaturizing the rest of the additional components. It'd be nice to use technological gains to either add to the photosensitive area's size, or add additional photosites. And with the current dynamic range performance at 5.4 microns, whatever that photo-sensitive area is seems like that is doing just fine. So perhaps there's room for more resolution still in the future. But wouldn't you agree that at some point we don't want smaller and smaller photosites just in order to get more of them for bragging rights?

Arri says the Alexa's pixel size is 8.25 mircons, but for my money, they are pretty close to the same dynamic range and sensitivity of the MX. And although the Genesis is CCD, I'm told their pixel size is 4.1. But certainly CCD doesn't have the same fill factor issues as CMOS.

Everyone I have talked has universally agreed that CMOS sensors are the present and the future, and even though I'm only about to roll my first epic shoot, I'm already curious what the next generation of imagers will bring.

 

[Graeme Nattress]

I was keeping things simple, so I was just looking at overall pixel size (pitch) without thinking on fill factor.

You don't want more pixels just to have more pixels. That doesn't help. However more pixels can lead to other benefits, like the improved MTF and lower aliasing.

I don't design pixels, so I don't know where the limits are. I'm always happy though to see better pixel designs and what they can do for an image.

I think CMOS is the current best direction, but you never know what might come up in the future - I'm not a sensor designer and I don't have a crystal ball.

F35 pixels are non-square, probably around 3.3 microns across and 5 microns down in size, roughly?? Just doing the math off the top of my head

Graeme

 

[Dj Joofa]

Two extremes in sharpness - range for deMosacing

Two extremes in sharpness - range for deMosacing

For a visual comparison I captured more or less the same image using a Bayer sensor and a monochrome sensor of the same variety, while keeping other parameters the same. An extreme in raw sharpness is the data from the mono sensor, shown on right below. For the other end I used bilinear interpolation for deMosaic, which is a simple and fast, but relatively inferior method, shown on the left below.

Any, deMosaicing algorithm, including Red's, will give produce an image with sharpness between these two extremes. None will give a sharpness equal to the right image.

Sincerely,

Joofa

 

[Dan Hudgins]

OLPF?

OLPF?

Any, deMosaicing algorithm, including Red's, will give produce an image with sharpness between these two extremes. None will give a sharpness equal to the right image.

Was there an OLPF on the monochorme image?

 

[Dj Joofa]

Was there an OLPF on the monochorme image?

Hi Dan,

No OLPF on either sensor. Everything, including the sensors, was the same except one sensor having a CFA of Bayer type and the other monochrome. But they were the same sensors in technology.

Sincerely,

Joofa

 

[Dan Hudgins]

Sharpen or not to sharpen...

Sharpen or not to sharpen...

No OLPF on either sensor.

I think you can sharpen both, as there are some loses from the taking lens and flare in the micro-lens array etc. If both had the same OLPF, and both were processed to increase the HF, then the gap between them might be closer.

Also the way you interpolate the two green pixels should not be the same as the red and blue color planes, as the green plane can come closer to the monochrome image due to the "missing" pixels being surrounded by four "real" pixels.

For test charts the repeating pattern of lines that cover several pixels, in the green half the pixels that fall on the line are "real".

One issue I have been having is that the interpolated pixels and the "real" pixels do not have the same edge contrast, and under very heavy sharpen the interpolated pixels seem to sharpen to higher or lower values making an artifact. There are ways to compensate for that artifact (such as ARRI using the 1/1.5x oversample ratio), but some of those may not work all the time, RED seems to have a good way of dealing with that issue, and they are using much lower sharpen values, so that would help.

It is possable to get better results using the right methods AND an OLPF, then not using an OLPF and no sharpening. If you don't sharpen at all and don't use an OLPF, then you cannot compensate for the lens softness, if you sharpen without an OLPF being used you get out of control aliasing and moire (you should see what that ACAM cat image looks like under heavy processing! OMG), so I think its better to sharpen with an OLPF then to not sharpen without an OLPF as I have looked at data from several cameras going both ways. And that seems to be the path RED has taken also.

It seems Kodak thinks the CCD pixel fill factor being larger than CMOS needs an OLPF less (i.e. not at all at least for stills, second hand info), but you cannot depend on the lens to act as an OLPF, the CoC is not an amorphous blob, it has fine features that can cause aliasing and moire.

Normal de-Bayer software (for DNG) cannot automatically sharpen enough in and in the right ways to get as good results maybe on some sensors. If you look at the Kinor-2K demo frames it seems they also do not have a working OLPF in their camera and when you do shapen the chroma moire makes bright colored spots on small highlights.

That seems to be a flaw in the DNG based workflow, the DNG format does not seem to have the fine adjustment needed for (small sensor) cinema work, at least I have not seen better overall results so far than what I can get from the program I am working on, in particular the DNG de-Bayer programs, or at least some of them, show pink highlights and other color balance issues in the ACAM test demo sample files.

I'm reading the DNG files, then do everything with full manual adjustments shot by shot, it seems the main way to get "usable" results since people want to shoot under un-controlled lighting. In many cases there is nothing to use for a white or gray ref in the frame, so fully automatic processing would probably fail, only preset or manual adjustments might work.

Anyway, just interpolating the Bayer sensor would not be usable due to lens softness, CA and other lens issues, some processing and filtering is needed to get a usable image, so the end Bayer result would be sharper maybe than the same image on a monochrome sensor without any processing to compensate for the lens and other softness. It depends on the size of the sensor, its pixels, and the lens and f/ stop used. For 2 micron pixels the image on the sensor itself is always softer than the interpolation alone makes it (adds little more unsharp).

Also you can use an active interpolation on the monochrome sensor to "increase" its functional resolution as the active interpolation can follow curves and such to make it look less interpolated. Or you can use a fractle enlargement of the monochrome image to "increase" its resolution by making the original pixels smaller in relation the image height (I have not used fractles, but such software seems to exist). I wonder if fractles would work on four plane de-Bayer interpolation?

How active resolution increase impacts aliasing is an interesting subject, in other words can you get LESS aliasing from a resolution doubled image...

Was an IR filter used on the Bayer image?

 

[Graeme Nattress]

If you match the brightness between the two images, you'll see the bayer one looks sharper than it does now. As Dan points out, using an OLPF (as we know we should especially for motion imaging) will also reduce the perceived differences somewhat.

Graeme

 

[Dj Joofa]

Dan, the intention was not to show what can be done on the Bayer image to make it look better. That is the job of a good deMosaicing algorithm. Rather the intent was to show a a crude "range" between two rougly defined extremes where a deMosaic algorithm may be operating in the particular settings of where/how I acquired those images.

No IR filter was used on either sensor.

BTW, you might have noticed the light loss in the Bayer image, which makes it look darker. Here is a comparison of light loss in a Bayer CFA with an image acquired from a Bayer and a monochrome sensor of the same variety in the same conditions (not the same sensor as my previous images). Seems like a significant ight loss on the Bayer:

Top image is Bayer, bottom is mono.

Sincerely,

Joofa

 

[Dan Hudgins]

spectrum

spectrum

No IR filter was used on either sensor.

Seems odd that you got any color, as your light source would have had to not be producing much IR, otherwise your color balance is skewed, without the IR block you need to adjust the chroma matrix, and that can change the balance between the "green" pixels and the "red" and "blue" pixels. Since the red and blue are only 25% they do not show as much fine detail, to get "normal" Bayer results the green pixels make up most of the luma in the image and so make the image sharper, with IR the chroma matrix does not always have the right balance.

Another problem is that without an IR filter, the monochrome camera is not on the NTSC or other luma curve with relation to the lens CA correction, so there is an advantage for the monochrome sensor to pick up a peak in the yellow maybe etc. on the lighting, many lenses are less sharp in the blue and so with the monochrome sensor not seeing the blue as much you get a sharper image.

Its not ment to discount what you have said, but to be fair to the issue of mono vs. Bayer, you need to take into account the way the spectrum goes through the lens and how that impacts the image made on the sensor, as you are using an image to test the sensor with, and it is the way the light passes through the Bayer filter and the chroma matrix used that relates the lens issues into the interpolation.

If the light is not continuous spectrum, then where those peaks fall in relationship to the Bayer filters, or the lens corrections, or the monochrome sensors sensitivity all impact the issue.

My minor point was that it might indeed be easy to make the image from a Bayer sensor SHARPER than the unprocessed Monochrome image, more so on smaller sensors with tiny pixels.

But the Bayer sensor would need an OLPF in order to process the data right, and the Mono sensor should also have one, although with small enough pixels, you might not see aliasing in a three chip camera because of diffraction at the 1 micron pixel size range.

I understand what you are trying to show, but if you want to do a real comparison between a monochrome sensor camera without an OLPF and a Bayer sensor, then the Bayer sensor should have an OLPF+IR/UV cut, and also be processed in the best way possable, since just showing that you have bad processing of the Bayer data is only proving that bad processing of Bayer data looks bad.

The Bayer filters SHOULD absorb 66% of the light or so, well maybe not as you want some luma in all three colors, but yes you need to lose a stop. In a three chip camera you end up with bigger losses. The OLPF+IR/UV filter also cut about 2/3 stop at the far red and maybe 1/3 stop at the blue end, and 1/4 stop for green. Add those together and you get a Bayer sensor being about 1.5 to 2 stops slower than a "bare" monochrome sensor.

Your images don't look well gamma adjusted, and so that makes the difference look larger than it would be at mid-tone if both sensors were exposed so 90% white was 25% signal and mid-tone was lifted to 0.458 signal for a gamma 2.2 result.

If one wants to make monochrome images you get better results using a Bayer filter sensor as far as tonal range, at least in true RAW data, that is because you can use the Bayer filters to act as ND filters and then you get a kind of HDR image if you shoot through a filter like one or two 80A to increase the stop spread between the three colors (green is about 2/3" stop above red and about 1/2 stop above blue without a filter, but with a filter over the lens you can increase the spread). Then use a good de-Bayer and adjust the chroma matrix to give a monochrome result with the right mix of r-g-b signals in relation to the filter used.

Even for color results, you can add light green filters to increase the tonal range (spread between green and red/blue) if you adjust the de-Bayer to take advantage of that (using DRX type processing to increase highlight detail).

So its possable to get a wider dynamic range by using the SAME sensor with the Bayer filter, than without it. And dynamic range can be more important to making useful images than minor resolution gains that will for the most part be lost on final viewing.

 

[Dan Hudgins]

Big or small

Big or small

I was thinking last night, it seems there are four major options to using Bayer sensors:

1) Process for 1:1 pixel size, i.e. 1920x1080 to 1920x1080

2) Shoot with a 1.5:1 oversample ratio like ALEXA and SCARLET i.e. 2880x1620 to 1920x1080.

3) Shoot 2:1 oversample ratio, and do not interpolate, just R=R G=(G1+G2)/2 B=B.

4) Shoot 2:1 oversample ratio, and interoplate then reduce to half size (with gauss blur then sharpen, maybe).

I'm not sure if 3) would look better than 2), ARRI seems to have gone with option 2) because they could not make a 2:1 oversample chip and deal with the extra bandwidth.

Anyone know about the 2) vs. 3) choice?

It would seem that 4) is better than 3), and 2) is better than 1), but the 2) vs. 3) is a bit unclear...

Both 3) and 4) reduce the OLPF blur impact the most, 2) somewhat, important because the OLPF is not ideal, and at larger f/ stops the longer rays at the edge of the cone blur more so the blur is not constant over the f/ stop range.

 

[Graeme Nattress]

If you're aiming for 2k, shooting 4k doing full demosaic and then proper downsample filter produces the best results. There's also the downsampling demosaic we use in software decode and that can work superbly well too. I've tried 3) in the past and found it problematic.

Even for 3k to 2k you'll get better results with a very nice demosaic followed by a really nice downsample than any attempt to get their easily and quickly methinks.

Graeme

 

[Dj Joofa]

Seems odd that you got any color, as your light source would have had to not be producing much IR, otherwise your color balance is skewed, without the IR block you need to adjust the chroma matrix, ..., with IR the chroma matrix does not always have the right balance.

I think the colors are fine. They just look similar to what I see in reality of the box that was photographed. I used a special technique to color balance them that adjusts in realtime without any prior calibration. May be that is why they came out better than you expected.

Another problem is that without an IR filter, the monochrome camera is not on the NTSC or other luma curve with relation to the lens CA correction, ...

Not sure what NTSC has to do here.

I understand what you are trying to show, but if you want to do a real comparison between a monochrome sensor camera without an OLPF and a Bayer sensor, then the Bayer sensor should have an OLPF+IR/UV cut, and also be processed in the best way possable, since just showing that you have bad processing of the Bayer data is only proving that bad processing of Bayer data looks bad.

I mentioned before also that the intention is not to get a better deMosaiced image. But, to show an effective range between two extremes as visual examples. This has certain implications. Because, deMosaicing with bilinear interpolation coupled with that special color processing, which I mentioned above that I used, actually renders a rather acceptable image at a very fast frame rate - it helps in many realtime situations where speed is important.

Bayer sensor being about 1.5 to 2 stops slower than a "bare" monochrome sensor.

I think you are right about that.

Your images don't look well gamma adjusted, and so that makes the difference look larger than it would be at mid-tone if both sensors were exposed so 90% white was 25% signal and mid-tone was lifted to 0.458 signal for a gamma 2.2 result.

We don't do gammas

If one wants to make monochrome images you get better results using a Bayer filter sensor as far as tonal range, at least in true RAW data, ... Then use a good de-Bayer and adjust the chroma matrix to give a monochrome result with the right mix of r-g-b signals in relation to the filter used.

Monochrome images from color data is tricky. There is some new realization that standard notions of b/w or grayscale are not always very attractive. For example the image below shows three methods of converting to grayscale from color. Luminance and lightness shown are among the usual methods, where as, an alternate method is generated by using some color saturation properties. Most people have told me that they like the tonality of alternate method more among the three shown, while it uses a non-standard method that I derived using some notions obtained from certain aspects of human vision.

Even for color results, you can add light green filters to increase the tonal range (spread between green and red/blue) if you adjust the de-Bayer to take advantage of that (using DRX type processing to increase highlight detail).

If you mean color fidelity here, then, adding those shades of green will help that are not included in the linear span of existing filters.

So its possable to get a wider dynamic range by using the SAME sensor with the Bayer filter, than without it.

EDIT: Oh, OK, I get what you are saying here.

Sincerely,

Joofa

 

[David Mullen ASC]

So what was the "alternate" method of extracting a b&w image?

 

[Dj Joofa]

So what was the "alternate" method of extracting a b&w image?

Do you like it better? BTW, there are some interesting things to note about that method. For example, kindly notice the blue square in the third row on left. In both luminance and lightness methods it is difficult to separate its boundary with the black borders that criss-cross all colors. Where as, in the alternate method it is much easier to figure that out. There are some other fun things also.

Regards,

Joofa

 

[David Mullen ASC]

So what was the "alternate" method?

 

[Dj Joofa]

So what was the "alternate" method?

The basic idea is that when you want to reduce the number of colors you are doing dimensionality reduction, in this case to a single dimension, which actually means that you are projecting colors along a certain direction in linear terminology. In the color space representation the luminance is just a vector - pointing in a certain fixed direction. Similarly, CIE lightness is also a "vector" if you allow non-linear coordinates in the color space. There are infinitely many directions to pick. I picked a "middle ground" direction that gives importance to color saturation.

Sincerely,

Joofa

 

[Peter Moretti]

Dj,

I can't say I exactly understand what you're doing, LOL, but it sounds somewhat similar to what 3Prong does with their ColorFix plugin.

If you look at the first row images it shows how simply desaturating an image to get B&W can produce some pretty lousy results. For the color spectrum shown, all the colors desaturate to the same grey b/c they have the same luminance:

http://www.3prong.com/ColorCom.htm

Their filter takes into account hue and saturation, so there is differentiation between colors of the same luminance but different hues &/or saturations.

It sounds like you are doing something similar.