General Demosaicing Questions

[Aaron Newsome]

With the OLPF in place we know we can't measure 4k, so at that point we know there's little benefit to fully populating our monochrome sensor to sense detail we'll never see! And we can now use the gaps (where we removed some of the pixels from the fully populated sensor) to sense red and blue too. Hence the Bayer pattern sensor is born.

Given we want an image without nasty aliasing, it's easy to see why the Bayer pattern is a very efficient use of the pixels you have.

Graeme

This is why Graeme is always the smartest guy in the room. Succinct, simple, easy to understand. Even for us people who aren't totally up to speed on these concepts.

 

[Andrew M.]

So what about the claim of Sony that they will be able to feed 4K video out of the camera As oppose of other cameras not having this feature.
They didn’t mention if it will be 4:2:2 or 4:4:4 but ether way what interface they will use to get this bandwidth out?

I suspect something that is already in Sony arsenal?
They said 4K digital negative will be written to 1TB SR memory.
Looks like they are referring to RAW being written to the SR memory.
But will be this RAW available on some camera output interface as well?

 

[Dj Joofa]

Nothing has changed. Their analysis was wrong then, and it is wrong now. Dig up a link to the old thread....

Graeme

Graeme, please don't trash an otherwise very informative series of videos from Panasonic. You might some disagreement with some aspects of it, but the series overall is quite good.

Trust me, despite Red's incessant talk of supperiority of its technology, I have yet to see any dedicated effort for public education that is remotely close to the effort of this series of Panasonic's lectures. The fact of the matter is that Red is remarkably lacking in its presence in scientific avenues, conferences, and journals, compared to some of the companies being attacked in this thread. This is not meant as a provoking statement, rather putting things in perspective.

Sincerely,

Joofa

 

[Carsten Fenstermacher]

Graeme, please don't trash an otherwise very informative series of videos from Panasonic. You might some disagreement with some aspects of it, but the series overall is quite good.

Trust me, despite Red's incessant talk of supperiority of its technology, I have yet to see any dedicated effort for public education that is remotely close to the effort of this series of Panasonic's lectures. The fact of the matter is that Red is remarkably lacking in its presence in scientific avenues, conferences, and journals, compared to some of the companies being attacked in this thread. This is not meant as a provoking statement, rather putting things in perspective.

Sincerely,

Joofa

I see tens of thousands of posts by Graeme, Jim, Jarred, and on. That is much more valuable to us users and owners than scientific journal masturbation. I hope Red continues on it's current path.

BTW, I think you mean Panavision, not Panasonic.

 

[Dan Hudgins]

numbers vs. what you can see...

numbers vs. what you can see...

4:2:2 or 4:4:4 but ether way what interface they will use to get this bandwidth out?

In real world cameras of today, area filters are used because you cannot get good image at any ISO and more so at higher ISO without using at least area filters on the chroma.

Even with a large three chip camera, 4:4:4 would not have noise low enough to be as usable without some area filtering, as well as temporial filtering.

Just because the camera makers do not talk about their internal filters, does not mean that in the conversion of sensor data to what you see filters are not being used, and so sending a filtered image over 4:2:2 vs. 4:4:4 at 4K resolution may not impact the image as you would expect based just on the lower chroma bandwidth figures.

In fact compressed images and those sent over lower bandwidth chroma methods may LOOK "better" than full bandwith images that are not filtered because the sensor noise has been made lower in contrast.

The gain of shooting True RAW is to give you a larger range of adjustment on the digital negative, not to show the extra sensor noise to the public.

 

[Graeme Nattress]

Joofa, that'd be Panavision (with representatives from Panavision & Canon), not Panasonic, and if you read back to when those videos were first mentioned in this thread I've praised their explanation on MTF etc, and criticized their Bayer pattern explanations.

Now as for education, given John Galt's (of Panavision) article on "Marketing Pixels", I see his videos in exactly the same way.

Graeme

 

[Andrew M.]

In fact compressed images and those sent over lower bandwidth chroma methods may LOOK "better" than full bandwith images that are not filtered because the sensor noise has been made lower in contrast.

The gain of shooting True RAW is to give you a larger range of adjustment on the digital negative, not to show the extra sensor noise to the public.

This is exactly mine point. In cinematography I don’t know of any situation where you would like RAW real time monitoring.

For past 100 years film was loaded and unloaded from the camera and now loading memories and unloading it serves the purpose exactly well.

Full res 4:2:2 UHDTV monitoring for pulling focus and general scene adjustment in full resolution would be not bad though.
For now we have this 1:1 button that does the job and thanks to it the monitor screen do not have to be 60”

However Sony is signalling that this is one of the challenges to deliver 4K monitoring. This means that they are thinking about the future support of 4K or rather UHDTV broadcasting version of it seriously.

This would be very valuable for Sony to push for this new UHDTV standards bit harder now so they will not have to change the engineering details later to adjust for the standard changes.

 

[Bill Anderson]

Guys, if you have a problem with how someone is criticizing a piece of technology, do us the courtesy of trying to understand the nature and the scope of the criticism, before you make broad insulting statements about their intentions. People who continue to bring some of the most advanced and affordable cameras into the community, deserve to be treated fairly (as do others). To suggest that the likes of Graeme is capable of cruelly trashing another technology Co. out of hand is, in itself, the most mindless form of "trashing".

 

[Mike Harrington]

ok...not exactly demosaicing question here...but none the less

what are the limitations of a bayer pattern sensor that do not allow a global shutter, and why is it easier in a CCD...

also it seems to me Sony is claiming a global shutter for there bayer (CMOS) sensor....is it impossible or just really hard....

I understand that epic will have very fast read times that can negate the issues associated with rolling shutter....I still have a very limited understanding however why such limitations between the 2 types of sensors (CCD, CMOS)

Thanks

 

[Graeme Nattress]

Mike, from what I understand, and remember I'm not a sensor guy, all things being equal, adding global shutter to a pixel design increases noise, hence reducing dynamic range. It may also increase power consumption too, which leads to lower battery life.

Graeme

 

[Mike Harrington]

Thanks Graeme...I had just assumed you were the sensor guru as well :hat:

I am trying to figure out the difference in an electronic shutter in a CCD and a CMOS....
It would seem in every CCD i can think of there is a global shutter....while very few CMOS (i personally don't know of any) have a global shutter.
There must be a design difference that makes it far easier to implement on a CCD....

 

[Sanjin Jukic]

I'm not an expert in demosaicing field at all but still try to learn about it.

The latest I found on the Internet and it looks interesting demo about

Image Interpolation - Demosaicing Algorithms for Bayer Pattern Sensors >>>

Algorithms:

- Pixel Binning
- Nearest Neighbor
- Bilinear
- Smooth Hue Transition
- Edge-sensing Bilinear
- Relative Edge-sensing Bilinear
- Edge-sensing Bilinear 2
- Variable Number Gradients
- Pattern Recognition

Choose an algorithm from the above. Feel free to send suggestions/links for new ones to nathanyan@berkeley.edu.

Shown here is the original image for reference, and the simulated Bayer raw data which the different algoriths are run on.
The simulated Bayer data is generated by taking only green, red, and blue values from the corresponding pixel in a 2x2 GRGB Bayer pattern.
Assuming the image used is a 100% crop taken from an image with the top-left pixel (at position 0,0) being a first green photosite (G1),
this should be a fairly accurate representation of the actual raw data (assuming of course, that a camera using a Bayer filter sensor was used,
and ignoring any compression losses due to compression to 24-bit true color and possible JPEG lossy compression).

Note that this demo does not quite work as effectively with resized images (this demo resizes images greater than 300x300 for performance reasons),
or images not generated from a camera using a Bayer sensor. In these cases, resolution can approach or exceed that of a Bayer sensor and anti-aliasing (AA) filter combination,
and severe mosaicing artifacts should be expected to appear, no matter the algorithm.

 

[Graeme Nattress]

Sanjin, that's a great guide to some of the basic algorithms.

Graeme

 

[David Rasberry]

Sanjin, that's a great guide to some of the basic algorithms.

Graeme

After playing with those a bit, I have to say yours are much better Graeme.

 

[Sanjin Jukic]

Graeme,

as you probably know better this demosaicing area or topic is too big and with this link

above I just wanted to give an idea to an average RED user what it is about in general.

It's like a watching and immersing myself in a sort of real 3D SF movie...

 

[Andrew M.]

Detecting Moire effect in the frame.

Let’s take an example of two frames taken 1/12 second apart with both frame having Moire effect in it because our sensor pixel pattern is interfering with the stripe pattern on the talent jacket as an example.

If camera will not move at all we will not see pulsating or changing the Moire effect.
Yes the effect will be there but will be difficult to recognize if it is the pattern on the jacket itself or just the pattern on the jacket plus Moire effect.

In real movie such situations where nothing is moving, even moves of one or two pixel wide for longer than 2 seconds are unusual.

Now if there is a Moire effect in these two frames it will be a different Moire effect in the first frame and a different in the second. Now by figuring out how much the picture moved we could offset the picture in opposite direction and subtract the frames from each other. This will cancel everything but Moire.

Problem is that there are moving objects in the frame as well and to figure out these movements and cancel it against each other will be a bit tricky.

Kind of like trying to merge two HDR frames

 

[Rob Ruffo]

Andrew, when moire moves, it moves more than the object that caused the moire to move in the first place. This aspect was used in some micrometers I think as the moire amplified the apparent movement and made the device more accurate. I think I first saw that in an Open University programme on UK TV back in the 70s (you can guess my age back then) but I could be mistaken - it might have been the 80s, but either way, it's funy how you remember these things....

I remember as a kid being driven down the motorway near our house, and seeing the over-head signs that are held in place over the road by a mesh structure, which produced the most beautiful moire! It took me a while to find out what the effect was called and how it was caused, but I've been interested in moire and aliasing since I was a child.

Graeme

I hope if I ever have children they are at least a bit like you were.

 

[Graeme Nattress]

Rob, my eldest daughter Megan (she's 6) loves "Look Around You" - http://www.youtube.com/watch?v=drE5cHe6c3s which I'm sure you'll appreciate if you've ever seen any 70s UK schools science TV. I personally think it's one of the very best bits of TV comedy I've seen. Series two did to Tomorrow's world what series one does to schools science TV. All very funny indeed.

Graeme

 

[Rob Ruffo]

Rob, my eldest daughter Megan (she's 6) loves "Look Around You" - http://www.youtube.com/watch?v=drE5cHe6c3s which I'm sure you'll appreciate if you've ever seen any 70s UK schools science TV. I personally think it's one of the very best bits of TV comedy I've seen. Series two did to Tomorrow's world what series one does to schools science TV. All very funny indeed.

Graeme

Thanks for the awesome link. Made me smile.

 

[Christopher Probst]

Graeme,

Ironically, I'm knee deep in this very sort of topic for a highly technical Digital Imagers article I'm putting together for American Cinematographer. I know I've asked several times to both you, Jim and Jarred for Red to contribute to this article and been declind (very politely, of course), but what you're talking about in this thread is exactly the sort of commentary I'm looking for. I wish again Red would reconsider their position in having a voice in American Cinematographer. Believe it or not, AC (and specifically ME in this instance) is not an enemy!

Perhaps I'll have to do an Epic article as a means of amends... But of course, that stipulates I get my hands on one!

Still waiting.