sony

[Ian Laurie]

Bruce maybe this might help...

The alignment of photo sites dosen’t matter considerably in terms of the rectilinear array of pixels that you have to output. In CG and may other simple image processing operations, you first take an on screen pixel, and then you decide from what parts of another image, in this case the captured Bayer pattern image that you want to draw values from. You can use any number of sampling points that you wish, carry out any types of computation you wish in any combination or pattern that you want. You can apply all sorts of non-linear weighted functions. You can designate any number of xy positioned sub pixel point samples (floating point, single precision or double precision) to sample from and do math with. You can even carry out least squares adjustment computations from redundant samples to come up with a more probable spatial and radiometric result. The shape of the array of sub pixel precision points used to sample the Bayer image can be in the shape of Crusty the Clown if you want; this doesn’t affect the ability to output a rectilinear array of pixels. Then you can turn that on screen pixel to the computed and desired color value. Then you move onto screen pixel [1] from [0] and do the same.. and move onto the next and so on. Its just rendered, its no big deal really.

Thank you. At least someone is understanding exactly what i am trying to say.

Eric, you are talking about theory. I am talking about something that can be achieved realtime without custom chips. With as little power draw as possible. EG where processing power is so limited that they can't even do a full traditional debayer at 6K.

Again, if you look at that fxguide.com link you'll see that even the best algorithms still cause softness with just a 45 degree rotation.

So - you are telling me that there is no benefit to aligning the green sensor elements with the actual pixels being output? VS having the green sensor elements at a 45 degree angle so that every second line they are not aligned with the pixels and you have to average between them? Even for a realtime device?

Anyway, I'm out. The only people left arguing with me aren't really bothering to think things through.

Bruce Allen
www.boacinema.com

First, I have read every post you have written about this. particular close attention to the ones directed at me. Sorry if I have frustrated you, but as I said much earlier, I am not an engineer. But you suggest that what we are saying is theoretical. Thats just the point. It has to be theoretical because there is no one actually doing it. And I think you are drastically misunderstanding what i am suggesting. I am not suggest doing a 45 degree rotation in post or to the actual image. What i am suggesting is that I see no way that rotating the sensor 45 degrees is helpful in anyway shape or form other than vertical and horizontal aliasing. What my suggestion is that rather than rotate the sensor, which i still maintain is ridiculous logic that it somehow increases the colour retention, why not have the sensor and individual pixels READ in such a way that it is COMPILED and generates a file using the same sampling structure they are using in the Q67, targeting the same green pixels, but using the diagonal the 45 degree equivalent of vertical/horizontal Q67. Green pixels align. same processing as if they were vertical. doesn't mean your going to end up with a 45 degree picture that needs to be rotated, just means that your image is calculated diagonally.

When it comes down to it, it is absolutely the same pattern. whether your using the pixels beside and above to get the information or kitty corner on a 45 it makes zero difference colour reception wise.

And lastly I am not actually suggesting they do any of this. I am suggesting they are nuts for saying there is more colour information in a 45 bayer than a standard bayer. I too am firmly in the red camp. And i love all my sony products.... almost even bought a 10 year old Mini Disc recorder for nostalgic purposes. But this is mathematic BS. same surface area, same ratio of pixels is going to capture the same colour information. the only difference is processing.

 

[Bruce Allen]

Green pixels align. same processing as if they were vertical.

When it comes down to it, it is absolutely the same pattern. whether your using the pixels beside and above to get the information or kitty corner on a 45 it makes zero difference colour reception wise.

OK you just described two different processes:

A) 4K from a 6K 45 degree Bayer sensor (Q67) - sampling one G value
B) 4K from a 6K non-rotated Bayer sensor - sampling the G values beside and above, then averaging them.

You are telling me that there is no difference in terms of processing between these two functions? That if you tell one computer chip to do A and the other computer chip to do B that they'll complete both tasks at the same time, with the same power and memory use?

If you have unlimited processing power, there is no point to Q67 whatsoever. Just do a standard deBayer. So why does everyone only evaluate Q67 in a theoretical world of infinite processing power?

And lastly I am not actually suggesting they do any of this. I am suggesting they are nuts for saying there is more colour information in a 45 bayer than a standard bayer.

They are not saying that. They are saying that there is more color information in a 45 bayer with twice the pixels (that uses the lower-quality but cheaper sampling method of Q67) than a standard 4K bayer sensor.

In the next 18 months these things will not seem so theoretical even to you. The raw horse power of generalized GPU architectures are only just being realized; that’s the beauty, custom hardware is not so important any more for serious number crunching and signal processing.

Thanks for having a go anyway.

I am with you 100% on that. My old university coding partner has at nVidia doing GPU computing for a while and thinks a high quality damn fast CUDA Redcode decoder could be implemented right now, depending on the way the data is structured of course. I think RED are making too much selling Red Rocket cards to be interested in doing it though Personally I think they'd more than make up the difference in extra cameras sold, but understand resources are probably a bit stretched right now for them to look into it.

RE: the problem on hand, Sony would need something with less power use than a fat nVidia card if they wanted to stick it on one of their little remore cameras though? Maybe with 28nm GPUs in Q4 or something like AMD's embedded APUs?

Again, I think the Q67 is very much a "here and now" compromise. It must have been specced out back in 2009 if they presented those slides in Feb 2010.

Friendly cheers to you both, Ian and Eric.

Bruce Allen
www.boacinema.com

 

[paulherrin]

guys, we're still talking about this?

 

[Ian Laurie]

OK you just described two different processes:

A) 4K from a 6K 45 degree Bayer sensor (Q67) - sampling one G value
B) 4K from a 6K non-rotated Bayer sensor - sampling the G values beside and above, then averaging them.

You are telling me that there is no difference in terms of processing between these two functions? That if you tell one computer chip to do A and the other computer chip to do B that they'll complete both tasks at the same time, with the same power and memory use?[/url]

Ok, we are almost on the same page now. not quite, but almost. I am saying performing the EXACT same sampling technique except rotating the sampling technique. at which point I believe it will be almost identical ( there would be a little more over head for the rotational/ diagonal change in the sampling technique at most)

If you have unlimited processing power, there is no point to Q67 whatsoever. Just do a standard deBayer. So why does everyone only evaluate Q67 in a theoretical world of infinite processing power?



They are not saying that. They are saying that there is more color information in a 45 bayer with twice the pixels (that uses the lower-quality but cheaper sampling method of Q67) than a standard 4K bayer sensor.[/url]

maybe its just me, but watching that video and looking at those slides, that is not what they were saying, even if it is what they are doing. Also, I don't care if they jam 15 times as many photosites in the same area, or end up with an 8:4:4 colour information. it is still 2:1:1 mathematically. there is still no extra colour information to be had by doubling pixels. its still the same percentage, just with better resolution. call me crazy, but i just cant get past the shady voodoo math.

Friendly cheers to you both, Ian and Eric.

Bruce Allen
www.boacinema.com

Cheers to you too, and no hard feelings i hope. Once i get an idea in my head it becomes very difficult to shake. And i used to lay tile, which has given me a bayer pattern or two both on the flat and on 45, so patterns get stuck in my head and i can't see a difference between the two. Essentially it comes down to me not seeing why you can do something with a bayer on 45 that would be impossible to repeat on bayer standard.

 

[M Most]

Essentially it comes down to me not seeing why you can do something with a bayer on 45 that would be impossible to repeat on bayer standard.

At this point, I have to ask the obvious question: Who the hell cares? If Red had gone with a Foveon type design, very few people here would even know the name Bayer as anything other than aspirin. I'll say it again: unless you're the engineer or the manufacturer, the only thing that counts is the images. Everything else is technical detail that ultimately means nothing to the user. When Sony's product is available, everyone can evaluate the images to their hearts content and come up with at least some meaningful opinions. Until then, let the engineers argue about things that are only important to engineers, just as cinematographers are usually the only ones who argue about things like film stocks. The rest of us can wait until there are images to look at. Playing numbers games is for engineers and mathematicians. Most people here, including me, are neither.

 

[A. Järvinen]

This Q67-sensor looks a bit like Fuji's Super CCD they introduced to digital still cameras a decade ago. The cameras usually produced images that had twice the number of pixels what the sensor had. Fuji claimed this was necessary for getting all resolution out from the sensor. Sony's Q67 seems to work in quite opposite way. More information about early Fuji's Super CCDs: http://www.dpreview.com/reviews/fujis602z/

-A.Järvinen

 

[Sanjin Jukic]

I did intensive Google search (googling) and found out the following that a Bayer pattern rotated by 45° is an old idea:

Current displays use vertical RGB stripes. This is quite wasteful, especially considering how the human eye works.
The human eye produces a sharp image only for greenish colors, red and blue are blurred.
The eye therefore already spends 2/3 of the sensors in the fovea to green. Camera makers know how to use this,
and came up with the Bayer pattern. There, half the sensors are green, the rest red and blue.
Therefore, my suggestion is to use a Bayer pattern for the colors
(rotated by 45°, to have maximum resolution on green in horizontal and vertical direction):

You see, with a feature size of sqrt(2) of that of the RGB screen, we can have 1.5 the resolution;
using an identical feature size would allow for about twice the resolution, 200 instead of 94 dpi
(only green pixels count for resolution; when you have a camera advertized as 6 megapixel, the effective resolution is 3 megapixel).
The red and blue pixels can be used for sub-pixel anti-aliasing, which can improve percieved resolution and rendering accuracy further.


A High Resolution Display: Created 17nov2009. Last modified: 13feb2010 by Bernd Paysan >>>


That means it's an old engineer idea that we just have to wait and see Sony 4K Q67 image and then to conclude ourselves about that IQ...

 

[Eric Lange]

I’m starting to see what “SONY” is getting at. You basically have to scrumple up into a ball almost everything they presented and throw it over your shoulder as it is completely misleading.

However what follows IS PURE SPECULATION AND SHOULD NOT BE TAKEN SERIOUSLY!

So it boils down to what you think the most statistically significant portion of a group of photo sites is. The mysterious green “diamond” is used to contribute 100% of its value (unaltered (ish)) to what will eventually become a corresponding pixel (that has twice the notional area of the green photo site). However this centre diamond only accounts for 50% of the area of the corresponding (notional pixel) area that it will become. The belief here is that the centre portion of a pixel will be more important than the rest of the pixel area. You could draw a 3d graph of central tendency centered on the green diamond to one sigma i.e 67-68% of the total limits, and this bell curve will fall just inside the green diamond. However this leaves the blue and red photo-sites bordering the green diamond to pull values from. These are at edges and beyond so that color accuracy for those corresponding regions of space are much lower. I have to admit it’s a weird piece of lateral thinking to get very quick values, one of which the green remains largely unaltered. So it really could be that they don’t de-Bayer, but rather grab a set of rather quick and dirty values, the green possibly having better measures of central tendency for the pixel it gives rise to. If this is the case it’s no wonder the marketing guys were completely lost and just liked the number 67.

The idea being we see green response much more than red or blue so we have a nice green and a slightly dodgy blue or red as we can get away with that.

This is what I would expect. Essentially coming straight out of the camera a sort of RAW image, I would expect the green to be artificially punchy and the red and blues to lack color/spatial accuracy by comparison. I think the way that these images would be graded would be different from images that result from conventional Bayer.

The whole 33 vs 67% has to be ignored as it really does not come into it. They absolutely did not explain the sampling or the “belief” system and mathematics that is behind it.

(I have things to do so I can’t comment much and I am a little burnt out on this topic).

Cheers

Eric

 

[Ian Laurie]

Hey way to go sanjin! you found something that helped to make even just a fraction of sense to me! Although i think their slides are still complete garbage, i now can see their thought process behind it. still think its dumb, but at least i think i get it now.

 

[Michael Olsen]

At this point, I have to ask the obvious question: Who the hell cares?

There are three things at play here, I think.

1) What will this technology do. Knowing Sony, the footage that comes off these sensors will undoubtedly be excellent, and update Sony's highest end cameras, pushing digital cinema one step further. Yay!

2) The validation of 4K as a real deal. Yay!

3) The way Sony is presenting the above. This, in my opinion, is where I begin to take issue because it seems like they are really laying it on heavy with the marketing. (Yes, of course your 6K sensor does resolve more than your 4K sensor. And no, it doesn't have "100%" green in the same way that a striped or prism system has 100% green. Etc.) Boo!

 

[Sanjin Jukic]

As we all know well that only IQ (Image Quality) matters.

It looks to me that in a moment of launching new Sony 4K camera they could advertise it as a 4K camera

that has the highest digital cinema camera resolution and the most of RGB colors and more than RED EPIC 5K, Aaton Penelope Δ 4K, etc...

Also Sony could try out again to tell all these stories about pleasant skin tones colors, accurate blue skies,

low and available light best colors, etc,...only possible to get with a real 4K digital cinema camera by Sony.

 

[Lauri Kettunen]

I think that anybody who has a background in science agrees that it's quite an illusion to believe that by rotating the sensor one could increase the amount of captured information. Furthermore, as Graeme has pointed out in every possible way of wording, it's another matter of fact whether the demosaic algorithm makes the best use of the available data. If not, then adding the amount of sensor pixels obviously improves the outcome, but such a hardware solution tends also to be the most expensive choice to increase image quality. Interesting to discover that Sony has got itself to such a twist.

 

[Graeme Nattress]

Sony graphs even show they're reducing dynamic range by doing this.

Graeme

 

[I Bloom]

However what follows IS PURE SPECULATION AND SHOULD NOT BE TAKEN SERIOUSLY!

So it boils down to what you think the most statistically significant portion of a group of photo sites is. The mysterious green “diamond” is used to contribute 100% of its value (unaltered (ish)) to what will eventually become a corresponding pixel (that has twice the notional area of the green photo site). However this centre diamond only accounts for 50% of the area of the corresponding (notional pixel) area that it will become. The belief here is that the centre portion of a pixel will be more important than the rest of the pixel area. You could draw a 3d graph of central tendency centered on the green diamond to one sigma i.e 67-68% of the total limits, and this bell curve will fall just inside the green diamond. However this leaves the blue and red photo-sites bordering the green diamond to pull values from. These are at edges and beyond so that color accuracy for those corresponding regions of space are much lower. I have to admit it’s a weird piece of lateral thinking to get very quick values, one of which the green remains largely unaltered. So it really could be that they don’t de-Bayer, but rather grab a set of rather quick and dirty values, the green possibly having better measures of central tendency for the pixel it gives rise to. If this is the case it’s no wonder the marketing guys were completely lost and just liked the number 67.

The idea being we see green response much more than red or blue so we have a nice green and a slightly dodgy blue or red as we can get away with that.

This is what I would expect. Essentially coming straight out of the camera a sort of RAW image, I would expect the green to be artificially punchy and the red and blues to lack color/spatial accuracy by comparison. I think the way that these images would be graded would be different from images that result from conventional Bayer.

The whole 33 vs 67% has to be ignored as it really does not come into it. They absolutely did not explain the sampling or the “belief” system and mathematics that is behind it.

(I have things to do so I can’t comment much and I am a little burnt out on this topic).

Cheers

Eric

What you need to understand is basically this:

Q67 is about direct downsampling to RGB from bayer pattern in camera. It's possible that the orientation might make for a more efficient algorithm to do this. This is what Sony is saying. I believe Graeme feels it does not.

Bayer pattern, or Raw images, are about debayering, i.e. filling in missing color data for each photosite using a variety of algorithms. The debayered RGB image is then often downsampled to the actual image size. i.e. a 5K bayer image should yield a full resolution 4K RGB image. You can also directly go from raw to RGB in camera, though Sony seems to argue at a disadvantage.

We all agree (I think) that Q67 doesn't make sense for an actual debayer, because it would just add a 45 degree rotation to existing techniques, in addition to making border pixel calculations more complicated.

Red has made a pretty good case for not debayering the image in camera: keeping the image raw yields an immediate 1:3 compression advantage over RGB. Color correcting an image prior to downsampling and sharpening also yields a quality advantage. Sony will be hard pressed to convince me that compressing an RGB image that originated from a bayer pattern, is better quality or more efficient that direct raw compression.

So for me the debate is really about what is best algorithm to get from a bayer pattern to a somewhat smaller RGB image. How much does it matter in terms of quality? How much does it matter in terms of efficiency? How much oversampling is needed?

Another way you can phrase the question is to ask: If we compared a 4K 4:4:4 feed from an 5K Epic camera, to a 4K 4:4:4 feed from a future ?K Sony camera, which would we prefer. My guess is the downsampling won't be the most important element of that decision, dynamic range and color rendition probably will be.

People seem to really like the 1080 ProRes from the Alexa, even though to my eyes the debayering artifacts in that image are pretty obvious. The more you understand about debayering, the more you understand that there are certain aliasing artifacts and color fringing that can only be avoided with multi-pass, computationally expensive algorithms.

For me it all just speaks to the need for a best possible in camera downsampling to an additional codec on an Epic module. H264 is going to be really useful, but ProRes 4:4:4 or something similar would be better for post. I've been saying this since 07 and I just love to blow this vevuzela whenever I can

IBloom

 

[Philipe Ratton]

At this point, I have to ask the obvious question: Who the hell cares? If Red had gone with a Foveon type design, very few people here would even know the name Bayer as anything other than aspirin. I'll say it again: unless you're the engineer or the manufacturer, the only thing that counts is the images. Everything else is technical detail that ultimately means nothing to the user. When Sony's product is available, everyone can evaluate the images to their hearts content and come up with at least some meaningful opinions. Until then, let the engineers argue about things that are only important to engineers, just as cinematographers are usually the only ones who argue about things like film stocks. The rest of us can wait until there are images to look at. Playing numbers games is for engineers and mathematicians. Most people here, including me, are neither.

This.

 

[Lauri Kettunen]

... to put the idea in tangible words, say one shoots hand held from a car on a bumpy road and suddenly while shooting the most interesting event the car suddenly rolls to 45 degrees. To save what can be saved one decides to make a crop from the center of the raw image. But then one gets an idea, what about if instead of debayering and rotating the image the photo was created directly from the raw data which was rolled by 45 degrees.

Now, in such case would there be an algorithm that was better in this case? That is, if one went to Graeme saying, my camera was accidentally rolled by 45 degrees, please turn this accident to a golden moment by creating a new demosaic algorithm, would he be able to come up with a better one? If so, wouldn't it even be quite an akward thing, if the algorithm was dependent on the rolling angle? For, say one shoots light poles in a row and some of them were tilted by 45 degrees, and in such case the resolution of those poles were lower than those standing straight.

Summing up, AFAIC, the demosaic algorithm should not be sensitive to the angle. And now, as Graeme gets 3.2K out of 4K and 4K out of 5K, the obvious question is, where's the space for improvents? Of course, one may come up with a solution to get 4K out of 6K, but where's the beef?

 

[Bill Anderson]

From an image maker's point of view, one that previously demanded that I know a fair bit about film emulsions/developer combos, etc., I have found this discussion to be very informative, and revealing in unexpected ways. Thanks.

 

[Eric Lange]

What you need to understand is basically this:

Q67 is about direct downsampling to RGB from bayer pattern in camera. It's possible that the orientation might make for a more efficient algorithm to do this. This is what Sony is saying. I believe Graeme feels it does not.

Bayer pattern, or Raw images, are about debayering, i.e. filling in missing color data for each photosite using a variety of algorithms. The debayered RGB image is then often downsampled to the actual image size. i.e. a 5K bayer image should yield a full resolution 4K RGB image. You can also directly go from raw to RGB in camera, though Sony seems to argue at a disadvantage.

We all agree (I think) that Q67 doesn't make sense for an actual debayer, because it would just add a 45 degree rotation to existing techniques, in addition to making border pixel calculations more complicated.

Red has made a pretty good case for not debayering the image in camera: keeping the image raw yields an immediate 1:3 compression advantage over RGB. Color correcting an image prior to downsampling and sharpening also yields a quality advantage. Sony will be hard pressed to convince me that compressing an RGB image that originated from a bayer pattern, is better quality or more efficient that direct raw compression.

So for me the debate is really about what is best algorithm to get from a bayer pattern to a somewhat smaller RGB image. How much does it matter in terms of quality? How much does it matter in terms of efficiency? How much oversampling is needed?

Another way you can phrase the question is to ask: If we compared a 4K 4:4:4 feed from an 5K Epic camera, to a 4K 4:4:4 feed from a future ?K Sony camera, which would we prefer. My guess is the downsampling won't be the most important element of that decision, dynamic range and color rendition probably will be.

People seem to really like the 1080 ProRes from the Alexa, even though to my eyes the debayering artifacts in that image are pretty obvious. The more you understand about debayering, the more you understand that there are certain aliasing artifacts and color fringing that can only be avoided with multi-pass, computationally expensive algorithms.

For me it all just speaks to the need for a best possible in camera downsampling to an additional codec on an Epic module. H264 is going to be really useful, but ProRes 4:4:4 or something similar would be better for post. I've been saying this since 07 and I just love to blow this vevuzela whenever I can

IBloom

In principal I agree with what you are saying, but I think what SONY are doing has less and less to do with how WE de-bayer.

Now that I see what they may be going on about, it’s ludicrously simple.

There is no “heavy de-bayer” and its questionable if you could even call it a de-mosaic algorithm. It’s surprisingly direct and requires very little computation.



For one target pixel, take the value for G from the green diamond.
i.e. (g = G);

For the same target pixel, take the top and bottom adjoining blue photo site’s values and average them, i.e. (b1+b2/2) = B (for target pixel);

For same target pixel, take top and bottom adjoining red photo site’s values and average them,
i.e. (r1+r2/2)= R (for target pixel);

For next horizontal pixel, repeat same process with the next green diamond horizontally to the right;

And so on.

(that’s a bit of simplification as there’s a lot of signal conditioning etc. etc but the basic data flow and structure perhaps suffices)



This is very direct, very simple, ludicrously simple…

The Q67 is a misnomer, it would be better if they called SIGMA 1G, because that’s what it is.

After the images are graded I don’t think that you will see the difference between SONY 4k, film, and Red/EPIC.

The most noticeable difference you will see BY FAR, is the difference between a global shutter and a rolling one, as well as speed in processing output. .. And of course as many have said, ergonomics and price.

As I understand it there are slightly different cooling and heating effects between diagonally arranged photo sites than rectilinear ones.

Just to be clear there is no 100% equivalence with a true Green from a true RGB system.

I also see that there is spatial inefficiency as mentioned by Graeme (the difference between 1 and root 2).

Ta.

 

[Robin Balas]

This might be mentioned somewhere else, but what Sony is doing is quite similar to some old Fujifilm sensor design called "Super CCD" which come in several flavors which either extends DR or resolution. Please see http://www.dpreview.com/news/0301/03012202fujisuperccdsr.asp and do a Google search on Super CCD SR II
That definitely required de-bayering/mosaic'ing. It also did have visible issues with jaggies on diagonals when the increased resolution approach was used. Whats worth to mention is that the RGB pattern is identical on SONY's and Fuji's color filter.

MHO

 

[Eric Lange]

Just as a funny idea, you could put the SONY algorithm and method to the test (right now) just about.

1.

Using a CCD shoot various target images with the sensor and camera turned at 45 degrees. Given that the SONY pattern happens to be the same as the layout for a rotated Bayer. (A medium format camera with giant pixels and sharp lenses would be good for this to eliminate other factors, but not essential).

2.

You would have to write a sampling algorithm that reads, filters and processes the RAW image in the way they suggest, i.e. pull the green values directly and average the adjoining blue and red values for the corresponding target (rendered pixel) moving in a rectilinear way hopping ("horizontally") from green diamond to green diamond.

3.

You can end up with a 1k by 1k image that is rectified for normal display.

4.

Compare this with a rectilinear conventional square array of demosiac’d conventional Bayer pattern RAW image that has the same number of RGB photo elements as the SONY.

This is a good project for college student or out of work programmer/technician or or professional enthusiast with time on his hands. Probably not more than three days of programming, or if you’re really good it could be done in one day.

Initially I would predict that there would be a fair amount of tweaking to make the jerry rigged/home brew “Q67” produce imagery that is as good as a conventional de-Bayer.
Rotated rectilinear resolution targets would be fun to try out also.

Why wait?

(maybe a good project for Dan...??)

You could call it the “Q67 Homebrew Challenge”...