Welcome to our community

Be a part of something great, join today!

  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

OLPF vs. MTF: why the mathematical limits of resolution make 12K the new 4K

Michael Tiemann

Well-known member
Joined
Oct 4, 2008
Messages
2,216
Reaction score
1
Points
38
Website
manifoldrecording.com
I came across some interesting maths curated by Frans van den Bergh regarding theoretical and practical MTF of digital sensors. The article that really caught my attention is here, but he's been writing and coding about the topic for about 5 years and his MFMapper toolkit has been posted to GitHub.

The long and short of it is this: if one considers a signal of repeating lines (such as a fully resolved 100 lp/mm chart), one can analyze how such a pattern will hit a sensor of a specific geometry and implementation, such as a 5 micron bayer pattern assumed for DRAGON. A 5 micron pixel pitch means 200 pixels per mm, which means that without any OLPF, the pattern might be perfectly resolved (with the white line perfectly hitting one row or column of pixels and the black line perfectly hitting an adjacent row or column). Alternatively, the pattern might be at a 45 degree angle, or it might be offset by 2.5 microns, in which case the white line half-fills one row, column, or diagonal of pixels and half-fills the adjacent row, column, or diagonal, leading to a perfect middle gray with no resolution at all. Moreover, if the pattern move relative to the pixel array, aliasing will be observed as the alternating white and black lines go from the center of one pixel pair to the next.

Just as the Nyquist limit governs the absolute limit of signal information that can be derived from a given (in this case, spatial) sampling rate, there are some other hard limits as to what can and cannot be resolved across a range of signal/sensor configurations. For example, at every multiple of the pixel pitch, the resolution drops to zero because there are an equal number of white and black lines that average out to zero. And above the 1:1 ratio, higher (spatial) frequency information "wraps around" and appears as if it is really lower (spatial) frequency information. Such ringing can make it appear that something is being resolved, when actually it's just noise.

The long and short of the analysis, however, is that even at 1/2 the spatial frequency of the array (two pixels per line or four pixels per line pair), the aliasing problem is still very severe: the sensor readings will vary between 100% MTF (1, 1, 0, 0) and 50% MTF (.5, 1, .5, 0). Applying an "optimal" gaussian point spread function to the input signal lowers the MTF from the 100%-50% range to a much smaller one, albeit below 20%, which is not very well resolved. However, at 1/3rd of the spatial frequency of the array, the post OLPF image recovers to about 50%, as shown in this chart:

mtf.png


Thus, the math tells us that if we want 50% MTF and decent anti-aliasing, the DRAGON sensor can only resolve a 100 lp/mm signal to 33 lp/mm, not 50, and definitely not 100. And that's before we consider the question of how the Bayer filter might further confound our equations. And obviously an incoming signal with only 50 lp/mm resolution is going to have a lower (though not terribly much lower) final resolution than a 100 lp/mm input.

I'm posting this because I think it's interesting that an 6K DRAGON sensor should properly be expected to resolve only 1024 line pairs across its width, and only a 60 mm sensor could be expected to resolve true 4K information (2000 line pairs across its width).

Comments? Criticisms?
 
So your conclusion is not specific to Dragon but all CMOS sensors?
I allways thought RED's color science pushed the resolution capabilities to 70% of total amount of pixels.
I think Graeme knows best about his friend Nyquist.

Pat
 
So your conclusion is not specific to Dragon but all CMOS sensors?

That's how I read this paper. The Bayer sensor can only make it worse (because a pair of red or blue lines would only have half the pixels to play with).

I allways thought RED's color science pushed the resolution capabilities to 70% of total amount of pixels.
I think Graeme knows best about his friend Nyquist.

Pat

I can believe the number may be relative to "baseline" resolution, but the point of the article is that baseline isn't what you think it is...it's somewhere between 2x and 3x less than you might have thought. Or, if you already accepted the 2:1 discount, somewhere between 0% and 50% less than what you thought.
 
I didnt work through all the math, but this is a perfect reason why I like red's "smaller" sensor in that it starts to bump up against the diffraction limit of the lens f-stop. Also pure red and blue's lines are a bit worse, since there's a 8 gaps around each sensor. Best way I know threading this needle is going to a larger sensor, kind of pushing home RED's exact strategy (i.e. keep current sensor size constant and start going up to larger sensors). To me btw this isn't just esoteric exercise, but when I shoot clothes & paintings (i.e. 90% of my shots have "threads" in them) - if I don't see the weave of the threads I instantly know its "fake". I think this is even worse in VR since you can "walk" up to the projected images and they just blur out or go "icky" patterns as you get close.
 
...RED's exact strategy

I don't know if this is a strategy or a economical necessity to use their existing tech. But I'm sure they took the pixel size into the design equation. Arri took the route of more DR/color response so bigger pixels, RED the resolution route and there fore smaller pixels size to cover s35mm with 5k pixels. Sony with the A7s serie and it's huge pixels covering 4k with a 24x36mm sensor is clearly a good example for this design choice.

RED has showned some resolution charts from their Dragon sensor resolving more than 1024 lines I think... I can't remember where though...

Edit.

The Epic-MX one from 2010!

http://reduser.net/forum/showthread.php?47529-EPIC-and-Alexa
 
I think I follow what the article is saying, in that there's a trend in DSLRs to pull the OLPF and not low-pass filter the image ahead of the sensor, thus the conclusion "So there you have it. A sensor without an AA filter can only really attain a significant increase in resolution at large apertures, where diffraction is not attenuating the contrast at higher frequencies too strongly. Think f/5.6 or larger apertures." In that diffraction works similarly to an OLPF to low-pass-filter the image, so it's only when your lens has low diffraction can you see a resolution increase from removing the OLPF. But in doing so, you'll open yourself up to aliasing issues, which are of course, unwanted, leading to the author's preference "Personally, I would take the sensor with the AA filter".

Where I'm not following you Michael is when you say "The long and short of the analysis, however, is that even at 1/2 the spatial frequency of the array (two pixels per line or four pixels per line pair), the aliasing problem is still very severe:" - because if you have two pixels per line you are way below Nyqvist and thus you won't get aliasing. (assuming we're talking a monochromatic sensor array here). The math for a Bayer pattern CFA is more complex - the partial sampling on green gives (with a naive approach a root2 factor in that you should be able to achieve a resolution of the post-interpolated green at around 0.707 of linear resolution of a monochromatic array - i.e. a 5.6k bayer array is equivalent to a 4k monochromatic array for luma resolution). Better interpolation approaches can get that higher, to say around 80% meaning the 4k mono array would have similar measured resolution to the 5k bayer array in luma. That would mean for a Dragon 6k sensor, you should have meaningful resolution out to around 4.8k, and beyond that you could run into aliasing, but usually the lens and scene would limit any real issues.

For MTF50 which is a great measure of perceived sharpness you'll get >3k with a good lens on a Dragon 6k sensor. A pixel pitch of 0.005 gives you 200 pixels per mm, and given the above math we should be able to resolve at MTF50 100px/mm (3k) and certainly 4k albeit at a lower MTF (but still visible, not a blur out to pure grey) and only get aliasing with an input >4.8k, and because of the lens MTF and OLPF, that will be at such a low level, it's not normally visible (unless shooting specific charts to show it).

Adding in the colour components of red and blue into a Bayer CFA does complexity this a bit, but again, unless you're shooting a specific chart to show up the potential issues, you don't generally see any issues, making it, for the very most part, a theoretical problem but almost irrelevant in practice.

Graeme
 
I think I follow what the article is saying, in that there's a trend in DSLRs to pull the OLPF and not low-pass filter the image ahead of the sensor, thus the conclusion "So there you have it. A sensor without an AA filter can only really attain a significant increase in resolution at large apertures, where diffraction is not attenuating the contrast at higher frequencies too strongly. Think f/5.6 or larger apertures." In that diffraction works similarly to an OLPF to low-pass-filter the image, so it's only when your lens has low diffraction can you see a resolution increase from removing the OLPF. But in doing so, you'll open yourself up to aliasing issues, which are of course, unwanted, leading to the author's preference "Personally, I would take the sensor with the AA filter".

Indeed. In several of his papers he stresses that if you don't filter on the way in, you are way, way behind in what you can correct with digital signal processing later.

Where I'm not following you Michael is when you say "The long and short of the analysis, however, is that even at 1/2 the spatial frequency of the array (two pixels per line or four pixels per line pair), the aliasing problem is still very severe:" - because if you have two pixels per line you are way below Nyqvist and thus you won't get aliasing. (assuming we're talking a monochromatic sensor array here).

I think this explains the first disconnect--and it's because I used the wrong terminology. Once you are below Nyqvist you don't get the modulo kind of aliasing--you are correct--but you do get a condition where the contrast ratios of the pixel pattern can vary from 100% to 50%, which, depending how it moves, can create the appearance of aliasing. But technically it's due to the MTF varying across a wide range, not because of actual aliasing. And it may look like aliasing, but it's not.

[...]

For MTF50 which is a great measure of perceived sharpness you'll get >3k with a good lens on a Dragon 6k sensor. A pixel pitch of 0.005 gives you 200 pixels per mm, and given the above math we should be able to resolve at MTF50 100px/mm (3k) and certainly 4k albeit at a lower MTF (but still visible, not a blur out to pure grey) and only get aliasing with an input >4.8k, and because of the lens MTF and OLPF, that will be at such a low level, it's not normally visible (unless shooting specific charts to show it).

I think this is the second disconnect. From what I understand in the article, a monochrome DRAGON with minimal OLPF can resolve 100 px/mm at MTF50 on a 200 px/mm array, but only if you accept that same system delivering a higher MTF when the lines and pixels get lucky and perhaps a lower MTF when the lens resolution isn't quite up to the task. WIth the circular ring target, I would expect 100 px/mm (which is 50 lp/mm) to have inconsistent contrast readings as one goes around the circle (thus "seeing" many different line/pixel intersection conditions). If I understand the article correctly, the proper amount of OLPF point spread forces 6K down to 2K px/mm to achieve *consistent* (and thus perfectly smooth and round) circles.

Now, it may be that the noise characteristics of the sensor and the mathematical "randomness" introduced by a circular target as opposed to a linear one is such that these lucky/unlucky ratios wind up in a much narrower range than 50%-100%. Perhaps they really range more like 66%-44%, with the majority living in the 60%-50% range. In which case the full power of the theoretical point spread function isn't necessary and 3K is good spatial resolution expectation for a 6K sensor after all.

Thanks for weighing in!
 
But it's a sampled system, so looking at the results as pixels is going to make you think you're seeing pixel brightness vary as lines in the scene cross over pixel boundaries and move. However, that's because you've forgotten the other part of any sampled system - the reconstruction. Although we represent point samples as square pixels with area, they're really point samples and should be displayed with appropriate reconstruction filtering, which is essentially ignored because of the angle of view, eye acuity, diffraction etc act as our reconstruction filtering. If you're close enough to see pixels, you're too close to be getting a reasonable approximation of reconstruction.

Graeme
 
But it's a sampled system, so looking at the results as pixels is going to make you think you're seeing pixel brightness vary as lines in the scene cross over pixel boundaries and move. However, that's because you've forgotten the other part of any sampled system - the reconstruction. Although we represent point samples as square pixels with area, they're really point samples and should be displayed with appropriate reconstruction filtering, which is essentially ignored because of the angle of view, eye acuity, diffraction etc act as our reconstruction filtering. If you're close enough to see pixels, you're too close to be getting a reasonable approximation of reconstruction.

Graeme
One weird thing, i've love my monochrome epic ... and have noticed as things progress the dragon epic and even the weapon feels like the monochrome epic. I'm wondering what in "color" theory did you learn while working on the monochrome epic that was applied to the dragon color science, there's something there but I can't figure it out.
 
I strongly believe that before we even reach 8K as anywhere near a standard resolution, we will see a new sensor technology emerge. Debayer sensors has been around since the 70's, we are just building on top of an old structure, not advancing technology from the ground up. Just by the tech alone, it's very crude that we need more resolution than what we deliver, just to get all the colors and sharpness right. I don't mind having a little head room for cropping and stabilizing, but I think we need something new soon.
 
I don't mind having a little head room for cropping and stabilizing, but I think we need something new soon.

Just had a few well exposed Red GS shots and Alexa RAW shots on my Flame. Can't say anything negative about Bayer arrays. One has to understand the simple math of dropping approx. a third of the given RAW resolution for the desired RGB resolution but then there is plenty of information in the RGB picture that allows for keys that are easier to pull than we had with good exposed 35mm.

So I question the "need anytime soon".


---

Personally I find my Dragon's OLPF (skin tone) pretty strong. The OLPF works well with exceptional sharp glass such as Master Primes (use them often for TVCs) but is a bit soft with vintage glass such as my Zeiss Super Speeds which are designed with an in-build OLPF that gets a bit weaker at T 2.8 - 4 again but stronger from 5.6 on. For those lenses a less stronger OLPF would be meaningful. Since Red follows the policy "choices are good" I wonder wether such OLPFs are in the makes. This subject pops up every here and then.


---

I have a new sharpen tool in my box that works very subtle and without introducing rings (Resolve's sharpen does that badly) so we can leave things as they are. But nonetheless.


Hans
 
But it's a sampled system, so looking at the results as pixels is going to make you think you're seeing pixel brightness vary as lines in the scene cross over pixel boundaries and move. However, that's because you've forgotten the other part of any sampled system - the reconstruction. Although we represent point samples as square pixels with area, they're really point samples and should be displayed with appropriate reconstruction filtering, which is essentially ignored because of the angle of view, eye acuity, diffraction etc act as our reconstruction filtering. If you're close enough to see pixels, you're too close to be getting a reasonable approximation of reconstruction.

Graeme

Graeme, thanks again for weighing in. I think I agree with you, and I think that your answer agrees with mine. Namely, if we allow a reconstruction implementation of a 3x3 matrix to "resolve" an effective minimum feature size of 1.5 pixels then we can present a max high-frequency MTF as

(0, 0.25, 0)
(0.25, 1, 0.25)
(0, 0.25, 0)

When this pattern shifts 1/2 pixel across the sensor (or display) it becomes (with its neighbor's contribution)

(0, 0.125, 0.125)
(0.25, 0.625, 0.625)
(0, 0.125, 0.125)

Shifted again it becomes

(0, 0, 0.25)
(0.25, 0.25, 1)
(0, 0, 0.25)

So we don't see the individual pixels, we see the reconstruction of a 1.5 pixel feature size using a 3x3 matrix. Which effectively means we are cutting our end-state resolvability by 50%. The only question that remains for me is whether this 50% reconstruction hit can be 100% folded into the other 50% hit, meaning that we really should expect to resolve > 1500 concentric circles across the width of a 6K DRAGON sensor, or whether they take from one another and its really more like 1000 concentric circles. Which may also be related to what is a reasonable MTF number for high-frequency resolution.

I'm certainly not arguing that movies should be judged by the MTFs we can measure at the pixel or 3x3 or 5x5 matrix level or by counting the number of circles we can resolve in a bullseye. I'm just trying to get some reference points for where I am when I am down in the pixels, looking at sharpness, and trying to evaluate what I can get vs. what I can mathematically expect from the system.
 
What reconstruction hit? Sure, if you sit way too far away from the display you can't see the detail it produces. Look at reconstruction filtering in digital audio - it doesn't turn a 44.1khz sample rate (capable of producing an audio signal with a 20khz bandwidth) into an analogue signal with a 10khz bandwidth. Thinking about a pixel matrix is not what I'm getting at with reconstruction. I'm addressing your concern that "but you do get a condition where the contrast ratios of the pixel pattern can vary from 100% to 50%, which, depending how it moves, can create the appearance of aliasing".

MTF is useful. MTF50 is a very good proxy for overall image sharpness. I've measured that for numerous camera systems. Looking at limiting resolution (do you pick MTF 5, or 10 perhaps) is perhaps less useful, not least as it's a bit harder to measure accurately, but it does tell you more of what the system is capable of when you look at a full MTF plot.

Graeme
 
What reconstruction hit? Sure, if you sit way too far away from the display you can't see the detail it produces. Look at reconstruction filtering in digital audio - it doesn't turn a 44.1khz sample rate (capable of producing an audio signal with a 20khz bandwidth) into an analogue signal with a 10khz bandwidth.

No, but my 51 year old ears do that...

Thinking about a pixel matrix is not what I'm getting at with reconstruction. I'm addressing your concern that "but you do get a condition where the contrast ratios of the pixel pattern can vary from 100% to 50%, which, depending how it moves, can create the appearance of aliasing".

MTF is useful. MTF50 is a very good proxy for overall image sharpness. I've measured that for numerous camera systems. Looking at limiting resolution (do you pick MTF 5, or 10 perhaps) is perhaps less useful, not least as it's a bit harder to measure accurately, but it does tell you more of what the system is capable of when you look at a full MTF plot.

Graeme

Roger that. I'll keep studying. Thanks!
 
Well, I think my ears are heading that way also.....

Graeme
 
... Look at reconstruction filtering in digital audio - it doesn't turn a 44.1khz sample rate (capable of producing an audio signal with a 20khz bandwidth) into an analogue signal with a 10khz bandwidth. Thinking about a pixel matrix is not what I'm getting at with reconstruction. ...

Graeme
This is a great way of explaining it, I wasn't thinking this way. My new way of thinking on this is RED is transforming a physical wave space to a wavelet space (r3d) via a integration over a state space. Also since red is in a time domain inside that state space, they can do almost magical things during that transform(much in the same way that the audio guys are doing some real interesting things on their A/D, only RED is dealing both a time and space wave as they integrate within the shutter time).
 
There's real benefits to understanding how sampling works in different areas. Many people, even those that deal with digital audio on a daily basis has some fundamental mis-understandings on sampling theory, why and where over-sampling is used in audio recording, and probably the least understood aspect is the reconstruction filtering. I've seen people look at a "line graph" join-the-dots representation of a sample, and declare that it looks nothing like the source (because they're looking at frequencies up near Nyqvist) and declare that sampling theory is bunk! They ignore that sampling theory requires a band-width-limited signal, and also that reconstruction filtering is a necessary component of the process, an AFAIK (audio people step in here and help me...) no audio software will show you the waveform of what a sample will look like post reconstruction.

I remember learning a lot of this for myself the hard way, by getting it wrong, and then taking an old analogue oscilloscope to my mac's audio output and looking at the post-reconstruction signal rather than the digital stair-step or "join the dots" representation of it.

Graeme
 
I seem to remember Bob Stuart talking about improving the response in bandwidth limit systems just below the cut-off frequency.

+) Specifically avoiding ringing : MQA ref : http://www.digitalaudioreview.net/2015/05/munich-high-end-2015-is-meridian-mqa-the-new-dsd/

+) From the archives for people to 'see' what reconstruction looks like in audio : http://www.hit.bme.hu/~papay/edu/DSP/ZEN/DAC86.htm

Bayer with Secondary colours?

On a different (random?) thought - I wondered if reconstruction of colour would still be possible if the Bayer pattern was replaced with secondary colours : ie

Yellow. Cyan
Magenta. Yellow

=

RG. BG
RB. RG

If 66% of the light is collected rather than 33% for Bayer pattern, would this reduce noise & increase contrast?
... or is this a non-starter as resolution would be greatly reduced to determine colour absence?

AJ
 
Back
Top