Komodo 12.2 stops of dynamic range in CineD test.

[Alex Stolpakov]

Maybe Gunther can answer that (I presume they used a light meter or scopes, as that'd be camera independent). The problem with changing the parameters to better suit one camera over the other is that it would be even harder to maintain consistency/objectivity. Also if you need to jump through hoops just to get to par, what's the point?

Does anyone have Xylas of Dragon, Helium, Gemini, Monstro? I would love to see the RGB scopes of those sensors (hoping their highlight recovery isn't as aggressive as Komodo's and the IPP2 highlight recovery is limited to the first/top stop.)

Point is not to injure Gunther or dismiss him of his work.
Consistency I find in all the ISOs shoot with a lot of under and over exposure for each one. It's nothing unobjective in it.
For myself I shoot normal tests with a person, some coloured light, some black, light bulb - nothing fancy, except a lot of exposure pairs.

PS Scopes are coming through camera track? Or how else?

 

[Gunther Machu]

Maybe Gunther can answer that (I presume they used a light meter or scopes, as that'd be camera independent). The problem with changing the parameters to better suit one camera over the other is that it would be even harder to maintain consistency/objectivity. Also if you need to jump through hoops just to get to par, what's the point?

Does anyone have Xylas of Dragon, Helium, Gemini, Monstro? I would love to see the RGB scopes of those sensors (hoping their highlight recovery isn't as aggressive as Komodo's and the IPP2 highlight recovery is limited to the first/top stop.)

The 60% luma is with Log3G10. But that is just a point of reference to not confuse our readers. Of course different cameras distribute the stops differently. It does not really matter where you start over / under. Another point of reference could be 18% gray (33 IRE). That would have been close to the stop labelled as "5" in our waveform. From this reference you could overexpose skin by 4 stops and underexpose 3 stops - gving you the same 7 stops of latitude.

 

[Christoffer Glans]

Judging by those scopes, it looks like you'd have to expose Komodo at ~ISO3200 to have a more even over/under. But in the French footage, it looked like green-shift and noise were too much at 3200 (without NR, anyway). That said, Nick mentioned Komodo's mids being much cleaner than other REDs, which would certainly help when pushing those ISO limits to capture more/accurate high-end stops.

Here's the scopes after supersampling down to 4K at ISO3200 and using high-quality NR and WB at 3200K.

 

[Mike P.]

The 60% luma is with Log3G10. But that is just a point of reference to not confuse our readers. Of course different cameras distribute the stops differently. It does not really matter where you start over / under. Another point of reference could be 18% gray (33 IRE). That would have been close to the stop labelled as "5" in our waveform. From this reference you could overexpose skin by 4 stops and underexpose 3 stops - gving you the same 7 stops of latitude.

Thanks, Gunther. And yeah, I understand. The question was how you were measuring the 60% luma (external scopes [if so, by the file or a video feed], light meter)? End of the day, I don't think it would really impact the findings enough to be a concern.

There are pros/cons to exposing the skin at mid-grey, though that's certainly something you could do to protect a bit more of the highs (by exposing at a pushed ISO... 1600 or 3200), but it runs the risk of noise/exaggerated green-tint in the lower-mids... Speaking of which:

Here's the scopes after supersampling down to 4K at ISO3200 and using high-quality NR and WB at 3200K.

That looks a lot better from a DR distribution standpoint, but having to apply NR isn't what I'd consider a solution so much as a workaround (as it's one of the slower things to render at 4k+). Applying an S-curve and/or (most) generic LUTs to that kind of over/under should have decent heel and shoulder roll-off (contrast overall usually sits better too, imo). And you obviously have more push/pull room to play with in the mids/meat of the image (presuming noise isn't bad or noise reduction isn't that destructive).

 

[Björn Benckert]

Thanks, Gunther. And yeah, I understand. The question was how you were measuring the 60% luma (external scopes [if so, by the file or a video feed], light meter)? End of the day, I don't think it would really impact the findings enough to be a concern.

There are pros/cons to exposing the skin at mid-grey, though that's certainly something you could do to protect a bit more of the highs (by exposing at a pushed ISO... 1600 or 3200), but it runs the risk of noise/exaggerated green-tint in the lower-mids... Speaking of which:



That looks a lot better from a DR distribution standpoint, but having to apply NR isn't what I'd consider a solution so much as a workaround (as it's one of the slower things to render at 4k+). Applying an S-curve and/or (most) generic LUTs to that kind of over/under should have decent heel and shoulder roll-off (contrast overall usually sits better too, imo). And you obviously have more push/pull room to play with in the mids/meat of the image (presuming noise isn't bad or noise reduction isn't that destructive).

If capturing a raw image then that does not mean nothing should be done to the image in post, it's quite the opposite. Downsampling helps latitude, so does noise reduction, explain why such tools should not be used? Or what parameters do you consider such that can be used to improve the image, only those that are added destructively in camera?

 

[Brian Roberts]

Interesting discussion going on here.

So, if it is true that Komodo has the upper two highlight stops reconstructed to grayscale, can we also assume that these stops are included in the goal post readings?

If so, when exposing should we not be backing off a bit from that upper goal post to ensure our highlights retain all color information vs going into that "recovery" region?

 

[Tomasz Nizielski]

I think we need a comparison of cameras in a real-world scenario, some high-contrast scene, e.g. an actress in a room with a window. Cameras that I would like to see: Alexa Mini, Red Helium, Red Komodo, Pocket 6K, Ursa Pro G2, C70/C300. For me it will be better than DR chart interpretation.

 

[Alex Lubensky]

I suppose rating a camera to a 60% in LOG is wrong as a starting point. It’s 60% in REC 709 profiles, be it RG RC4, IPP2 BT1886 etc.

 

[Bill Ravens]

https://youtu.be/Xo17maQkAXI

 

[Michael Tiemann]

I suppose rating a camera to a 60% in LOG is wrong as a starting point. It’s 60% in REC 709 profiles, be it RG RC4, IPP2 BT1886 etc.

Indeed. Some arbitrary decisions are easy to justify. Others...not so much.

 

[Mike P.]

Downsampling helps latitude, so does noise reduction, explain why such tools should not be used? Or what parameters do you consider such that can be used to improve the image, only those that are added destructively in camera?

Don't get me wrong, NR should certainly be used when needed, but not relied on in order to merely achieve (or worse, artificially inflate) the advertised DR numbers of a digital cinema camera's raw capture. Similar to those monochrome-ish highlight recovered stops.

Your two examples - downscaling vs NR - is a good one; both reduce perceived noise, but downscaling 1.) doesn't really require additional render time and more importantly 2.) is non-destructive, while NR to dig stops out of the noise floor requires 4x the render time and *is* destructive... Those are pretty big, objectively negative, caveats to relying on NR. And not only is the amount of acceptable NR subjective, but it can be applied to every camera in order to increase their shadow DR numbers, not just Komodo. So it adds huge (and subjective) variables to an already contentious comparison.

Personally, I would think those stops (both the NR recovered lows *and* monochrome-ish recovered highs) should be considered the "+" in advertised numbers, not part of the whole/RGB/full stops... but that's just me (and Arri... and I think BMD... butt I digress.)

I suppose rating a camera to a 60% in LOG is wrong as a starting point. It’s 60% in REC 709 profiles, be it RG RC4, IPP2 BT1886 etc.

True. But I was under the impression that all the cameras in this comparison were done the same way, so the over/under exposure recovery comparison would still hold (which is why I've referenced the Komodo +1 vs P6k +4 on that skintone highlight). Conversely, regardless of where RED's "log 60%" compares to it's "709/normalized 60%", it'd still be a trade-off of either noise being introduced earlier or highlights clipped sooner (I thought that's why Gunther was saying it's just a point or reference). Basically taking a block from the bottom and putting it on top.

...Though, maybe it'd give a more even over/under at a more palatable ISO. e.g. ISO1600 instead of 3200, which would be a much nicer/cleaner starting point (maybe enough to skip that NR step!)

So, if it is true that Komodo has the upper two highlight stops reconstructed to grayscale, can we also assume that these stops are included in the goal post readings?

That's an important question (since, as you mentioned, if we know it is, we can just protect the highlights more to compensate). Since Komodo is the first time the HR recovery has been done in-camera, I would think that means it counts them in the goal post... but the stop lights would be popping before that, so you'd know... Also there's Jarred's new fandangled channel clipping exposure tool, too.

Have they put GioScope in Komodo yet? If so, it'd be easy enough to treat chip 14 as the last stop (instead of 16).

 

[Jon Dishler]

You all are aware he updated and corrected the article about 6 days ago? Interesting quote in there from Graeme.

 

[Mike P.]

You all are aware he updated and corrected the article about 6 days ago? Interesting quote in there from Graeme.

LOL. Yeah we know, but the results didn't change, so the concerns/questions (about those top two reconstructed-stops and comparatively poor highlight performance because of them) remain.

Similarly, questions about metering (to protect), exposing tricks (to compensate), and how Dragon/Helium/Gemini/Monstro/others HR compare are also being discussed.

 

[Michael Tiemann]

Don't get me wrong, NR should certainly be used when needed, but not relied on in order to merely achieve (or worse, artificially inflate) the advertised DR numbers of a digital cinema camera's raw capture. Similar to those monochrome-ish highlight recovered stops.

You might have missed this the first time around:

On the subject of noise reduction, I think it's perfectly reasonable to report numbers as long as the NR type and parameters are reported. 1-frame TNR in Resolve does a great job reducing shadow noise in RAW files. Not so much for mpeg-encoded files with macro blocking. Similarly, RED's Chroma Noise Reduction also works very well at calming things down and increasing SNR.

Almost all astrophotography photos use extensive NR, and it would be almost meaningless to try to evaluate the merits of a given imaging platform without consideration of how that platform supports (or prevents) the application of NR techniques essential to the craft.

Of course we don't expect testing orgs to run through every NR technique with every possible parameter setting that forces us to sift through 50,000 results to find the one that's most meaningful to us. But I don't think it's wrong to acknowledge when a very straightforward approach leads to a significant change in final results.

You continue:

Your two examples - downscaling vs NR - is a good one; both reduce perceived noise, but downscaling 1.) doesn't really require additional render time and more importantly 2.) is non-destructive, while NR to dig stops out of the noise floor requires 4x the render time and *is* destructive... Those are pretty big, objectively negative, caveats to relying on NR. And not only is the amount of acceptable NR subjective, but it can be applied to every camera in order to increase their shadow DR numbers, not just Komodo. So it adds huge (and subjective) variables to an already contentious comparison.

Downscaling *is* destructive. That doesn't mean it should be dismissed out of hand, nor that it should be accepted without comment. As long as the system parameters are sane and properly reported, they provide information for people who want to know "what can the system deliver?" and "what needs to be done to get that sort of performance?" Again, in some realms, advanced image processing is part of the craft. That's OK, as long as you have the tools, time, and treasure needed to get it done.

 

[Mike P.]

Downscaling *is* destructive. That doesn't mean it should be dismissed out of hand, nor that it should be accepted without comment.

Practically/comparatively it's not though; everything from sharpness and cleanliness to DR and colour is perceptibly improved by downscaling. It's a far more objective process, too. Besides, you always have to downscale with cameras that are 4k+ (it can't be avoided).

Pretty sure combining data to fit into a smaller spot (downscaling) is objectively better than *removing* data in an attempt to remove artifacts (NR). Or no?

As long as the system parameters are sane and properly reported, they provide information for people who want to know "what can the system deliver?" and "what needs to be done to get that sort of performance?" Again, in some realms, advanced image processing is part of the craft. That's OK, as long as you have the tools, time, and treasure needed to get it done.

Yeah, I don't disagree with that as a bonus, but it shouldn't be necessary to hit the advertised numbers... Conversely, I think RED's Chroma Noise Reduction & Flashing Pixel Removal is perfectly reasonable; it's imperceptibly destructive, r3d specific, and I think done at the debayer stage.

The problem (for me) is that the application of NR in post is variable. Not only is the strength subjective to taste (before it seems plasticky/overly done), but each individual sensor/codec/camera might see the most benefit from different settings... Render time aside (which shouldn't be understated), maybe p6k could do 5-frame TNR before it looks bad, netting you two additional stops in the low-end, but Komodo only 1-frame/0.5-stop (or vice versa)... And it also ignores that a lot of raw/digital cinema cameras don't need additional NR to achieve their advertised DR or have an even highlight range.

Hence why I lean more in the direction that the (potential) benefits of NR should be considered a bonus - again, part of the "+" like the mono-ish highlight stops should be.

 

[Gunther Machu]

The problem (for me) is that the application of NR in post is variable. Not only is the strength subjective to taste (before it seems plasticky/overly done), but each individual sensor/codec/camera might see the most benefit from different settings... Render time aside (which shouldn't be understated), maybe p6k could do 5-frame TNR before it looks bad, netting you two additional stops in the low-end, but Komodo only 1-frame/0.5-stop (or vice versa)... And it also ignores that a lot of raw/digital cinema cameras don't need additional NR to achieve their advertised DR or have an even highlight range.

Hence why I lean more in the direction that the (potential) benefits of NR should be considered a bonus - again, part of the "+" like the mono-ish highlight stops should be.

I see it exactly the same way, thats why we measure "pure" RAW files (without NR) and also do not downscale, we measure DR at native resolution. Also, no highlight recovery is counted in the DR measurements (as color information is not fully there any more). Again, its all about "usable" stops.

 

[paulcurtis]

Incoming 2p worth...

This is serving to highlight how people see different tests and there's nothing right or wrong here. Personally i think it's difficult to boil performance down to 1 number because there is so much detail in that number lost.

CineD chooses to publish a number where they feel they are showing usable stops. And by useable they mean that each of the chips has it's own colour information in. This means that none of the RGB channels has clipped. You can see from all the charts previously in the French(?) test that the Alexa appears to show colour information higher up. The point about using WB to separate the RGB colours reveals if those colours are identical (meaning that they have been mathematically generated because the underlying channel clipped) or not.

Personally i disagree with this, and i think any test should really show a number of figures. The chart range (the highest figure), the colour range (the cined figure) and maybe one with highlights in (which i would call usable, taking noise floor into account).

I do also disagree with not doing NR. With the same data there's at least another stop, stop and half in the shadows. NR is part of almost every camera whether it's onboard or in post. And i think highlight reconstruction should be factored in because you cannot get an image without it being in there and also in 99% of uses rolling to a solid colour is what you want.

But there's nothing wrong with the test that was done (revised one!) IMHO.

Unless as Phil says there are better ways of standardising highlight and shadow points that come with the chart - because we don't really know where the clip point is.

cheers
Paul

 

[Björn Benckert]

Practically/comparatively it's not though; everything from sharpness and cleanliness to DR and colour is perceptibly improved by downscaling. It's a far more objective process, too. Besides, you always have to downscale with cameras that are 4k+ (it can't be avoided).

Pretty sure combining data to fit into a smaller spot (downscaling) is objectively better than *removing* data in an attempt to remove artifacts (NR). Or no?



Yeah, I don't disagree with that as a bonus, but it shouldn't be necessary to hit the advertised numbers... Conversely, I think RED's Chroma Noise Reduction & Flashing Pixel Removal is perfectly reasonable; it's imperceptibly destructive, r3d specific, and I think done at the debayer stage.

The problem (for me) is that the application of NR in post is variable. Not only is the strength subjective to taste (before it seems plasticky/overly done), but each individual sensor/codec/camera might see the most benefit from different settings... Render time aside (which shouldn't be understated), maybe p6k could do 5-frame TNR before it looks bad, netting you two additional stops in the low-end, but Komodo only 1-frame/0.5-stop (or vice versa)... And it also ignores that a lot of raw/digital cinema cameras don't need additional NR to achieve their advertised DR or have an even highlight range.

Hence why I lean more in the direction that the (potential) benefits of NR should be considered a bonus - again, part of the "+" like the mono-ish highlight stops should be.

Ok so what about shooting 8k, do signal cleaning within 4 pixels and then downsample to 4k... Now such signal has not "lost" or been treated destructivly compared to a camera that just captured 4k at sensor level. On the conterary the 8k downscaled to 4k benefit heavily with almost four times better guess work for the debayering process and the noise reduction done has not really caused any lost any info compared to the straight 4k capture, on the conterary.

If comparing cameras you kind of need to bring them to a common ground before comparing, which is difficult so yes, it´s hardly ever just cameras that is being compared but also the post process, or the in camera baked process versus what could be done in post or what have you.

 

[Alex Stolpakov]

So WHY it's a secret how exposure was measured? Was it the same for different cameras or not?
It's raw, because of that we expose as we want is very controversial statement.
Exposuring 50D and 500T with the same exposure will favour one or other depending which was taken as a reference if it was printed with standard printer lights.

 

[Mike P.]

Ok so what about shooting 8k, do signal cleaning within 4 pixels and then downsample to 4k... Now such signal has not "lost" or been treated destructivly compared to a camera that just captured 4k at sensor level. On the conterary the 8k downscaled to 4k benefit heavily with almost four times better guess work for the debayering process and the noise reduction done has not really caused any lost any info compared to the straight 4k capture, on the conterary.

If comparing cameras you kind of need to bring them to a common ground before comparing, which is difficult so yes, it´s hardly ever just cameras that is being compared but also the post process, or the in camera baked process versus what could be done in post or what have you.

I guess this isn't about Komodo anymore, but the same logic applies, imo. It's more about comparing/testing against the advertised numbers in this case, anyway. Bear in mind, I'm not trying to convince any one, just noting that it'd be gnarly if you have to jump through hoops in post merely to meet the claimed specs (regardless of if NR is part of your workflow or not).

As for your example, is it an actual 8k camera or a 4k one? No doubt there'll be (substantial) benefits with even the mildest noise reduction with such a large downscale (more so than 6k Komodo), but RED doesn't spec/advertise their cameras as '16+ stop *4k* cameras from 6k~8k sensors when post NR is applied'. Hence, whatever advantages downscaling/NR provides is still more of a bonus, in the same way that 8k allows for 4k reframing/image stabilization is a bonus (you wouldn't typically rely on it; you'd change the lens or use a gimbal/steadicam/dolly when possible).

Forgive this exaggerated/hyperbolic anecdote to demonstrate the NR point further, but it's easier/faster to render a sky replacement on baked 4k footage than it is to use NR on 6k~8k raw footage; would we factor that in to the "usable" DR numbers of a camera that shoots baked 4k (to merely hit its advertised numbers, no less)? No, of course not; cause it's post.

Seems most here happily accept applying post-NR as a means to get closer to the advertised specs (which is fine). I'm just more inclined not to, partly because of how much RED's cost compared to similarly specced cameras (ironically some of which include software with decent post-NR, but don't need it to hit their DR spec), partly because it looks like a bigger delta in this particular Komodo test, and partly because I've seen how these hacks/tricks can fall short with my own (non-Komodo) RED footage.

And yes, it's difficult to have an apples to apples comparison, but I'd think adding more sensor/codec/resolution/model specific post tweaks and tricks (none of which work on all cameras in equal measure), is not the way you're supposed to go about objectively testing capture capabilities (again imo; feel free to disagree).

Anyway, we're drifting from Komodo, which is where these concerns stemmed from (the internal Highlight Recovery that looks to be a hefty two highlight stops, which I presume/hope is quite a bit more aggressive than other RED sensors... or at least Dragon, which was designed/measured/metered pre-IPP2 HR at a claimed 16+). It seems this thread is diverging into two issues; one about NR's effect on testing for DR (even though it was brought up as a fix for pushing Komodo's ISO higher for a more even over/under), and the other being Komodo's monochrome-ish highlight recovery and how to best deal with it (so it has the least adverse impact on the useable DR).

Unfortunately, the last couple of days/pages haven't provided any new answers/info, so we're kinda talking in circles now.