Komodo 12.2 stops of dynamic range in CineD test.

[M Harvey]

Not sure how to judge this particular dynamic range test. But on the bright (haha) side, one great thing about the Komodo vs. other cameras is that the goal posts and stop lights help you get the most out of the dynamic range you DO have. I've been shooting test footage outside in high contrast situations, and it's these tools that really make the difference.

 

[Phil Holland]

Same link Bastian posted, but linking their test video here.

https://www.repaire.net/forums/articles/testbox-red-komodo-comparatif-alexa-mini-c300-mark-iii.1095/?fbclid=IwAR0otL6hf3uSmHBoQVS9dW1SDa7IuA_u5ky0Fv8x 4i0r9NhpFtro5MvLk4k

Worth a translate and read, but mostly look at the footage. Decent side by sides.

 

[Gunther Machu]

Gentlemen,

this is my first post on reduser and I am the guy who is doing the CineD Lab tests. I am here to answer all your questions, and looking forward to a good technical discussion.

Phil has shared already our method (how we test) - thank you - and in the article I have clearly laid out the dilemma one has when trying to reduce DR to a single number.


https://www.cined.com/the-cinema5d-camera-lab-is-back-dynamic-range-tests/

a) Judging DR only from waveforms is a good method, but subjective, as some may see more stops than others and it doesn't tell you if all those stops are usable are not. USABLE is the term here!

b) The Xyla 21 chart has to be shot at 3200K - agreed. Nevertheless we purposely expand the RGB information to 5600K to see if all color channels are still there. In the french test shared by Bastien (previous page) you can clearly see the lacking color information on the top 2 stops of the KOMODO. Why, because Komodo uses an in - built highlight recovery mechanism (as the first RED camera). In the facebook thread where our lab results are discussed (RED KOMODO group), the man behind the IPP2 image pipeline himself, Graeme Nattress has confirmed this behaviour for the KOMODO, see attached screenshot.

c) IMATEST is the only objective way to mathematically calculate signal to noise numbers for each patch and hence dynamic range. Is it perfect - no. Because as Phil mentioned, flares can have an impact on the result, etc.. But it is a way to quantify and compare cameras to each other. We are conservative and use the Signal to noise ratio = 2 as our definition of usable (but give you both SNR = 1 and 2 values for full transparency).

d) Of course, in-camera or post noise reduction has an influence on the calculated signal to noise numbers. With noise reduction, the signal stays the same but noise is reduced, leading to a higher signal to noise value thus higher dynamic range. THe same happens by downsampling from 6K to 4K or to FullHD. IMATEST themselves warn against using noise reduction, and suggest to use RAW files as pure as possible, otherwise results will be distorted - as has been pointed out earlier in this thread as well. Thats why we give you pure values straight from the file with no post processing at native resolution.

e) on shooting the chart wrong: here is the download link for the R3D HQ file shooting our Xyla21 chart, judge yourself: https://we.tl/t-c1YhxlHqWj

f) and finally, we are also doing the latitude test, purposely over- and underexposing and bringing it back to see how usable the dynamic range is. To have a reference we choose a max. luma value of 60% for the subject's face. Which is hot for some cameras like the KOMODO but in the middle for others - the ALEXA for example has five usable stops above 60% Luma, the BMPCC6K has close to 4 stops, the C300MKII has 3 usable stops above 60% - that is just a matter how cameras distribute the stops across the range, which also depends on the ISO you choose e.g. for the KOMODO. And this test confirms that two stops above 60% at ISO800 the face cannot be recovered - that is exactly the stop from the waveform which has already clipped color channels but still looks good in the waveform (due to HL recovery - or soft roll off, however you want to call it).

Happy to answer any other questions you might have!

best regards, Gunther

 

[Bastien Tribalat]

Gentlemen,

this is my first post on reduser and I am the guy who is doing the CineD Lab tests. I am here to answer all your questions, and looking forward to a good technical discussion.

Phil has shared already our method (how we test) - thank you - and in the article I have clearly laid out the dilemma one has when trying to reduce DR to a single number.


https://www.cined.com/the-cinema5d-camera-lab-is-back-dynamic-range-tests/

a) Judging DR only from waveforms is a good method, but subjective, as some may see more stops than others and it doesn't tell you if all those stops are usable are not. USABLE is the term here!

b) The Xyla 21 chart has to be shot at 3200K - agreed. Nevertheless we purposely expand the RGB information to 5600K to see if all color channels are still there. In the french test shared by Bastien (previous page) you can clearly see the lacking color information on the top 2 stops of the KOMODO. Why, because Komodo uses an in - built highlight recovery mechanism (as the first RED camera). In the facebook thread where our lab results are discussed (RED KOMODO group), the man behind the IPP2 image pipeline himself, Graeme Nattress has confirmed this behaviour for the KOMODO, see attached screenshot.

c) IMATEST is the only objective way to mathematically calculate signal to noise numbers for each patch and hence dynamic range. Is it perfect - no. Because as Phil mentioned, flares can have an impact on the result, etc.. But it is a way to quantify and compare cameras to each other. We are conservative and use the Signal to noise ratio = 2 as our definition of usable (but give you both SNR = 1 and 2 values for full transparency).

d) Of course, in-camera or post noise reduction has an influence on the calculated signal to noise numbers. With noise reduction, the signal stays the same but noise is reduced, leading to a higher signal to noise value thus higher dynamic range. THe same happens by downsampling from 6K to 4K or to FullHD. IMATEST themselves warn against using noise reduction, and suggest to use RAW files as pure as possible, otherwise results will be distorted - as has been pointed out earlier in this thread as well. Thats why we give you pure values straight from the file with no post processing at native resolution.

e) on shooting the chart wrong: here is the download link for the R3D HQ file shooting our Xyla21 chart, judge yourself: https://we.tl/t-c1YhxlHqWj

f) and finally, we are also doing the latitude test, purposely over- and underexposing and bringing it back to see how usable the dynamic range is. To have a reference we choose a max. luma value of 60% for the subject's face. Which is hot for some cameras like the KOMODO but in the middle for others - the ALEXA for example has five usable stops above 60% Luma, the BMPCC6K has close to 4 stops, the C300MKII has 3 usable stops above 60% - that is just a matter how cameras distribute the stops across the range, which also depends on the ISO you choose e.g. for the KOMODO. And this test confirms that two stops above 60% at ISO800 the face cannot be recovered - that is exactly the stop from the waveform which has already clipped color channels but still looks good in the waveform (due to HL recovery - or soft roll off, however you want to call it).

Happy to answer any other questions you might have!

best regards, Gunther

Thank you for your post.
I didn't say this here but said it on Facebook (in Emmanuel Pampuri's Facebook Group "Geek Lumineux") : I am NOT saying that your testing was bad or not trustworthy and the french one is or vice versa. They are just two tests to get perspective on a specific matter.
And thank you so much for sharing the file I will test it myself just to get an opinion ON THE CAMERA, not on your method. ;-)

Take care.

 

[Christoffer Glans]

DR stops aside, the most interesting thing about all of this is the top stop of monochrome highlight. While I don’t think it’s much of a problem if you don’t screw up totally while shooting and that the Pocket 6K isn’t close to the color science of Komodo so its benefits in highlight stop quality aren’t enough for the rest of it, I don’t get why Komodo doesn’t have color in that part of the DR spectrum?

What is the reason for it? Is this “the magic” that enabled a global shutter to reach enough stops that previous global shutters did not? Is it built in or through the color science? Is it “fixable”?

I can’t really figure out the logic behind it?

 

[Bastien Tribalat]

DR stops aside, the most interesting thing about all of this is the top stop of monochrome highlight. While I don’t think it’s much of a problem if you don’t screw up totally while shooting and that the Pocket 6K isn’t close to the color science of Komodo so its benefits in highlight stop quality aren’t enough for the rest of it, I don’t get why Komodo doesn’t have color in that part of the DR spectrum?

What is the reason for it? Is this “the magic” that enabled a global shutter to reach enough stops that previous global shutters did not? Is it built in or through the color science? Is it “fixable”?

I can’t really figure out the logic behind it?

I'm in no way a sensor engineer but to me it doesn't seem to be related to the global shutter since on the french test the C300MKIII seems (at least to me) to have kinda the same characteristics.

 

[Michael Tiemann]

On the subject of noise reduction, I think it's perfectly reasonable to report numbers as long as the NR type and parameters are reported. 1-frame TNR in Resolve does a great job reducing shadow noise in RAW files. Not so much for mpeg-encoded files with macro blocking. Similarly, RED's Chroma Noise Reduction also works very well at calming things down and increasing SNR.

Almost all astrophotography photos use extensive NR, and it would be almost meaningless to try to evaluate the merits of a given imaging platform without consideration of how that platform supports (or prevents) the application of NR techniques essential to the craft.

Of course we don't expect testing orgs to run through every NR technique with every possible parameter setting that forces us to sift through 50,000 results to find the one that's most meaningful to us. But I don't think it's wrong to acknowledge when a very straightforward approach leads to a significant change in final results.

 

[Michael Tiemann]

Regarding waveform analysis, my benchmark is this: after white balancing the steps, if there is a distinct empty space below the step I count it as resolved DR. If there's a series of decreasing steps that all merge into the noise floor without visible gaps below, I count that as not resolved. If the bottom noise hairs overlap < 5% the top noise hairs of the next lower step, I consider that USEABLE because a key can adequately distinguish the two.

 

[Bastien Tribalat]

Here are my results after playing with Gunther's file.
IT IS NOT A SCIENTIFIC APPROACH.

Workflow is 6K timeline, Full Res PREMIUM decoding and that's it, everything else is specified.

Note : "Manual NR" = 3 frames/faster/medium and everything around 30-ish | SNR Enhanced, Small and again everything around 30-ish.

I repeat : it's just me playing with the file, nothing more.

 

[Gunther Machu]

DR stops aside, the most interesting thing about all of this is the top stop of monochrome highlight. While I don’t think it’s much of a problem if you don’t screw up totally while shooting and that the Pocket 6K isn’t close to the color science of Komodo so its benefits in highlight stop quality aren’t enough for the rest of it, I don’t get why Komodo doesn’t have color in that part of the DR spectrum?

What is the reason for it? Is this “the magic” that enabled a global shutter to reach enough stops that previous global shutters did not? Is it built in or through the color science? Is it “fixable”?

I can’t really figure out the logic behind it?

Dear Christoffer, what you are seeing is the highlight recovery algorithm at work - for this stop one, or probably two color channels are clipped already, and reconstructed - but to which color? The safe approach is grey. Thats it.

best regards, Gunther

 

[Björn Benckert]

Gentlemen,

this is my first post on reduser and I am the guy who is doing the CineD Lab tests. I am here to answer all your questions, and looking forward to a good technical discussion.

Phil has shared already our method (how we test) - thank you - and in the article I have clearly laid out the dilemma one has when trying to reduce DR to a single number.


https://www.cined.com/the-cinema5d-camera-lab-is-back-dynamic-range-tests/

a) Judging DR only from waveforms is a good method, but subjective, as some may see more stops than others and it doesn't tell you if all those stops are usable are not. USABLE is the term here!

b) The Xyla 21 chart has to be shot at 3200K - agreed. Nevertheless we purposely expand the RGB information to 5600K to see if all color channels are still there. In the french test shared by Bastien (previous page) you can clearly see the lacking color information on the top 2 stops of the KOMODO. Why, because Komodo uses an in - built highlight recovery mechanism (as the first RED camera). In the facebook thread where our lab results are discussed (RED KOMODO group), the man behind the IPP2 image pipeline himself, Graeme Nattress has confirmed this behaviour for the KOMODO, see attached screenshot.

c) IMATEST is the only objective way to mathematically calculate signal to noise numbers for each patch and hence dynamic range. Is it perfect - no. Because as Phil mentioned, flares can have an impact on the result, etc.. But it is a way to quantify and compare cameras to each other. We are conservative and use the Signal to noise ratio = 2 as our definition of usable (but give you both SNR = 1 and 2 values for full transparency).

d) Of course, in-camera or post noise reduction has an influence on the calculated signal to noise numbers. With noise reduction, the signal stays the same but noise is reduced, leading to a higher signal to noise value thus higher dynamic range. THe same happens by downsampling from 6K to 4K or to FullHD. IMATEST themselves warn against using noise reduction, and suggest to use RAW files as pure as possible, otherwise results will be distorted - as has been pointed out earlier in this thread as well. Thats why we give you pure values straight from the file with no post processing at native resolution.

e) on shooting the chart wrong: here is the download link for the R3D HQ file shooting our Xyla21 chart, judge yourself: https://we.tl/t-c1YhxlHqWj

f) and finally, we are also doing the latitude test, purposely over- and underexposing and bringing it back to see how usable the dynamic range is. To have a reference we choose a max. luma value of 60% for the subject's face. Which is hot for some cameras like the KOMODO but in the middle for others - the ALEXA for example has five usable stops above 60% Luma, the BMPCC6K has close to 4 stops, the C300MKII has 3 usable stops above 60% - that is just a matter how cameras distribute the stops across the range, which also depends on the ISO you choose e.g. for the KOMODO. And this test confirms that two stops above 60% at ISO800 the face cannot be recovered - that is exactly the stop from the waveform which has already clipped color channels but still looks good in the waveform (due to HL recovery - or soft roll off, however you want to call it).

Happy to answer any other questions you might have!

best regards, Gunther

Regarding the screen dump and the quotation from Graeme. As I understand it, Komodo does not really do anything different from previous Red cameras except it can display IPP2 to its full potential straight out of the SDI. Where as other red cameras footage only can do the same off camera / In resolve or what have you. How red cameras display the raw files onboard is of little or no relevance as I see it. The raw file is what matters and the color science that comes with it.

 

[Alex Stolpakov]

f) and finally, we are also doing the latitude test, purposely over- and underexposing and bringing it back to see how usable the dynamic range is. To have a reference we choose a max. luma value of 60% for the subject's face. Which is hot for some cameras like the KOMODO but in the middle for others - the ALEXA for example has five usable stops above 60% Luma, the BMPCC6K has close to 4 stops, the C300MKII has 3 usable stops above 60% - that is just a matter how cameras distribute the stops across the range, which also depends on the ISO you choose e.g. for the KOMODO. And this test confirms that two stops above 60% at ISO800 the face cannot be recovered - that is exactly the stop from the waveform which has already clipped color channels but still looks good in

Do i read this correct, that hitting with the same amount of light different sensors one with native 250 ISO and other 1600 ISO you will have different amount of recoverable stops above middle gray on different cameras?

 

[Curtis boggs]

Sidenote Phil have you ever even used a bmpcc6k? Wondering what youd say or think.

I did, .. P6k was my B cam to my Helium before I got the Komodo.

In comparing the images in real work shooting situations I quickly sold the P6K.
The P6k clipped highlight MUCH sooner then Komodo and had a harsher highlight roll off.

Have NO idea what DR numbers are or even give a F%$K about those numbers and tests, ..
I only know what I get in final images.

C

 

[AndreeMarkefors]

Do i read this correct, that hitting with the same amount of light different sensors one with native 250 ISO and other 1600 ISO you will have different amount of recoverable stops above middle gray on different cameras?

High ISO shooting is essentially "stopping down exposure to save highlight information, and lifting shadows to compensate".

Consider the exposure triangle: ISO, shutter speed and aperture. In your example the first two are fixed (assuming traditional video settings): ISO 250 and 1/48 and ISO 1600 and 1/48 respectively.

You stand in a room and shoot against a window. The interior is the subject and the lighting is such that you can't just stop both cameras down to keep all highlight information intact.

A camera ISO of 1600 will allow you to close your aperture more than an ISO of 250 and still reach the same exposure value. Stopping down means "protecting/saving highlights".

On the sensor itself, everything is linear at capture. But your file setting of ISO 1600 basically lifts the shadow areas of the linear data more than a setting of ISO 250. Since the ISO 1600 setting allowed you to close your aperture more, you will have captured more highlight information and you will have more stops available above middle gray. Since you lift the shadows more, the price you pay is nosier stops below middle gray.

If this discussion was concerning different settings on the same camera, it's pretty simple. This way of playing with the ISO is often referred to as 'rating' the scene or camera at a certain ISO. It makes sense when you know your camera well and want to maximise its strengths.

But if you say "different sensors"... well... many might just assume the same applies. But in reality, I guess we're talking about different brands, different gamma curves and color science. The basic principle of the exposure triangle is the same, but there are more moving parts.

A manufacturer going after as much DR as possible is likely to 'rate' the camera at a higher ISO, which in turn leads to allowing a slightly noisier file, while winning some headroom in the highlights.

 

[Alex Stolpakov]

If this discussion was concerning different settings on the same camera, it's pretty simple. This way of playing with the ISO is often referred to as 'rating' the scene or camera at a certain ISO. It makes sense when you know your camera well and want to maximise its strengths.

But if you say "different sensors"... well... many might just assume the same applies. But in reality, I guess we're talking about different brands, different gamma curves and color science. The basic principle of the exposure triangle is the same, but there are more moving parts.

A manufacturer going after as much DR as possible is likely to 'rate' the camera at a higher ISO, which in turn leads to allowing a slightly noisier file, while winning some headroom in the highlights.

Sure nobody speaks about the numbers you can dial in the menu of the camera.
They were checking different cameras, didn't they?
Checking highlight recovery without knowing real sensitivity of the sensor and it's noise floor is a bit of a ride on high horses.
How would you know that you not oversaturating the sensor and just placing middle grey two stops over.

PS Its written in the manual about rating of the camera is not a good answer for tests.
And in real life who uses meter? Usually lets set the camera at 800ISO and push some sensible amount of exposure before clipping. So its common to rate camera very differently when spoken on forums.

 

[AndreeMarkefors]

How would you know that you not oversaturating the sensor and just placing middle grey two stops over.

From memory: I'd expect them to set the camera at ISO 800 (the most used default) and then expose for the grey card or skins tones (with compensation). The over/under shots were done with a person in frame as well as a chart to explore against.
From there it's easy to go up and down, of course.

 

[Blair S. Paulsen]

Even IF Komodo has only 12.2 stops of useable DR, that should be plenty for all but corner cases. Even with HDR, how many displays can actually show more than 10 stops? Yes, it's nice to have latitude in post - but unless the DP misses exposure by 2 stops or more, which is hardly the camera's fault, no worries.

IAC, as long as the CineD testing methodology treats all cameras/sensors reasonably equitably, then the data point created has some value. That said, placing too much faith in tests you didn't do yourself is inherently risky. Just sayin'

Cheers - #19

 

[Mike P.]

Do i read this correct, that hitting with the same amount of light different sensors one with native 250 ISO and other 1600 ISO you will have different amount of recoverable stops above middle gray on different cameras?

LOL! Yeah, pretty much. Bare in mind, RED is the outlier though, as they have a high-end DR deficit at the recommended ISO compared to others.

RED's recommended ISO seems to be whatever is the highest/brightest rating that is still considered noise-free, regardless of how the DR is distributed above/below mid-grey at that ISO (on average ISO800, but it varies depending on the sensor/OLPF). The rest - Sony, BMD, Canon, Panny, and Arri (which still does it best even though ALEV3 is 11 years old) - seem to do the opposite; find out what ISO has an even-ish over/under, and then recommend that as their "base ISO"... (FYI: that's all presumption; take with a huge grain of salt... Maybe it's custom mapped independent of ISO, but if so, it's a wonder why would RED choose to go with such a below-mid-grey weighted DR distribution... aka GioScope Chip11 of 16 = mid, instead of Chip 8~9).

Judging by those scopes, it looks like you'd have to expose Komodo at ~ISO3200 to have a more even over/under. But in the French footage, it looked like green-shift and noise were too much at 3200 (without NR, anyway). That said, Nick mentioned Komodo's mids being much cleaner than other REDs, which would certainly help when pushing those ISO limits to capture more/accurate high-end stops.

 

[Alex Stolpakov]

From memory: I'd expect them to set the camera at ISO 800 (the most used default) and then expose for the grey card or skins tones (with compensation). The over/under shots were done with a person in frame as well as a chart to explore against.
From there it's easy to go up and down, of course.

So as I expected scientific approach coming to reading online manual of 3 pages and make assumptions.
Can we check 2 stops under For Komodo against two stops over for Alexa? Maybe the noise floor is comparable? And the highlight retention will be favourable other way around. Those test are not one plus one.
Which transforms they were using for those 60%?
How they decide to expose for grey card and which method they choose to check exposure, some inbuilt tool?
Why not to hit sensors with the same amount of light? And move accordingly exposure wise.

 

[Mike P.]

So as I expected scientific approach coming to reading online manual of 3 pages and make assumptions.
Can we check 2 stops under For Komodo against two stops over for Alexa? Maybe the noise floor is comparable? And the highlight retention will be favourable over way around. Those test are not one plus one.
Which transforms they were using for those 60%?
How they decide to expose for grey card and which method they choose to check exposure, some inbuilt tool?
Why not to hit sensors with the same amount of light? And move accordingly exposure wise.

Maybe Gunther can answer that (I presume they used a light meter or scopes, as that'd be camera independent). The problem with changing the parameters to better suit one camera over the other is that it would be even harder to maintain consistency/objectivity. Also if you need to jump through hoops just to get to par, what's the point?

Does anyone have Xylas of Dragon, Helium, Gemini, Monstro? I would love to see the RGB scopes of those sensors (hoping their highlight recovery isn't as aggressive as Komodo's and the IPP2 highlight recovery is limited to the first/top stop.)