Camera shootout 2011

[Mark Pugh]

Why drag out the episodes? in the interest of getting useful information out quickly, or in dragging people back to Zacuto's website month after month.
It really would be useful to release the white paper with the test results.

 

[Graeme Nattress]

We want to simply add our data into the knowledge base of what is known about all these cameras.

The problem being here that without the white paper on how the measurements were achieved, we are not getting knowledge just data. It is through interpretation of data that we gain knowledge and without any information on how the data was collected we cannot properly interpret it.

The second issue, and again, it's more for the SCCE than you is that figures were presented at NAB that were later revised, but no attempt that I know of has occurred to inform those who saw the incorrect figures at NAB that they were indeed incorrect and have been updated.

The third issue is the overall reliability of any figures presented. Adam Wilt's post where he points out the errors in resolution figures that show measured camera resolutions much greater than the number of pixels on the cameras sensors, (or in the case of the RED, greater resolution than is actually achievable) are enough to show that some of the presented figures are wrong.

All this is compounded by the lack of the white paper.

Doing good tests is hard - doing it accurately across multiple cameras is very hard. Testing multiple parameters across many cameras is nigh on impossible to achieve without some errors creeping in.

I think for the most part the discussion here has been quite reasonable. And if you're brave enough to attempt a multi-camera test, you've got to be brave enough to answer the questions. It's great that you're here Scott to answer the questions! It's just a pity you've not been given the backup to answer them for us.

Graeme

 

[Roberto Lequeux]

or ascribing to conspiracy theories, etc.

I made light of this earlier. But I wasn't trying to fuel the fire. That wasn't my intention. I was just pointing at the elephant in the room with tounge in cheek.

 

[Jared Caldwell]

Agreed with Graeme here. I'm glad you're here on the thread. It is easy to bristle a little bit on both sides of the test, and I appreciate your presence and fielding our questions and concerns. It's unfortunate that it is hard to continue the discussion without the white papers, but at least we can get a better idea of what's going on with your posts.

 

[David Mullen ASC]

I suspect if you handed all these cameras to engineers and scientists, not working cinematographers, and had them tested for DR, color, resolution, etc. you would find some mild differences in results -- so I don't need to ascribe negative motivations to cinematographers, who are not engineers and scientists, when their test results vary even more wildly from each other. I even suspect that if I personally did the comparison tests myself twice, one week apart, I may find some slight differences in results (I certainly had a hard time in my college chemistry class getting consistent & repeatable results when I mixed chemicals...)

The other thing to remember is that shooting these sorts of big evaluation tests require a lot of time & effort on those involved, a lot of elements come together at the last minute, a lot of the work is done on a volunteer basis, so I'm tempted to say that anyone who feels they can do better (and maybe they can, who knows) should do it themselves and post their results.

I will also say that pointing out that so-and-so works for "x" company or owns "y" camera and therefore cannot be trusted to work unbiased on a comparison test is ridiculous... you're never going to find people who are completely free of connections to one product or another short of recruiting elementary school children to conduct the test.

The only reason I bring this up is that this furor over a test where a Red camera did less well than expected or desired keeps coming up over and over again -- it goes back to the beginning of Red as a product. Crying foul, implying corruption or incompetence was involved, etc. just comes off like we are in a schoolyard and that the average age of a Red shooter is under 16 years old. This isn't a sporting event, we don't have to overturn cars and burn them when our side "loses". For those of you that rationally pointed out errors that can be corrected in the future, and/or are waiting for the white paper to make final judgements as to what really happened, I applaud you. Even better would be to try and get involved next year in Zacuto's testing.

Ultimately, the best solution to countering inaccuracies is with more facts. All you need are a couple of independent comparison tests of DR between the Alexa and Red One (and Epic) that can be published online and linked to whenever someone brings up some test result you feel is inaccurate. It's one thing to say some result is wrong, it is much better to simply SHOW how it is wrong with examples.

 

[David Mullen ASC]

This article compares the Alexa to the Epic (not the Red One):
http://provideocoalition.com/index.php/freshdv/story/epic_vs_alexa_dynamic_range/

It finds only a .5-stop difference in DR between the Alexa and Epic, and if you throw away more of the noise at the bottom in the Alexa image, there really isn't a difference in DR it seems.

I did dig up Alfonso Parra's old articles about testing the Red One and Alexa (separately, more than a year apart I believe) and his original Red One test said 12-stops of captured DR and his article on Alexa said 14-stops, so that's a 2-stop difference, not far from the 2.2-stop difference that the Zacuto test claims. I'm not saying that this is accurate, only that it seems possible to test and get these results without trying resorting to sabotage.

 

[Sam Eilertsen]

REDuser's own Gunleik Groven found a 1.8-stop difference between MX and Alexa in his evaluation.

 

[Peter Dmitriyev]

Mr. Lynch (Scott) can you please answer one important question then on the minds of many here that is adding fuel to the 'conspiracy' theories. Why is it that the Alexa tests show marked differences in setup than how all other cameras were shot? It seems evident that the Alexa was shot differently or on a different day with different setup than all the other cameras. In the underexposure test the lamp is turned a completely different way in the frame than all other camps. In the over exposure tests there are key highlights (on the couch) missing that are present in all other cams and there is strange halos around the sun outside the window that is not present in MX or film.
What accounts for the moving around of lights and props for the Alexa's test? Was there a problem that caused Alexa to be shot on a completley different day that required resetting all the sets?

 

[Peter Moretti]

The problem being here that without the white paper on how the measurements were achieved, we are not getting knowledge just data. It is through interpretation of data that we gain knowledge and without any information on how the data was collected we cannot properly interpret it.

The second issue, and again, it's more for the SCCE than you is that figures were presented at NAB that were later revised, but no attempt that I know of has occurred to inform those who saw the incorrect figures at NAB that they were indeed incorrect and have been updated.

The third issue is the overall reliability of any figures presented. Adam Wilt's post where he points out the errors in resolution figures that show measured camera resolutions much greater than the number of pixels on the cameras sensors, (or in the case of the RED, greater resolution than is actually achievable) are enough to show that some of the presented figures are wrong.

All this is compounded by the lack of the white paper.

Doing good tests is hard - doing it accurately across multiple cameras is very hard. Testing multiple parameters across many cameras is nigh on impossible to achieve without some errors creeping in.

I think for the most part the discussion here has been quite reasonable. And if you're brave enough to attempt a multi-camera test, you've got to be brave enough to answer the questions. It's great that you're here Scott to answer the questions! It's just a pity you've not been given the backup to answer them for us.

Graeme

Graeme,

With all respect, there is a "little holier than tho" creeping in here. When Geoff Boyle presented his test that showed about a two stop DR advantage for the Alexa, he was summarily slammed and the thread was soon closed. He explained his methodology completely. It was simple and repeatable, providing you could get your hands on the cameras. IIRC, Jim said that Red would do their own version of the test and post the results b/c Geoff's results were wrong. Well we haven't seen anything yet.

I'm NOT trying to say that Geoff's test was any more accurate than other tests showing under a stop advantage for the Arri. I'm saying that I think the Zacuto documentary actually made the Red look very good, so slamming it makes questionable sense to me. But if you're going to criticize someones test for not providing supporting details, well Geoff's test--which did provide full methodology--got slammed too; and it was never countered.

BTW, I remember an earlier comparison presented here by LightIron between the Red and F35. What was not mentioned up front was that an S-Log LUT wasn't applied to the attached Sony images. So of course they looked like crap, b/c they're NOT supposed to be viewed w/o a LUT. But the test was still trumpeted by Red as clear victory.

With love comes bias, it's unavoidable. And the people at Red love what they do.

 

[Peter Moretti]

This article compares the Alexa to the Epic (not the Red One):
http://provideocoalition.com/index.php/freshdv/story/epic_vs_alexa_dynamic_range/

It finds only a .5-stop difference in DR between the Alexa and Epic, and if you throw away more of the noise at the bottom in the Alexa image, there really isn't a difference in DR it seems.

I did dig up Alfonso Parra's old articles about testing the Red One and Alexa (separately, more than a year apart I believe) and his original Red One test said 12-stops of captured DR and his article on Alexa said 14-stops, so that's a 2-stop difference, not far from the 2.2-stop difference that the Zacuto test claims. I'm not saying that this is accurate, only that it seems possible to test and get these results without trying resorting to sabotage.

David, I think no matter how you read that article, the most likely difference is one stop, maybe just .75.

"I hesitate to say that’s a usable stop of shadow detail, so would conservatively rate the Alexa on this chart at 15 full stops, with maybe an additional noisy half-stop available in the shadows."

"So on my curves-abused chart I’d conservatively rate the Epic at 14 stops with maybe an additional noisy 1/4 stop available in the highlights."

 

[Graeme Nattress]

Ah, but Peter, you'd expect a red bias here though, wouldn't you And if you read my comments they're all about bogus resolution measurements that actually rate the RED One higher than it should be, and I'm saying those numbers are too high for reality. But because we know some figures are wrong, the rest could be suspect too, and without any document describing the process, we're left with all this discussion and very little knowledge. I actually think the methods of measurement are almost more important than the numbers themselves! The only argument about DR is that it's much harder to measure than resolution, and even more politically charged, so to put out DR measurements to the .1 of a stop (when the charts that appear to have been used are not able to give .1 stop precision but allow only for discrete steps) is obviously going to cause discussion.

Graeme

 

[Scott Lynch]

Was there a problem that caused Alexa to be shot on a completley different day that required resetting all the sets?

To be honest, I don't actually know. Some of the other cameras have slight differences in the position of props and/or shadows as well. We had 12 cameras we were testing and the testing for these scenes happened over two consecutive days. Perhaps a bulb went out, or something was bumped, a light wasn't turned on, I wish I could tell you. In the Overexposure scene my guess is a light or flag was bumped from the left side of the frame after the Alexa shot the scene. For the Underexposure scene someone may have bumped the table with the lamp, but these are just guesses. I think the Alexa was the first to shoot the Overexposure scene, but I think it shot the Underexposure scene 3rd or 4th.

To give you an idea of the timespan testing happened over 2 weekends that were a month apart. We shot the first weekend in February and the first weekend in March. There were prep-day/lighting days on the Friday before the weekend and then we would start shooting Saturday and wrap late Sunday.

The first phase of testing shot three Backlit Illuminator charts, a series of 6 Front-lit charts, A Still life, Skin Tones (with actors on black), and three motion artifacts tests. The second phase had the Underexposure scene, the over exposure scene, and a low-light scene, and some re-shooting of the backlit and frontlit charts. The reason we reshot some of the charts from phase one was we wanted to test a production Sony F3, we had a beta version in phase one and a production version in phase two. There was concern that the beta version of the F3 may preform differently (to my knowledge it didn't). We also reshot the charts using the ALEXA's ARRIRAW which was released to us right before Phase 2. We also wanted to retest the F35 with the 12bit processing.

-Scott

 

[Robert Knoll]

The marketing folks at Zacuto must be doing a lot of high fives these days. Why? The goal of any website is to drive customers to their site. PERIOD. Really, why else would Zacuto have done this in a episodic manner or at all? Do they like us .... probably not so much but they love sales and marketers love statistics as in "hits".
I can't take this too seriously and I hope not too many of us do. Interesting conversations sparked absolutely. Will change my life or the way I view a product....not a chance.

Have a Red day!

 

[Martin Stevens]

I, for one, am quite glad that Scott Lynch is posting here.

Thank you Scott. And thank you for trying to clear things up.

 

[Eki Halkka]

REDuser's own Gunleik Groven found a 1.8-stop difference between MX and Alexa in his evaluation.

I did some testing with his stuff (I originally posted this elsewhere, but thought it might be of interest here too)...

I took the original files provided here (r3d for Red, a dpx for Alexa):
http://www.gunleik.com/tests/alexmx/RED1_MX_and_Arri_Alexa/Matching.html

After playing around a little, i found Gunleik's match pretty good: using 2 stops more ND, with otherwise the same exposure settings, he got pretty close to the same sensor saturation in Red as in Alexa. Alexa clips just a tiny bit earlier than red.

I then proceeded to After Effects and tried to adjust Red's RAW developing settings to more or less match the Alexa's log output. I can give the exact details if you wish, but the main things for a rougher match are: White balance to 4800 with 20 tint, Color version 2, ISO 1280, Contrast -40, Color space REDColor2, Gamma Curve REDlogFilm. In addition to this, i made small adjustments to the per channel gain/gammas, as well as a subtle curve adjustment. The After Effects (CS 5.5) project is here, you will need to download Gunleik's files from the earlier link:

http://eki.pp.fi/digivideo/RED/AlexaMatch/RedToAlexa.zip

The result was a pretty good - but not perfect - match between the two. All following links will be in the order of 1. Alexa, 2. Red MX (matched), 3. Checkerboard comparision of the two:

http://eki.pp.fi/digivideo/RED/AlexaMatch/Alexa.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_to_Alexa.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_Alexa_checkerboard.jpg

I then added an adjustment layer, and grossly increased the contrast of the above, crunching blacks so that only the highlights in the sky are visible - this was to check that the exposure was indeed similar on both, and that the difference between the cameras is NOT in the highlights, as long as they are exposed similarly. The result was very, very close:

http://eki.pp.fi/digivideo/RED/AlexaMatch/Alexa_Highlights.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_Highlights.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_Alexa_Highlights_checkerboard.jpg

On another adjustment layer, i made the opposite of the above, a high contrast adjustment burning everything to white except for the darkest shadows. This is where the differences start to show - Alexa is indeed the winner here, Red is noisier and "mushier". The mushiness likely also partly depends on the compression settings used on the Red, i do not know what the settings were in this case. As far as the noise level goes, it's clearly there, but i'm not qualified to quantify it in stops, or decibels. The color differences (red tint on Red etc.) are due to my match not being perfect, not due to inherent differences in the cameras.

http://eki.pp.fi/digivideo/RED/AlexaMatch/Alexa_Shadows.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_Shadows.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_Alexa_Shadows_checkerboard.jpg

Albeit the noise differences, both cameras actually did unbelievably well - in vast majority of cases, this kind of extreme color adjustments simply do not get made, both cameras can deliver a great final image even in high contrast scenes like this.

...and when it comes to Epic, it should be even better than these cameras, with it's HDRx giving multiple stops more latitude.

PS: Here's how the Red image looks like at factory default settings:
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX_factory_default.jpg

PS2: And here's one way the shot might have been color corrected in the real life:

http://eki.pp.fi/digivideo/RED/AlexaMatch/Ccor_Alexa.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/Ccor_RED1MX.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/Ccor_RED1MX_Alexa_checkerboard.jpg

PS3: There was a Red shot exposed two stops higher than the above examples in Gunleik's materials - it's noise characteristics in the shadows look pretty similar to the Alexa. So the range difference of roughly two stops between R1MX and Alexa sounds indeed plausible.

http://eki.pp.fi/digivideo/RED/AlexaMatch/Alexa_Shadows.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX+2stop_Shadows.jpg
http://eki.pp.fi/digivideo/RED/AlexaMatch/RED1MX+2stop_Alexa_Shadows_checkerboard.jpg

 

[Steve Dent]

Sounds plausible. 12-stops of latitude for Red one MX is still amazing, but I'd be curious to know how Red arrived at 13.5 stops shortly after the MX sensor came out.

 

[Eki Halkka]

Sounds plausible. 12-stops of latitude for Red one MX is still amazing, but I'd be curious to know how Red arrived at 13.5 stops shortly after the MX sensor came out.

My guess is that the sensor is 16 bit - and the difference comes from 16 bit processing (Epic) vs. 12 bit processing (Red One) the sensor output. I could be wrong tough.

 

[Ash Worth]

Given that RED didn't want to be involved in the tests I think it's a lot to ask / demand the test info asap. It's gracious of Scott to actually be here and answer questions.

I really hope the more critical members here read all of David Mullens posts in this section, what a gentleman, and then think about what else influences the decision to hire a DP with a RED camera because I would much rather hire David with a just a 5D then a DP with a RED and the attitude so often present in these discussions.

Producers hire people they can work with not how much DR their camera has.

 

[Timothy Eaton]

Given that RED didn't want to be involved in the tests I think it's a lot to ask / demand the test info asap. It's gracious of Scott to actually be here and answer questions.

I really hope the more critical members here read all of David Mullens posts in this section, what a gentleman, and then think about what else influences the decision to hire a DP with a RED camera because I would much rather hire David with a just a 5D then a DP with a RED and the attitude so often present in these discussions.

Producers hire people they can work with not how much DR their camera has.

Ash, David is indeed a gentleman, but so are many others who were among those asking for more information from the SCCE group. Graeme is among those who believe more information would be useful.

On any list of this size that allows people from all over the world to post, people with different backgrounds, different levels of experience, and different opinions, there are going to be some loose cannons. I don't believe the loose cannons represent the Red community any more than some of the cranky people who write letters to the New York Times represent New York. This is a pretty lively forum. I mostly take it with a grain of salt, sometimes objecting when I think users have crossed a line. For the most part, though, it's good stuff.

I didn't observe anyone "demanding" a white paper. Some made strong requests for more information, because we believe that any camera test is only as good as the methodology used, along with the data and how it is interpreted. It shouldn't be difficult to release the information. If they did the test properly, every step of the procedure would have been documented on the spot, with settings and observations recorded. Zacuto managed to create a slick video in a relatively short period of time, and it would have been easer to deliver the data than create the video. If, however, they didn't record all the information as they worked, that's another issue.

I don't know whether you read all the posts about people making a living from their Red cameras, but it's a significant factor. Nobody is asserting that producers are scouring the web to find cameras with the greatest DR. The problem is that a widely disseminated video claiming that Alexa has substantially superior DR (in contradiction to some earlier tests), and featuring a bar chart incremented in tenths of a stop, can make it more difficult to exclaim the benefits of using Red. Opinions have a way of transforming themselves into facts, and hearsay can become legend. How Red cameras are viewed in the film community is very important to those who own and use them. For some, it can relate directly to their income. So it seems only reasonable to ask the people who conducted the evaluation to produce the data. Perfectly fair, yes?

One more note. While it is axiomatic that every DP should conduct his/her own tests, my experience has been that many do not. Given tight schedules, lack of resources, and limited availability of cameras, many DPs simply rely on hearsay, or only what they might read in a cinema magazine, and/or tests like this "shootout." It's unfortunate, but real. I cannot count the number of times I have heard information, which I know to be suspect based on my own tests, bandied about with certitude by members of my profession. Filmmakers are only human, after all.

As a D.P., I try to conduct myself as a gentleman with producers, directors, and crew. If for one reason or another beyond my control, however, Red cameras develop an iffy reputation with respect to certain important characteristics, it can make my job and my desire to shoot Red more problematic. I think this is the genesis of the strong emotion with respect to the "shootout." Hope this helps.

Tim Eaton

 

[Scott Lynch]

I've been talking to Jack Holm, who did the calculations for the cameras. And I'd like to respond to the point that Graeme has brought up is how the resolution numbers presented are higher then then the actual pixel count of the sensor. The numbers were calculated using the slanted line approach which calculates the Spatial Frequency Response (SFR) of the system.

Here's a document by Doug Kerr which goes into more detail. (SFR is the average of several MTF's)

http://dougkerr.net/Pumpkin/articles/MTF_Slant_Edge.pdf

With this method the it allows the SFR to be measured beyond the Nyquist frequency, any frequency response above Nyquist will result in aliasing which is generally undesirable.

I also have a paper by Jack Holm, that he wrote with some other people while they were on the ISO committee for SFR, if anyone is interested in that just PM me with your email address.

I'll be honest and say that I don't completely understand this science (I failed Physics in college ), so I can't tell you in simple terms why this was the preferred method. But this was the method used to calculate the numbers.


-Scott