Camera shootout 2011

[Peter Dmitriyev]

please see this thread as proof that this SCCE test is already being used as evidence that the "MX is sad" compared to Alexa and film


http://www.dvxuser.com/V6/showthread.php?250789-Alexa-RED-Super-16-comparison-footage/page2

 

[Mike P.]

I don't disagree with anything any of you guys are saying, I just dislike the idea that there seems to be no way to objectively compare these cameras without operating them for solid couple of weeks or paying for a workshop. Having to spend a lot of time finding out how to get the most out of each platform, before you've actually decided to use that platform, is a frustrating pill to swallow, and also works in the favour of every camera out there (the improvement we know can be made to the MX imagery can also be made to any other cams available to varying degrees.)

 

[Graeme Nattress]

Mike P. - yup, it's hard. As for getting the most out of a camera - happy to help all REDUsers get the most out of a RED with as much advice and help as we can give out online. In the end our job is to make great cameras - and your job is to make great images with them. The end result is great images. If you read REDUser and get advice that helps you get better results from another camera, well, the upside of that is you get better results, and the upside for us is that you got that advice on REDuser and you'll think of using a RED when it's appropriate for your project.

Graeme

 

[Roberto Lequeux]

Glee 3D will be in theaters august of 2011...this will be a "suto shoot-out" as the Big three were there and shooting...

We had 3-3D Alexa rigs/ 3-3D Epic's, One Sony 1500 3-D rig and One sony P-1 3D rig...

I propose a "blind" viewing much the way a blind Tasting is conducted.
I know what cameras were in the positions and would like to pick the last song "Firework" to see if any one can pick-out the correct camera?? I'm open to ideas and maybe even a 1st pize for the person that guesses the correct cameras...
Cable cam was rigged for 250 pounds and had an amazing shot list!!

Sorry if this is a highjack!

Sounds like a good idea. I would like to think I could pick them out easily on a theater screen... we'll have to wait and see.
Being a concert, with wild concert lighting, it might be hard to pick them apart. But a blind viewing is always a great idea.

 

[Scott Lynch]

I would like to know who you heard that from. As the producer from Zacuto on this project and the previous comparision's I can tell you without a doubt that there was no bias towards any camera in this testing. I have seen the RAW footage, I have seen the graded footage and I was there on set.

-Scott Lynch

this was in reply to this:

I've heard in private from one of the participants there was a heavy bias to make some of the cameras looks good. This kind of makes sense now looking at the inconsistency in the tests.

 

[Scott Lynch]

I am as well disappointed by this test. I see that Michael Bravin was in charge of all the cameras, doesn't he work for ARRI? that seems like a bit of a conflict there.

It also amazes me that with "hundreds" of professionals there , and having so many months of post onthis that there have been so many mistakes made. I am not surprised RED did not take part in this test after seeing the results. If i was a conspiracy guy I would suspect something rotten in denmark.

It is no question Zacuto has helped so many filmmakers along the way and for that I give them tremendous respect. But this just lowered that respect a few notches. And it honestly has nothing to do with the results.

Hi Peter,
To clarify Michael Bravin was not in charge of ANY cameras. He was the station chief for Station 1 which used a back-lit illuminator where he setup the ARRI Dynamic Range Chart, a OECF Chart, and a Wratten Color Chart. He was only in charge of maintaining the consistency of the test he did not setup or run any of the cameras tested.

Also Michael Bravin did not work for ARRI at the time of testing. But beyond that, with so many people involved in the this whole comparison it has become bigger then any one person, or one company. No person involved could have skewed the results even if they had wanted to. And beyond that, this was one of the most altruistic group of people I've ever worked with. We wholeheartedly want people to do independent tests of their own and continue to do so. Graeme is completely right when he says the the only person who can really make a choice about a camera and do a test is YOU.

We want to simply add our data into the knowledge base of what is known about all these cameras.

Ultimately people tend to believe whatever they want to believe.

If you doubt us, then I would like to offer out a challenge: Be involved with us next year, help us improve the way all cameras are tested.

-Scott

 

[Timothy Eaton]

Scott,

I am always happy to receive new information about cameras and film stocks, so I applaud Zacuto and the SCCE team for doing the testing. However, any camera test is only as good as the methodology used to conduct it. The team, if the video is representative of their findings, is woefully short on explaining methodology and offering test parameters and settings, but seems perfectly happy to offer hard conclusions about dynamic range, pegging Alexa at 14.1 stops and Red MX at 11.9. This can have real world consequences for owner / operators as well as rental houses. It seems only fair, if you are going to publish hard conclusions about dynamic range, to offer complete written descriptions of the methodologies used in the test, as well as the settings, spot meter readings, and procedures for each step of the test for each camera. As a simple example, at one point the video appears to say that all of the cameras were rated at ISO 320. This and other claims lead to a host of questions that are simply unanswerable unless we have the hard data about methodologies. Is there any plan to publish the data?

We want to simply add our data into the knowledge base of what is known about all these cameras.

Yes, but unless we know how the tests were actually conducted, the data will only serve to cloud the landscape. The hard data conclusions of the team need to be backed up by a robust standard for how the data was collected.

Ultimately people tend to believe whatever they want to believe.

The purpose of the test is to move beyond "belief" and into the realm of substantive data which allows us to draw informed conclusions. If you don't provide the data, then we are left with conjecture or whatever biases we began with, and the test is relatively useless. Again, the goal is not to question the motives or commitment of SCCE or the testers, but simply to encourage the release of information which will make the tests useful.

Tim Eaton

 

[Jared Caldwell]

Scott, my main questions are in terms of methodology. I'm just pretty surprised the DR of the Red One MX was so low, especially when other tests contradict these claims. When will the white papers be released?

I'm not saying I hold any of you in anything other than the highest regards, but as a scientist, anytime there is a presentation of quantitative data, I reserve judgement until I see the complete test methodologies...especially when there are some figures that show discrepancies with other tests.

 

[David Mullen ASC]

People on RedUser have to stop beating up on other people every time they run a test and get different results than some Red users get. Calling them incompetent or ascribing to conspiracy theories, etc. just ensures that people will be loathe to shoot and post tests in the future, or that they will completely avoid testing Red equipment out of fear of a backlash. I already recall one evaluation test that was going to be done where a professional organization was just going to leave the Red One out of it rather than deal with the politics -- I had to convince them otherwise.

 

[Timothy Eaton]

People on RedUser have to stop beating up on other people every time they run a test and get different results than some Red users get. Calling them incompetent or ascribing to conspiracy theories, etc. just ensures that people will be loathe to shoot and post tests in the future, or that they will completely avoid testing Red equipment out of fear of a backlash. I already recall one evaluation test that was going to be done where a professional organization was just going to leave the Red One out of it rather than deal with the politics -- I had to convince them otherwise.

David, I think most of what's going on here is reasonable discussion. There is some stuff that's inflammatory and unfair, and that's too bad. A lot of it happened initially as a result of what seemed to be the surprising conclusions of the test. But a good part of it is the result of releasing hard conclusions - DR data incremented in tenths of a stop - without the hard data to back it up. Much of the speculation could have been avoided by publishing a white paper, as Jared suggests, instead of a whizbang video with hard conclusions and very, very sketchy data. The video simply invites speculation.

I happened to run across some old stock tests I did with another cinematographer many years ago - 5248, 5293, 5296, 5298. I still have all of the graphs we made, the complete spot meter readings, aperture and shutter settings, as well as printer lights. I would never have thought of publishing our conclusions without publishing the methodology and settings we used. I think asking for the same kind of data from the SCCE test is perfectly reasonable.

Tim

 

[Jared Caldwell]

It was mentioned in a DVXUser thread that the SCCE would in fact be releasing a white paper. I enjoy the Zacuto documentaries as entertainment (and I get to see what the footage looks like), so I think the videos are worthwhile. We just need at least a white paper with more supplemental information in order to accurately discuss the findings. I'm not expecting the tests to be perfect, but I bet there are some interesting findings. Though people involved in the tests may not enjoy (constructive) criticism (the only kind that truly matters), it should show that their work is being taken seriously. The only way we learn is by presenting new findings and placing them through a peer reviewed process...good, bad, and ugly.

 

[Scott Lynch]

Tim and Jared,
I also want the whitepapers and methodology to be released publicly, and I am currently working with the people involved to help compile all of the information. The documentation is dense, the amount of footage recorded is in the 8 TB range and there are literally millions of files. Most of the volunteers that have worked on this have given the better part of the past 5 months to this project and no one is getting paid to compile the info. So it is coming, but I don't have any timeline as to when it will be actually released.

If I can be completely frank, there has always been a ticking clock on the footage. Even as post production started, there were things like the Technicolor profile for the Canons, the F3 S-Log update, the Epic being released, ect. That already started to date the evaluation. It seemed that no matter how fast or hard we worked the footage was becoming more and more out of date. This production is like any other, where there are the real limits of time, money and people's patience. We try to stress in the show that this is not a "winner-take-all" contest, it is in fact said in the episode that the important part is the person behind the camera more then the actual tool. We also say in the episode that numbers don't tell the whole story, and shouldn't be the only basis for evaluating image quality. But I also understand that, unfortunately, the numbers do influence people and I know that there have been and will be producers or directors or even dp's who will pick a camera for a job solely based on the numbers that have been published, whether they are the manufactures numbers or the SCCE numbers, or some other testers numbers. Is it the right way to choose a camera? I don't think so. Will it still happen regardless? Yes, because it's a shortcut to actually testing and comparing the tools.

The white papers are coming, all we can ask is for your patience until that happens.

-Scott

 

[M Most]

We try to stress in the show that this is not a "winner-take-all" contest, it is in fact said in the episode that the important part is the person behind the camera more then the actual tool. We also say in the episode that numbers don't tell the whole story, and shouldn't be the only basis for evaluating image quality.

In that case, I would strongly suggest not calling it a "shootout." That term instantly invites controversy, as it did here. I pointed out that at no time did Bob use that term during the presentations in both Las Vegas and Los Angeles, instead referring to it as what he intended it to be, which is an evaluation, and an examination of different characteristics of the cameras involved. By using the word "shootout" you make it confrontational, and many people, including a large number of people on RedUser, are going to immediately pounce on that.

 

[Timothy Eaton]

Scott, My thanks go out to you and everyone who participated in the test, and I look forward to the white paper. I know it's particularly hard to conduct a test in a rapidly changing landscape.

As a Red One owner, I have already had many discussions about dynamic range. Let's face it, it's one of the critical characteristics of a sensor or film stock that every DP is concerned about. I have been privy to discussions involving reputable DPs where the capabilities of MX have been disparaged relative to Alexa because of dynamic range. It is often unclear whether or not the criticisms of MX are based on tests or just vapor.

It's one thing to say there are no winners. It's quite another thing to distribute a video in which Alexa was claimed to be the clear winner by 2.2 stops over Red MX. I'm not just concerned about someone making an initial decision based on the SCCE evaluation. I'm concerned about people who may have already made up their minds about which camera is "superior" for a variety of reasons that may have nothing to do with the true capabilities of the cameras. They may have little or no technical knowledge but carry a great deal of authority in making camera choices. Some are simply looking for validation of their preconceptions. In this situation, the SCCE "shootout" becomes an 800 pound gorilla in the room. I believe there are many Redusers in Europe who can attest to this phenomenon with respect to Alexa.

Again, none of this is to question the integrity or the contribution of the testers. I just want more information and am eager to read the white paper. Not sure it will address all the issues, but at least we will better understand the process. Many thanks Scott for participating in the evaluation and wading into the fray.

Tim

 

[Steve Dent]

I haven't a Red One MX, so I don't have a chance to perform my own tests. But I'm interested in buying a used MX when the Epix-X's come out... but I was under the impression from previous posts on Reduser that the Red One MX had a DR of about 13.5. That's a big difference from 11.9.

I'm concerned that the Zacuto tests are hanging their hats on an SCCE test that seems flawed, at least in some areas (resolution) according to what Graham said. Measuring resolution as higher than the sensor pixel count is clearly a fault in the process.

On the other hand, the images from the "shootout" make the Red look very good, in the area of highlight retention as well as shadows. It seemed nearly the same as the Arri, yet the people present at the test theatres were raving about the Arri, and not the Red... were they seeing something we weren't?

Anyway, I'm using this test, as well as many other sources, to make economic decisions which affect these companies, so I sure hope the methodology is flawless or nearly so.

 

[Mark Pugh]

To set the Red MX at 320 ISO, and point it to a contrasty scene, with no regard to a histogram that is showing clipped highlights, is ridiculous.
Especially when you know that changing the ISO setting to the manufacturer's recommended setting, and exposing the camera accordingly will will give fine results.
ANYONE who has read the user manual, or glanced at the exposure info on the LCD, knows that raising the ISO on the camera from this setting that's not recommended, and setting exposure accordingly will give better results in the highlights.
So what's the point in setting the MX to 320 for this test? It's moronic.


The Zacuto video is labelled "Episode One" but there is no indication of whene "Episode 2" will emerge.
I could be wrong, but I think that there is also no indication of what "Episode 1" is supposed to cover.
All it did cover, though, in a pointlessly flawed way, is dynamic range.
One gets the impression watching it that all the people at the screenings talked about was shadow and highlight detail, which I find a little hard to believe.
That no-one noticed lack of resolution or aliasing in a 7D or 5D shot, or milky highlights in an Alexa shot...
Don't get me wrong - to me dynamic range is a big issue, always on my mind.

To my eye the MX performed fine in the test with the girls, looking at the highlights out the window, but very differently in the "low light test"
Highlights in the glassware blown out.
I can only guess that the camera was set up very differently, or used different setting in Redcine-x, or whatever was used.
It's the only way I can explain the inconsistent results.

There should probably be a warning that appears on the the Red One and Epic, that alerts the user to highlight loss, when the camera is set to a low rating.
There'd be more good shots taken with the cameras, because unnecessary highlight clipping is the most common user error, and has a big impact on how people feel about the camera's results.
It would therefore be in RED's interest.

 

[Scott Lynch]

Hi Mark,

The Zacuto video is labelled "Episode One" but there is no indication of whene "Episode 2" will emerge.

At the end of the episode we say that episode 2 will come out in July, and episode 3 will come out in August.

I could be wrong, but I think that there is also no indication of what "Episode 1" is supposed to cover.

At the very beginning of the episode we say that this episode covers Dynamic Range and Latitude, at the end we say that episodes 2-3 will cover resolution, compression, noise, color, motion artifacts and skin tone.

One gets the impression watching it that all the people at the screenings talked about was shadow and highlight detail, which I find a little hard to believe.
That no-one noticed lack of resolution or aliasing in a 7D or 5D shot, or milky highlights in an Alexa shot...

As we wanted to keep the episode specifically about dynamic range we have not included the many comments about the other issues. You will see those comments in episodes 2 and 3 when we examine those particular topics.

-Scott

 

[Mark Pugh]

Sorry I missed the mention of Ep 1 being only about DR and Latitude - my mistake.

 

[Sanjin Jukic]

Definitely that whenever desired the numbers can be adjusted but only the real images will tell you that there is no big or

if any difference in DR between Alexa, RED MX and Kodak 5219 in SCCE Vimeo HD 1080p video presentation.

I talked and showed about that here at this tread before.

SCCE Latitude stops chart has mixed (or mistakenly wrong numbers) about RED MX and Alexa.

As I showed you that SCCE Arri DRTC showed exacly 0.7 T stop in favor of Alexa.

That was expected just because Alexa's Alev II sensor has larger cells than RED MX that also provide more dynamic range and in this case is about 0.6 T stop more.

14. 1 T stops is so called Potential Latitude/Dynamic Range for Alexa and 13.5 is Potential Latitude/Dynamic Range for REd MX.

Difference is again 0.7 T stops.

Click on the image below for a full-resolution jpg



Original SCCE charts posted on Vimeo video.

Click on the image below for a full-resolution jpg



I’ve added numbers to the original charts that SCCE Vimeo video posted.

According to the SCCE published Vimeo video screen shots from ARRI DRTC Alexa has a degree of exposure latitude about 12.6 T stops of dynamic range values and RED MX 12.6 T stops.

Somebody made a mistake and wrote that Alexa has exposure latitude degree about 14.1 T stops of dynamic range values.

That's obviously not a true just because at the picture above from Arri DRTC can be seen only about 12.6 T stops.

It is a picture taken from ARRI Dynamic Range Test Chart that was supervised by Michael Bravin.

Maybe somebody whated to say that Alexa has 14.1 T stops of Potentially Acceptable Extractable Latitude but that totally differs from 12.6 T stops of Useable Extractable Latitude.

But somehow SCCE test charts made mistake with mixing Extractable and Potentioal DR numbers or we can go further here and could even say that is something adjusted with the numbers there.

Click on the image below for a full-resolution jpg


Comparison between Alexa, RED MX and Kodak 5219 at the low key or underexposure scene.

Click on the image below for a full-resolution jpg


Comparison between RED MX, Kodak 5219 and Alexa at the high key or overexposure scene.

If they would blow up SCCE example footage @ 4K projection then only Kodak 5219 and RED MX will shine and obviously Alexa will fall short of DR because lack of resolution.

People should finally realize or know in advance that Observable Dynamic Range/Latitude and Resolution are related.

Now to conclude is that you will not see in a near future so many 2D features shot on Epic neither Alexa.

Why?

Just because Red and Arri have to fix their digital cinema cameras pretty soon otherwise...

RED has to finally fix Epic DR with custom design of new Epic sensor (a.k.a. RED DRAGON)

and also Arri has finally to make 4K digital cinema camera (that means also 4K capable sensor).

That's what I think why film is still prefer choice for acquisition of any serious 2D feature today if of course budget can allow this option...

Also that's why Epic is the first choice for many today's blockbuster 3D production (there is no better system available)

and why Alexa slowly becoming the first choice for many TV production today (there is no better system available).

A serious 2D production still today will choose to shoot on film.

 

[Scott Lynch]

The white paper will have all of the camera settings and go into the exact methodology and post workflow, but according to the slate in the footage that I have the MX was recorded as ISO800 in both scenes. There may have been the use of ND filters to bring the exposure down to ISO320 @ F/2 which was the base exposure of the scene. By base exposure I mean the spot reading off the girl in the foreground's face in the dark scene, this did not mean every camera was set to ISO 320, the camera master could set the camera how ever they wanted, but they could not change any settings between the Overexposure Scene and the Underexposure Scene. The methodology behind these pairs of tests will take quite of few pages to explain exactly, and I am not the best person to explain the choices in detail.

The white paper will address these issues.

-Scott