The Great Camera Shootout 2011 - Episode 3

[Eric Z]

Episode 3 is out. Check it out on Zacuto.com

Spoiler:
Again they f**#-up, and had different lighting for the Alexa (on the scene with the fruit and spices).
Why did they do that? I don't get it.

 

[Scott Lynch]

Hi Eric,
When the SCCE was shooting that scene, the Alexa was the first camera to go through the station. There was a source 4 that was simulating a slash of sunlight that came from screen right. After the Alexa had shot the scene, that light was either bumped or panned and then remained that way for the next two days. The problem with the lighting on that scene was not discovered until the SCCE was being edited about a month later, and at that point it was impossible to reshoot or fix in anyway.

-Scott

 

[Zeb B]

Sounds like the Z team needs a pro team to do these productions if they are to be a fair test

 

[Timothy Eaton]

Scott,

As I've said in the other discussions about this shoot out, thanks to the SCCE for conducting the test. On the other hand, lights getting bumped, use of non-optimal ISO in some parts of the test, lighting differences in other sements of the test, discrepancy between the results of portions of this test and other well documented tests, and screening of the test at 2K instead of 4K, all lead me to view the results with a fair amount of skepticism. When somebody bumps a light in a major test with all this horsepower and nobody notices until a month later, mmmm, color me less interested in the results.

By the way, what about the promised white paper? Was it ever published?

Tim Eaton

 

[Mark Toia]

Circus....

I just want to know why you view in 2k and 4k camera?

So you have to down resolution a RED camera down to the rest of them... was that fair?

but... one thing interested me was the rolling shutter test... very obvious slur for the RED & ALEXA compared to F35.

Other than that... I'm not sure what they got out of it. All the people in the test liked different cameras anyway... Some liked soft film, some like canons, nikons, sonys, arri..

Did they do a bang for buck test? ... I doubt it.

 

[Scott Lynch]

Hi Josh,
Just to clarify, Zacuto did not design, run or administer these tests. There was a sperate organization (the SCCE) that had complete control over all aspects of testing. We only made a documentary about the tests and peoples reactions to the test. For a complete list of the people that ran the test if you watch our documentary there is a second set of credits at the end which lists the people involved with the SCCE. I think if you look at those credits, it's hard to say that the crew that ran the SCCE isn't pro. Although this mistake was made, in all the presentations of the SCCE this issue has been called out.

-Scott

 

[Kemalettin Sert]

Hi Josh,
Just to clarify, Zacuto did not design, run or administer these tests. There was a sperate organization (the SCCE) that had complete control over all aspects of testing. We only made a documentary about the tests and peoples reactions to the test. For a complete list of the people that ran the test if you watch our documentary there is a second set of credits at the end which lists the people involved with the SCCE. I think if you look at those credits, it's hard to say that the crew that ran the SCCE isn't pro. Although this mistake was made, in all the presentations of the SCCE this issue has been called out.

-Scott

Zacuto Senior Editor Karen Abad edited these videos.she is working for ZACUTO right?

 

[Scott Lynch]

Zacuto Senior Editor Karen Abad edited these videos.she is working for ZACUTO right?

Yes, she was the lead editor for this documentary series, and she works for zacuto. The SCCE edited a separate presentation of just the test footage which was shown at NAB and in screenings that we held all over the place.

-Scott

 

[Michael Mayda]

GIGO...

 

[Sanjin Jukic]

Now at Episode 3rd after some criticism they did it much better.

At the end of the game film won simply with its exceptional skin tones and digital acquisition lost as it was expected before.

 

[Zeb B]

Hi Josh,
Just to clarify, Zacuto did not design, run or administer these tests. There was a sperate organization (the SCCE) that had complete control over all aspects of testing. We only made a documentary about the tests and peoples reactions to the test. For a complete list of the people that ran the test if you watch our documentary there is a second set of credits at the end which lists the people involved with the SCCE. I think if you look at those credits, it's hard to say that the crew that ran the SCCE isn't pro. Although this mistake was made, in all the presentations of the SCCE this issue has been called out.

-Scott

If the goal is to produce objective Science: when a experiment gets compromised - you throw out the results and rerun the tests

If the goal is to use "science" to promote a brand by driving traffic to a website - don't fret the details that affect the results and publish regardless

 

[Patrick Miao]

I was really surprised by the rolling shutter test. Red didn't do too good; I was under the impression that the rolling shutter was greatly reduced but apparently it can still be a problem. I guess that has to be accounted for in the shoot but it would have been nice if it was reduced a bit more.
I though overall the test was relatively fair. Arri makes a great camera, as does Red. The reason I prefer the Red is the ability to manipulate the image more, at least more so than with the other cameras. That is in terms of exposure, color correction and the ability to crop more due to it's higher resolution. All video has to be color corrected in post, so whether I have to bump the blue channel or red channel a tiny bit means nothing. I also thought the skin tones were good with Red but I thought that film looked the best in this area.

 

[Eston Diaz]

I watched the previous ones and it always felt like they had an issue with Red.

Whatever, I learned a lot but didn't really care about their results.

 

[Eric Z]

To my eyes, RED One MX seemed to have the cleanest footage. Alexa, F3 and F35 are great as well.
The rolling shutter is a problem that needs to be addressed. I'm sure everyone will come up with great solutions (Global Shutter or some other clever idea).

 

[Peter Moretti]

Now at Episode 3rd after some criticism they did it much better.

At the end of the game film won simply with its exceptional skin tones and digital acquisition lost as it was expected before.

I hate to say it, and I know that lots of money and time were spent and copious amounts of equipment donated, but this was by far the worst one yet.

It almost felt like a spoof produced by "The Onion." Except for Rodney Charters, it seemed that the people in the audience didn't know what they were looking at. Their comments often times contradicted the images being displayed. For example, the Red footage looked poorly graded, dull and rather sickly, but the complaint from the audience was that Red images are overly saturated. Huh?

The practical requirments of each camera were completely ignored. E.g. no mention of the extreme extra cost, size and weight of the external recorders used with many of the other cameras. There was no mention of the color grading post production workflow. For example, how easy is it grade S-Log in the NLE's most of us use? How do you post F35 footage? Don't you need an HDCAM SR deck? How well does Alexa 4:4:4:4 work in Premiere and Avid, are there any color shifts due to Quicktime?

And of course, no mention at all of how easy or difficult it is to manipulate the footages in post, i.e. does RAW give you more latitude in your grading choices? How about the differences with using 4:4:4, 4:2:2 and 4:2:0 while grading?

Honestly, this last installment was incomplete to the point of being pretty much useless. (I say that knowing that a lot of hard work went into it and being a fan of Zacuto products and a defender of the first two installments.) Some schmo might watch this episode and think for $13K he can get an Alexa-lite in the Sony F-3. But wait, you need the S-Log and 4:4:4 firmware upgrade which costs $2K. And you need a recorder and media which costs $??. And you need a way to post the file. Do you really want to create some ghetto S-Log color effect or do you want use a proper LUT? And how do you do that in an NLE? Or do you need a grading suite for that?

I could go on, but really, this episode proved very little. Except to those who mistakenly believed that film's skin tones sucked.

 

[geoffboyle]

Having run a series of tests I can only say that it's really easy to sit back and criticise.

You should just try doing a test like this to see what the problems are and the time and cost involved.

If you really need the information from tests to be relevant to what you do YOU need to go out and run some tests.

Of course if you then have the balls to put those tests up for public criticism...

 

[Karim D. Ghantous]

I have watched all three with eagerness. I think film got some unfair treatment (ep.1, underexposure test where the film was treated as if it were digital - you have to expose for the shadows, people!). I also think that the film was underexposed by a little bit in the skin tone test.

IMHO film was handled improperly. From memory, the test showed more overexposure tolerance than underexposure tolerance, which proves that the official ISO is wrong by at least one stop. Film really has to be exposed in the context of the scene in question. I even noticed a shot or two in Indy4 where the shadows were a bit muddy - proving to me that you have to be careful with shadows. IMHO.

These tests are certainly not effortless so I have no seriously negative criticism to offer. They couldn't get a hold of the EPIC so they had to use a ONE MX. Fair enough, I think.

The two full-frame Canons were the worst in terms of resolution. I watched the low-res, 360-line file as my computer can't play HD files well. And the two FF Canons still did poorly. The lone Nikon (crop sensor, like the 7D) had better resolution than all three Canons to my eye. Why people think that the 5DmkII is the Second Coming is beyond me. It's an awesome video camera in one sense; but for serious work it turns into a piece of junk. It's a fad camera. That's sads, because it really has some good points (I'll pass on it anyway, thanks all the same).

If you have knowledge of technology (from a minimum degree upwards) then this test was still very useful. So I thank Zacuto!

 

[Graeme Nattress]

If you really need the information from tests to be relevant to what you do YOU need to go out and run some tests.

That's the right answer! Do your own test to your own specifications based around how YOU shoot and what YOU are going to be using the cameras for. Nothing else is better than that!

Indeed doing such a test with such a wide range of cameras is hard, and mistakes and errors do and will occur. I thought Josh's comment above was important criticism though - "If the goal is to produce objective Science: when a experiment gets compromised - you throw out the results and rerun the tests". I know that if I'm doing some camera tests as I sometimes do, if I screw up I'll do the test again and again until I get it right. That's because I care about the answers and I need them to be correct. That said, I'd rarely test more than two cameras at once which makes it much easier to repeat a test.

With the earlier episodes going into measurements and graphs we were promised a white paper on their derivation which has not appeared. There has been strong and valid criticism from myself and others about these graphs. Many many viewers saw these graphs at their NAB presentation and those viewers have never been properly informed that the numbers they saw were wrong (mistakes were pointed out to them and numbers revised).

Graeme

 

[Eric Z]

Sure, it's easy to criticize when you yourself don't run these tests.
But, for these tests to really be as credible as possible, several things need to be in order:

1. Each camera must shoot at its highest capability using the full potential of its sensor and recording codec (which the SCCE did).

2. Lighting of the scene must be exactly the same. I don't accept any variation in lighting due to "someone accidently knocked off one of the light stands".

3. Position and angle of the front glass to the scene must be exactly the same . It's hard to compare when the scene is captured a little differently by each camera.

4. Same lens should be used on all cameras, with Focus point and Aperture exactly the same. Pick a certain point in the scene and focus on it throughout the test.

5. The only variation allowed is distance of front-of-lens from the scene, due to crop factor of sensors, but this is to ensure that the scene is captured in exactly the same focal length by all cameras.

6. The scene must not change, especially with still subjects. The table with the fruit is a good example (some of the fruit was repositioned between camera changes).
Even with human subjects, their placement, orientation and facial expressions must be kept in check all the time to ensure they are exactly the same.

7. Once points 1-6 are guaranteed, and it's time to move to post, footage must be cropped to the smallest form-factor, so we can get to see the exact same frame on all cameras.
So for example, a 16:9 crop would be decided upon. That means the DSLRs would remain in 1920x1080, and a camera such as the RED One MX would be cropped to 4800 x 2700.
That means we won't lose any resolution or detail, just crop out what the other cameras couldn't see.

 

[Patrick Miao]

I think one of the points of the test was that no one camera does everything well. Some excel in certain tasks more than others. I wouldn't take this test as a definitive answer as to which camera is best. Arri looked real good in this test as well as Red. Of course, if one knows their camera, they will shoot in a way that optimizes the cameras strong points and minimizes the cameras weak points. The appeal of Red is it's price point, resolution and latitude. But I wouldn't necessarily exclude using other cameras if I felt it made it easier to get the results I wanted. This test is just an overview and I was impressed with a lot of cameras, camera I haven't even heard of.