Camera shootout 2011

[David Mullen ASC]

Technically speaking isn't the camera itself irrelevant in this situation? Would the results have been any different if it had been an XL2? Or an Arricam studio? I'm no expert here but I don't think it would be.

Sure, the 35mm camera type doesn't matter too much as long as it has pin-registration, but the film format size matters -- 36mm x 24mm is a lot bigger than Super-35, which is half that area.

 

[Peter Dmitriyev]

I see that now. I am looking at the edge halo/ringing on the glasses on the table. It looks like it's had a sharpening applied and the stop looks much much greater than most of the other cameras even film.

Do you mean that the Alexa appears stopped down more i.e. it has GREATER depth of field? Because that's what I remember seeing, that the Alexa has more elements in focus (shot at let's say f/11+ while other cameras seem like they're shot at f/5 to f/8 etc i.e. slightly shallower depth of field but I could be wrong.

 

[Roberto Lequeux]

Remember each camera has a different sensor, and sensor area. All things remaining equal on the optics, there should be variations in DoF from having to back away or get the camera closer to achieve similar framing and set focus at different distances.

 

[Eki Halkka]

Remember each camera has a different sensor, and sensor area. All things remaining equal on the optics, there should be variations in DoF from having to back away or get the camera closer to achieve similar framing and set focus at different distances.

Zoom lens...

 

[Mike P.]

Given the differences in the cameras used, I think the tests were fairly decent. I understand that, had the MX been at ISO800 it would have held a boatload more info in the highs, but then they would have had to change to the optimal settings for every camera and it would have been a nightmare (and just as objectifiable, since all the footage would have looked mighty similar at 2k projection.)

As they stated (multiple times), every one of the cameras can give great results if you light properly for that camera's strengths. I think these dynamic range and exposure tests were merely to see which camera covers the broadest base, with a particular emphasis on users who were planning on switching from film (hence the 320asa). I mean, the underexposed film looked like crap, the Canons could have had slightly more DR if they used the Technicolor picture style (which wasn't available at the time), both MX AND Alexa would have been better in highs and lows at ISO800, and so forth. That's also why they did the straight "stops" test before doing the "scene" test (which is where they got their numbers from.) I think most of the people watching knew it wasn't an equal comparison -- one guy even said, it was more like 'apples to pears, rather than apples to apples.'

Also, if they had gotten an Epic, should they have been able to use HDRx even though it requires additional (mixing) steps in post to get a usable image?

 

[Roberto Lequeux]

Zoom lens...

Yea, I remember now that they used zooms. The new Fujinons, right? Still, DoF would change from different focal lengths, different perspective, etc... The framing wasn't the same all the time.
It is impossible to get everything perfect, but I don't think they changed aperture. Didn't they state it was locked from camera to camera, and through both under and over scenes?

 

[Ash Worth]

Ash, look - they even screw up M-X resolution claiming 3.8k which is not right. For a 4k shooting on a RED M-X I've measured around 3.2k, maybe more, but not as much more as to make it 3.8k. It's not about RED not coming out on top - it's a about professionals spending a lot of time and effort to conduct a test and getting an awful lot wrong in the process.

Graeme, I asked this before. Why weren't you there or a member of the RED team? Steve stated all manufacturers were invited to set up the cameras in the best way possible. So who set up the RED camera and what was the reason for RED not having a representative present. This makes no sense to me.

 

[Graeme Nattress]

It makes perfect sense not to be involved Ash. Tests like this are always flawed in some way - there's always something not right, and in the end, as a manufacturers rep about the only thing you can do if there's something not to your liking, some disagreement or whatever, is to pull out. And then they'll do the test without you anyway. Just look at what happened in this test with the measurement of resolution - they used a method I totally and utterly disagree with because it produces daft answers. From what I understand on how they measured dynamic range I disagree with that as well. The only thing I could do would be to say "no" and beyond that withdraw support. How does that look?

In the end, tests should be utterly independent with no manufacturers involved at all, and done by YOU, because it's only YOU that can test a camera the way YOU use it.

And if I had participated, I'd be utterly ashamed to be part of tests that rated cameras resolutions beyond what sensors are capable of.

Graeme

 

[Mike P.]

In the end, tests should be utterly independent with no manufacturers involved at all, and done by YOU, because it's only YOU that can test a camera the way YOU use it.

I think it's just as unreasonable to expect DoPs, Directors, Cameramen, or what-have-you, to be able to test every camera themselves and expect them to know every nuance of all the cameras they are testing. For example, when you guys gave Epics to the Hobbit or Spiderman or any of the big names for testing, you probably helped them quite a bit to get the most out of the camera... Tests like these shootouts are gateway examples so users can find out how each camera performs in general, and then focus their personal testing on only a couple.

I mean, it's just as likely (if not more so) that someone considering a RED will take it home for testing themselves and get crappy results. Chances are they'll blame the equipment before blaming their own "tried and true" test methodology that has always worked for them, which is particularly true for people migrating from film... This is also probably the only reason the Alexa has done so well -- it gets film-like results without futzing around, without tweaking anything, or needing HDRx or doing anything beyond what film-centric pros are used to and have been doing their entire professional lives (with the added benefit of drag and dropping files to the timeline.)

In other words, this shootout went on without you and the MX results (so far) aren't nearly as flattering as they could have been. People who watch this shootout (clients?) may not even bother testing a RED regardless of the lack of professionalism or precision in these specific tests. I mean, hell, they even made film look like crap (in that underexposed test), but film has the benefit of people knowing that it can do much better. The same can't be said for any of the digital cinema cameras out there, since most digital cameras (including REDs) don't have the 100 years worth of proof backing them up. You guys know the difference, most of REDuser knows the difference, but a lot of the people watching this shootout probably don't know or even care to know.

In this case, you're damned if you do and damned if you don't... But that doesn't necessarily mean you shouldn't even try.

 

[Ash Worth]

Graeme, I really appreciate you taking the time to respond. It seems like there is more going on here than I thought. Thank you for sharing this information.

My comments in a prior post were off on this. Apologies.

 

[Dan Kanes]

yikes is all I have to say about this test.

 

[Peter Rollins]

I am as well disappointed by this test. I see that Michael Bravin was in charge of all the cameras, doesn't he work for ARRI? that seems like a bit of a conflict there.

It also amazes me that with "hundreds" of professionals there , and having so many months of post onthis that there have been so many mistakes made. I am not surprised RED did not take part in this test after seeing the results. If i was a conspiracy guy I would suspect something rotten in denmark.

It is no question Zacuto has helped so many filmmakers along the way and for that I give them tremendous respect. But this just lowered that respect a few notches. And it honestly has nothing to do with the results.

 

[Craig W. Bickerstaff]

I am as well disappointed by this test. I see that Michael Bravin was in charge of all the cameras, doesn't he work for ARRI? that seems like a bit of a conflict there.

It also amazes me that with "hundreds" of professionals there , and having so many months of post onthis that there have been so many mistakes made. I am not surprised RED did not take part in this test after seeing the results. If i was a conspiracy guy I would suspect something rotten in denmark.

It is no question Zacuto has helped so many filmmakers along the way and for that I give them tremendous respect. But this just lowered that respect a few notches. And it honestly has nothing to do with the results.

According to his bio at Zacuto he also worked closely in the development of the f35 so I guess that makes him conflicted against Arri as well, I'm sure the guy wasn't sabotaging the red camera or something as you seem to be suggesting.

 

[Peter Moretti]

...
Besides, wait for all the Epic-Alexa shoot-outs... and you'll see a lot of people who can't wait to shoot on the Epic. Especially if any comparison test is finished and shown in 4K.

I love the picture quality that the Alexa delivers, but honestly, to me it's a TV camera and the Epic is a M...O...V...I..E camera. With the Epic, I can finally get VistaVision resolution and IMAX grain (or lack thereof) in a tiny camera for relatively low costs -- that's an amazing development. ...

FWIW, I agree with you completely. I've read the argument that HD is all the resolution needed for the majority of seats in a movie theater. And some people say, "I've never left a movie hearing someone say they wish that it had more resolution."

Well, while watching "Tree of Life" I wanted more resolution. I wanted to be able to make out the individual leaves, the pores in Jessica Chastain's skin. Any grain and softness was a distraction, b/c they occluded a beautiful image. I wanted MORE detail, 4K, 8K.

I wanted the clarity and detail that I saw in some of the shots in Red's "700 Reel" displayed at Red Studios on their 4K projector. It's akin to a new aesthetic. And for some/many projects, it will be very appropriate/beneficial.

Anyone who trusts the current film workflow is trusting that a lot will go right from camera to projection. Shooting a million miles of film to get the perfect image but trusting even just the film print and projection process seems like a major disconnect. Having a true 4K (or higher) pipeline from camera to projection will come closer to insuring that what's seen while sitting next to the colorist will match what you see when watching the film in the movie theater sitting next to your significant other, best friend or boss.

 

[Eki Halkka]

Yea, I remember now that they used zooms. The new Fujinons, right? Still, DoF would change from different focal lengths, different perspective, etc... The framing wasn't the same all the time.
It is impossible to get everything perfect, but I don't think they changed aperture. Didn't they state it was locked from camera to camera, and through both under and over scenes?

Zooming doesn't change the perspective, only framing, just like changing sensor size does. What it does change though, is DoF, if you compensate for sensor size with the lens - if the f-stop remains constant, longer focal lenths will have a shallower DoF, giving the false impression of a more open aperture.

 

[Eki Halkka]

Given the differences in the cameras used, I think the tests were fairly decent. I understand that, had the MX been at ISO800 it would have held a boatload more info in the highs, but then they would have had to change to the optimal settings for every camera and it would have been a nightmare (and just as objectifiable, since all the footage would have looked mighty similar at 2k projection.)

Exposing all the cameras at the same f-stop and the same ISO creates differences that are bogus, especially for a camera that shoots RAW, with an ISO setting that's only virtual and doesn't actually affect the captured exposure. Essentially, the more sensitive your camera is, the less highlights it will appear to hold, regardless of the range it's actually capable to reproduce.

As they stated (multiple times), every one of the cameras can give great results if you light properly for that camera's strengths.

But that's a totally different thing: changing the lighting for each camera (wrong) vs. proper exposure for each camera under the same lights (right). What they did now was to essentially use a wrong exposure for most of the cameras, blowing out highlights by decision, not because of the limits of the cameras.

I think these dynamic range and exposure tests were merely to see which camera covers the broadest base, with a particular emphasis on users who were planning on switching from film (hence the 320asa). I mean, the underexposed film looked like crap, the Canons could have had slightly more DR if they used the Technicolor picture style (which wasn't available at the time), both MX AND Alexa would have been better in highs and lows at ISO800, and so forth. That's also why they did the straight "stops" test before doing the "scene" test (which is where they got their numbers from.) I think most of the people watching knew it wasn't an equal comparison -- one guy even said, it was more like 'apples to pears, rather than apples to apples.'

That guy was right.

Also, if they had gotten an Epic, should they have been able to use HDRx even though it requires additional (mixing) steps in post to get a usable image?

My educated guess is that good old Red MX, and even the DSLR:s (with a low contrast in-camera gamma) would likely have produced perfectly usable results in all the tests, if properly handled.

 

[Graeme Nattress]

I think it's just as unreasonable to expect DoPs, Directors, Cameramen, or what-have-you, to be able to test every camera themselves and expect them to know every nuance of all the cameras they are testing.

Ah, but that's the real purpose of testing - to learn how to use a camera properly and get the best out of it. And on the other hand, if it's as unreasonable you say, the people involved in the SCCE test prove your point remarkably well. And as I noted, in that situation all a manufacturer can do is play along with things that are wrong or leave.

For example, when you guys gave Epics to the Hobbit or Spiderman or any of the big names for testing, you probably helped them quite a bit to get the most out of the camera... Tests like these shootouts are gateway examples so users can find out how each camera performs in general, and then focus their personal testing on only a couple.

Somehow, I get the feeling Peter Jackson has no idea that this SCCE test even exists.

Of course we help any user get the best out of our cameras. But you can imagine what a "camera test" would loo like if everyone tried to get the complete best out of the cameras - each shot would look the same or near enough - about the only thing you couldn't match would be resolution.

I mean, it's just as likely (if not more so) that someone considering a RED will take it home for testing themselves and get crappy results. Chances are they'll blame the equipment before blaming their own "tried and true" test methodology that has always worked for them, which is particularly true for people migrating from film... This is also probably the only reason the Alexa has done so well -- it gets film-like results without futzing around, without tweaking anything, or needing HDRx or doing anything beyond what film-centric pros are used to and have been doing their entire professional lives (with the added benefit of drag and dropping files to the timeline.)

Same thing applies to any camera system. I don't see that the Alexa is a harder camera to screw up than the RED at all. RED Doesn't NEED HDRx - it's only there if you want to move beyond. We know film pros who have now shot RED with superb results. It's not hard and if you're a pro DP it's your job to get the best results from any camera system.
Direct to Edit only applies to FCP, and very soon FCP will cease to exist, with the new FCPX being very much untested.

In other words, this shootout went on without you and the MX results (so far) aren't nearly as flattering as they could have been.

Well, I don't know what tests you've been watching but the M-X looked as good as any camera, perhaps better in their highlight test for instance. And it didn't need moved lights or sharpness to achieve those results either.

People who watch this shootout (clients?) may not even bother testing a RED regardless of the lack of professionalism or precision in these specific tests. I mean, hell, they even made film look like crap (in that underexposed test), but film has the benefit of people knowing that it can do much better. The same can't be said for any of the digital cinema cameras out there, since most digital cameras (including REDs) don't have the 100 years worth of proof backing them up. You guys know the difference, most of REDuser knows the difference, but a lot of the people watching this shootout probably don't know or even care to know.

In this case, you're damned if you do and damned if you don't... But that doesn't necessarily mean you shouldn't even try.

Which is why you must test yourself - only then can you ensure it's done right. And what this thread does, and in a remarkably sane and reasonable discussion, is point out the flaws and discrepancies in even the most professional and expensive of multi-camera tests. As I said before, there's no way in hell I'd attach my name to something like this as:" I'd be utterly ashamed to be part of tests that rated cameras resolutions beyond what sensors are capable of."

And these are not the first tests Primes has been involved with and not the first test to have issues either.

Graeme

 

[Sanjin Jukic]

Mistakes are a part of being human

Mistakes are a part of being human

Virtually everything can be adjusted by the tester or user (if they so desire).

The results of the tests were adjusted obviously but they made one huge mistake showing at the video from the first episode of

The Great Camera Shootout 2011/Single Chip Camera Evaluation tests (SCCE)

those ARRI Dynamic Range Test Charts where I easily found the fact (or the proof) at the shown DRTC video image that Alexa DRTC didn't showed (desired) 14.1 T stops of DR

and only can be counted about 12.6 T stops of DR.

http://reduser.net/forum/showthread.php?59507-Camera-shootout-2011/page12

They made two big mistakes, the first they "wrongly concluded" or "simply adjusted" numbers of Alexa DR and the second they "mistakenly" published video with the real result of Arri DRTC that showed another number for Alexa DR,

so that was not hard to see and discover that sort of "adjustment" or the "mistake".

That's why they didn't include Epic in the test just because it could be easy become the winner between all those Single Chip Cameras except film Kodak 5219 stock

(or any new Fuji fast film stock) that are still mastering all type of high key shooting.

With a great hope that RED will soon make properly fix of HDRx (as I suggested before) and with new RED Dragon sensor will join next battle with the newest film stocks, Sony F65 and Aaton Penelope Δ.

Arri is simply out of the next game until they try to make 4K Alexa...or sort of Alexander the G...

Also mistakes are a part of being human...

 

[David Birdy]

Glee 3D will be in theaters august of 2011...this will be a "suto shoot-out" as the Big three were there and shooting...

We had 3-3D Alexa rigs/ 3-3D Epic's, One Sony 1500 3-D rig and One sony P-1 3D rig...

I propose a "blind" viewing much the way a blind Tasting is conducted.
I know what cameras were in the positions and would like to pick the last song "Firework" to see if any one can pick-out the correct camera?? I'm open to ideas and maybe even a 1st prize for the person that guesses the correct cameras...
Cable cam was rigged for 250 pounds and had an amazing shot list!!

Sorry if this is a highjack!

 

[Eki Halkka]

Of course we help any user get the best out of our cameras. But you can imagine what a "camera test" would loo like if everyone tried to get the complete best out of the cameras - each shot would look the same or near enough - about the only thing you couldn't match would be resolution.

That of course depends on what kind of scene is being shot. The scenes in the SCCE test didn't look particularily difficult. For dynamic range, a scene similar to Red's "HDRx Barn" would be a great candidate - set each camera so that the exterior is not blown, and see what (if anything) can be resurrected from the inside of the barn. There would definitely be quantifiable differences that way... differences that also matter in real life shooting.