Dec 2009, academy cam assessment test

[Tim Whitcomb]

Not sure where to put this... this is DIFFERENT than CML test...

I just got this email from a filmmaker... friend.

Hi Tim, Greg, Shawn,

Thought you guys maybe interested in the Camera Assessment Series latest test results which was displayed last week here in LA at the Academy by SMPTE; the same results will be displayed Dec 2nd, 2009 in grander fashion at the main Academy Building on Wilshire.

Arri D21, New Panasonic Varicam (3700), RED ONE (with 2009 firmware update), Viper FilmStream, Panavision Genesis, Sony F35 and F23 were all split screened against a 35MM camera for the same Ext. and Int. shots and all cameras were at full tilt (in essence Uncompressed 4:4:4 RGB HD at 2K, or 1080p, or in the case of RED -- 4K REDCODE RAW because it does not offer Uncompressed). All images were shown at 2K (including 35MM) and particular LUTS were used to match the 35MM Control shots, however no sharpening or noise reduction was allowed.
It was a technical discussion that was too much for me but the general consensus was that SONY F35 and Arri D21 were the best (in matching the color depth, gamut, and contrast of the 35MM) and RED ONE was the worst (the culprit being the 10-1 RED CODE RAW compression).

To me the VIPER, D21, Panavision Genesis and F 35 were all number ones. The rest were all a close second and the RED was the worst. Highlights were blown out in the RED (they had this fire flame which all cameras handled except RED) and the image looked ruddy Int. and soft in deep focus.

Overall 35MM film was the best although it did have less blue and red saturation than the D21 and F35 in Ext. Shots. It was mostly ahead in resolution and handling highlights.

RED Camera seemed ahead of the game in marketing and camera body price point but behind in image quality.
VARICAM 3700 from Panasonic was unable to have shallow DOF because of ENG style sensor although the image was stunning.
SONY F23 looked video-like in certain INT. Shots but the image produced was very high quality.
There was an argument in the post discussion that the inability to sharpen the image in post hurt the RED more than other cameras given that RED CODE RAW comes out soft to start but the SONY, ARRI, and PANASONIC reps frowned at this.


-WP (Ill keep his name private for his safety)

Thoughts? Comments?

Whats most obvious to me is this was shot in Jan 2009, pre- NEW COLOR SCIENCE... but once again looks like RED is getting unfairly compared due to a

LACK OF UNDERSTANDING of RAW IMAGE FILE PROCESSING.

Why dont they let company REPS attend these tests to ensure fairness?

 

[Neil W. Smith]

Already been discussed ...

Already been discussed ...

Tim,

The whole issue of the legitimacy of the ASC and BSC CAS Tests has already been discussed ad nauseum on RedUser .... see this thread:

http://reduser.net/forum/showthread.php?t=32101&highlight=neil_hdi

If you ask David Mullen, ASC, he has one view that supports the ASC rationale behind the tests ... and David is a very fair minded chap and a solid RED shooter ... his point of view is totally valid.

If you ask me, I have a different view .... I think the test is BS quite frankly ... have taken up the issue with the ASC Tech Committee on several occasions but to no avail .... the tests stand as what they are ... oh and by the way, several other ASC DPs I know also share my concerns

I was with Ted at the ASC CAS presentation a few months ago and we both left the theater fuming ...the problem of such tests is that technically minded DPs seeing the results on a large screen know that there is much more to the RED picture quality than shown in these tests if you expose the RED correctly and post it in a non-DPX pipeline .... but the real danger of such tests is that when shown to non-technically minded Producers, Directors or even film-based DPs, it is very easy for them to jump to the conclusion that you can't shoot a decent movie with the RED ONE ... which we all know is just not true.

I also sat in a room of 500 Producers when the ASC CAS was first shown at the PGA weekend many months ago ... you could tell by the level of questioning afterwards that most of the people in the room didn't have a clue as to what they were seeing and why they were seeing the RED look so bad .... all they knew was that the RED looked the worse.

No doubt David and Mike Most will take umbrage with my stance but I have to repeat ... the ASC CAS tests did not show how good RED can look when exposed correctly and posted in a RAW .r3d workflow no matter that they were set up against the benchmark of the HEAVYIRON Telecine/DPX workflow that is prevalent in this town.

I'm not going to say anything more on the subject ... if your DP friend has any doubts over how good RED footage can look when graded properly, send him around to us .... we'll sort him out.

Neil

 

[Tim Whitcomb]

Thanks Neil- Yeah, that was my problem, the danger you speak of, as this director is now absolutely convinced RED is inferior now, so I challenged him to take our camera out for free and let us PROCESS the RAW footage... properly. Then let him decide.

All this test says to me is that when plugged in to the status quo pipeline, RED will suffer... well DUH!

Can certainly understand why you and TED were fuming... especially the 4K to 3.2K fully debayered somehow makes the RED "fake 4K?" LOL!

I actually look forward to the challenge. Just thankful Im not in LA.

 

[Blair S. Paulsen]

SMPTE meeting

SMPTE meeting

I was at the same SMPTE meeting that Tim refers to and thought the split screen presentation revealed a great deal. I do not necessarily concur with all of his comments but they likely represent the majority viewpoint.

One concern in the methodology of the CAS that Neil pointed out that I agree with, is the reliance on the DPX file format. Why should that matter? The Cineon log curve is engineered (AFAIK by Kodak) to compensate for the inherent non-linear properties of negative film. Moreover, it assigns numerical values in a way that maximizes the ability to retain information most obvious to the human eye/brain system. Makes sense - IF your source element (film) has an inherently log profile gamma. There are many other issues related to the characteristics of print film as the target space that favor a film source.

All due respect to the people who put on the CAS, they did, after all, have to create some parameters to mount their tests. One point of reference was that since a huge majority of distribution and exhibition is on print film it's the only rational target choice. Sounds good at first blush, but if you really want to explore the capabilities of a camera system that relies on linear sensor data, IMHO its a stacked deck. Care to dispute the contention that as the years roll by digital projection will gradually supplant film projection due to economics? Perhaps a DCP to DCI spec at 4K resolution as the target would yield different perceptions? Perhaps not, but I'd sure like to see that test.

Beyond that, I question one of the most fundamental elements of their test design - that 35mm negative film is the best acquisition standard to strive for. While it creates stunning images that I am a big fan of, film does not create a "perfect" scene rendering. Neither of course does the RedOne or any other camera system. My point is that a more objective standard would provide a neutral finish line so every horse has the same shot.

Yes, I realize that a clinically perfect image might lack intrinsic values widely appreciated by actual human beings. Unfortunately once you head down that road in a camera test the "eye of the beholder" factor quickly destroys any measurement metric.

Cheers - #19

 

[Steven Caesare]

...RED ONE (with 2009 firmware update),...

Sorry, but I have to laugh at that.

For any other digital camera, that statement makes sense.

For Red, that could mean one of a dozen different versions.

I'm afraid the source should probably aim to be a bit more specific when dealiing with a company that likes to give major feature updates out for free, rather than charging an arm and a leg for minor fixes once every 3 years...

-sc

 

[David Mullen ASC]

One reason why I wasn't as upset at the results of the ASC/PGA tests was that, honestly, my own experience shooting on the RED for two features was not that far off from that the ASC testers experienced, but the caveat was that this was based on taking the footage to a major D.I. facility and going the 4K LOG DPX route.

In a nutshell, I mainly struggled with some dynamic range problems and the converted 4K image was not quite as sharp as I hoped it would be. I followed that shoot up with a Genesis shoot and found generally that the dynamic range of the Genesis was better, that the sensitivity was higher, and that the noise in 3200K lighting was lower. And the sharpness was about the same.

And obviously I didn't shoot the RED with some agenda to make it look bad, quite the opposite.

But I also freely admit that I've seen better RED footage than what I shot -- better lit and exposed, better processed, better color-corrected.

So what does that all mean? Should a test show you only the best results or the most common or likely results? Should it be shot in processed in the best manner or the most industry-wide manner? It seems both types of tests are valid and informative.

If the RED camera didn't do well going through a standardized 10-bit LOG DPX post route out to 35mm film -- the most common way of doing D.I.'s in the industry -- then doesn't that tell you something, if only that you have to look for more specialized workflows and post houses to get better results?

Honestly, there were just as many fans of 35mm film at these screenings of the ASC/PGA tests who cried "foul" at the notion of 35mm film going through a 2K D.I. instead of being contact-printed, or even better, shooting 35mm anamorphic and contact-printing it, skipping digital post altogether. To those purists, 35mm film was being "dumbed down" to the level of the other HD cameras. They didn't care that RED did worse than the F35, they were upset that the 35mm wasn't much better than the F35.

So it just shows me that there is no pleasing everyone. When people don't see the results that they want to see, they get upset. Almost everyone has an agenda and wants their favorites to "win".

This director who was discouraged by these ASC tests could have easily done his own tests AND GOTTEN THE SAME RESULTS. He would have just been disappointed by his own test instead of someone else's. Or he could have shot his own test AND GOTTEN DIFFERENT RESULTS and preferred the RED. Who knows?

The fact that it's hard to post a RED shoot "correctly" is a real problem for RED to solve because it's going to happen over and over again. You can only cry "foul" for so long before you have to do something about it. So I'm sorry if I can't get quite as upset as others here about the poor results of the RED camera during the ASC test. It was shot by an ASC cinematographer with a RED representative present to advise and make sure that the camera was being properly used. RED was also invited to the color-correction sessions. They were not shut-out of these tests.

Yes, you can argue with the basic methodology of the test, which was the common post workflow using 10-bit LOG DPX files. But to me, that was a completely legitimate form of testing because this is by far the most common D.I. workflow used worldwide for cinema images, digital or film. It would have been more controversial to NOT use this workflow and instead, give every camera a specialized workflow that was rare, that required proprietary software unique to certain post houses or specialized equipment.

As Curtis Clarke was keen to remind everyone, that test was a snapshot in time, and the times change. As methodology evolves, there will be new tests and new results and I'm absolutely positive that RED will do better.

I will add that now some of my own RED projects have been moved to another post facility, Lightiron, and the footage is looking better because these guys know how to work with RED footage. So times change, things improve, opinions shift. Nothing is set in stone. And anyone who sees one single test and forms a definite opinion that cannot be changed over time with new information is, well, an idiot. And that means they are going to be an idiot about a lot of other things.

Look, I want RED to do better, and I want these post issues to be solved, because I can guarantee that I'll be shooting more RED projects in the future. So it's in my best interests that things improve. Which is another reason why in some ways, I'm fine with the ASC tests pointing out the problems with RED's dynamic range, and noise and color reproduction in 3200K lighting -- because I have had the same problems myself and I want them fixed by whatever means possible, both on my end of things, on RED's end of things, and on the post house's end of things. We all have to work together to solve this problem so that RED can achieve its full potential consistently and efficiently.

 

[jimhare]

Well said David.

I am a committed Red defender but everything you said makes perfect sense and is fair.

We just get frustrated since there are so many out there just waiting for results like this so they can scream "I TOLD YOU SO" from the rooftops! :cryin:

We all know how to get beautiful images from the Red, and we all invested and committed to Red. Naturally we will take it a bit personally. :smilewinkgrin:

I sometimes watch Mythbusters "prove" something and think, "they didn't prove anything, clearly they should have checked X."

 

[Tim Whitcomb]

Thanks David. I certainly get that context... although I am curious, if you still feel that way with the NEW color science? Have you shot a feature/project with that yet?

I come from a BG where other than a Panavision Gold PKG, Ive never shot with high end cameras, so RED has been an insane uptick in quality over video.

But as Jim Hare says, its these "I TOLD YOU SO" guys that annoy... all because of a test, none of them have actually shot with the camera. They now refuse to even consider it, and I agree, that makes them idiots.

I guess what is unclear is what the point of the test was? If its to demonstrate a "snapshot in time" fair enough... not sure thelogic, especially as its almost a YEAR ago the test commenced... but if its to determine which camera looked the best, I respectfully disagree with the parameters of the test...

and actually, it seems you answered my question... the point of the test was to determine which camera performed best with a defacto standard workflow...
and the winner is... defacto standard cameras... no surprise there.

THIS is why I can totally relate to RED being pissed off. They are reinventing the wheel, as it were, and learning a ton along the way.

I truly believe once they have the form factor handled (EPIC) and can get thru this transition. that more focus on the post side of things is inevitable, and its obvious with
FLUT ... that they have learned a TON in 12 months since the test.

This camera is really still so new, and I still think its awesome that it even was invited to the test and apparently the SI-2K was not!?

 

[Jeff Coatney]

I was at the same SMPTE meeting that Tim refers to and thought the split screen presentation revealed a great deal. I do not necessarily concur with all of his comments but they likely represent the majority viewpoint.

One concern in the methodology of the CAS that Neil pointed out that I agree with, is the reliance on the DPX file format. Why should that matter? The Cineon log curve is engineered (AFAIK by Kodak) to compensate for the inherent non-linear properties of negative film. Moreover, it assigns numerical values in a way that maximizes the ability to retain information most obvious to the human eye/brain system. Makes sense - IF your source element (film) has an inherently log profile gamma. There are many other issues related to the characteristics of print film as the target space that favor a film source.

All due respect to the people who put on the CAS, they did, after all, have to create some parameters to mount their tests. One point of reference was that since a huge majority of distribution and exhibition is on print film it's the only rational target choice. Sounds good at first blush, but if you really want to explore the capabilities of a camera system that relies on linear sensor data, IMHO its a stacked deck. Care to dispute the contention that as the years roll by digital projection will gradually supplant film projection due to economics? Perhaps a DCP to DCI spec at 4K resolution as the target would yield different perceptions? Perhaps not, but I'd sure like to see that test.

Beyond that, I question one of the most fundamental elements of their test design - that 35mm negative film is the best acquisition standard to strive for. While it creates stunning images that I am a big fan of, film does not create a "perfect" scene rendering. Neither of course does the RedOne or any other camera system. My point is that a more objective standard would provide a neutral finish line so every horse has the same shot.

Yes, I realize that a clinically perfect image might lack intrinsic values widely appreciated by actual human beings. Unfortunately once you head down that road in a camera test the "eye of the beholder" factor quickly destroys any measurement metric.

Cheers - #19

Excellent points, Blair.

 

[Jay A. Kelley]

David,

I think you are one of the best representatives that RED has if only because you do say "it's good, but can be better, here's how". From what I know of Jim and the guys, this is EXACTLY the kind of info they are after.

Having made the DVD.. I can say for certain the RED is it's own beast. And has it's own way of doing things. In industry that likes to standardize whatever it can, a completely unique system has it's work cut out for it.

So basically we're asking RED to be different, and the same...

Yikes.

Jay

 

[David Mullen ASC]

I think RED's goal is to both fit within standardized post workflows, such as 10-bit LOG 1080P HD or 2K and 4K DPX, etc. -- and customized ones. That's why they offer those all those options.

So if they feel that taking the footage through a 10-bit LOG 2K DPX D.I. workflow is fundamentally flawed and should be avoided, they should be more vocal about that, and if not, they should get that workflow to, well, work.

 

[M Most]

... if its to determine which camera looked the best, I respectfully disagree with the parameters of the test...

and actually, it seems you answered my question... the point of the test was to determine which camera performed best with a defacto standard workflow...

Having talked to quite a few people involved in both production and post for the ASC/PGA program, and also having been at the SMPTE presentation that some here are referring to (sitting next to my friend Blair, as a matter of fact), I have a few comments.

First, Tim's statement above indicates a very common misunderstanding as to the point of the exercise. It was not to "determine" anything, as was pointed out numerous times, it was not a "shootout." It was to provide a comparison of characteristics of these devices when shot under identical conditions, and posted in what came closest to what would be a similar work path, in a film targeted DI environment. The film target was used because like it or not, that is where we are today. Film is the dominant medium for motion picture distribution worldwide by a very, very large margin. That is why DI's today are almost exclusively done in a film targeted pipeline, because film prints are the primary means of exhibition. Everything else - digital cinema packages, video deliverables, and any electronic product - is derived from the film version, allowing a consistency of color and look across all these different mediums. At some point, that likely will change to a digital cinema targeted pipeline, but right now that is not really sensible because digital cinema represents a rather small portion of theatrical exhibition, and likely will remain that way until the world economy significantly improves. Now, if one understands that this was the setting and criteria for this particular event, the need to process all of the material through a film centric DI pipeline becomes clear.

My feeling is that the results obtained with Red in this particular program had much less to do with the particular file format used (DPX vs., say, the original R3d files) and much more to do with the fact that Red's color matrices - as implemented by the debayer routines - are not targeted to eventual film display, but are rather targeted towards electronic display. This is what Red ultimately wants and supports, and it is a choice that they have made. If you compare Red to, say, Arri, you would find that Arri has color matrices that are supplied with their debayering software that are targeted directly towards film finishing. If you take an Arri Raw file and process it using Arri's Image Booster software using any of their LogCFilm matrices, you get a result that looks correct when viewed through a typical film print preview LUT, often without any color correction at all. If you take a R3d file and use, say, Redlog, or even one of the PDLog LUT's, and look at it through the same film print preview LUT, you get an image that is rather flat, and in need of some serious color correction to pull the colors back into their proper display values. This is indicative of choices made by the manufacturers to target particular finishing paths and deliverable products, choices that are neither right nor wrong, but which do have a serious influence on the results obtained when passed through a pipeline other than the one they're optimized for. Both the D21 and the Red use CMOS sensors that deliver values based on linear light. And even though the filters in the color filter arrays of their sensors might be a bit different, it's really how the information is handled by the demosaic and color matrix that differentiates the ultimate colorimetry of the two devices. In theory, there is no reason the D21's images should work better in a film targeted pipeline than Red's other than than the choices made by both companies in terms of the color matrix applied to the original image. Even in the case of the Genesis and F35, the choice was made by the manufacturer to target a color pallette that is more indicative of film than electronic cameras, and thus requires less post color manipulation in a film targeted pipeline. Some of the other characteristics that were observed in the ASC/PGA footage - such as a bit more clipping in very bright fire on one of the day exterior shots - should have been expected, as the Red One as it existed at the time of shooting (January, 2009) had less dynamic range available than some of the other cameras, by Red's own admission.

Now, it is true that there are alternatives in approaches that can be taken to generate a film deliverable. One can set up a post pipeline that is video or digital cinema targeted and use an inverse LUT to create a file that will work on a film recorder and "fake" the electronic colors into a film recording. And that is a valid approach. However, the vast majority of DI's done in major facilities today are not Red sourced. They are primarily shot on film. And even among those that are not, at least up to this point these facilities have dealt with a lot more Genesis originated material than they have Red. One cannot expect these facilities to completely change their methodology completely for one product that to this point has not had particularly significant penetration into their market. That might or might not change, but to this point, that is the fact. So if one takes that into consideration, it becomes a bit more understandable that this particular program was done using the "standard" methodologies that work quite well for essentially every other device. To do otherwise would not represent real world approaches.

One can say that facilities should adapt their methodologies when new technologies come along, and that is probably true. But is is just as true that at least to this point, Red has not really devoted serious effort to providing imagery that can be used in the currently dominant film centric DI world without serious compromise or additional work, as other manufacturers have. Red has pursued its vision of a 4K digital projection world, which is certainly a worthy pursuit, but it really doesn't exist yet, certainly not to the level that would allow one to target that format as the primary deliverable. There is a current standard for theatrical exhibition that is dominant, and that happens to be 35mm film prints. Rather than complain about the results obtained when targeting material to that format, an alternate approach would be one similar to what Arri has done, which is to accommodate that format by designing color matrices that yield images that are appropriate for that pipeline. Whether Red chooses to do that is, of course, up to them.

 

[Tim Whitcomb]

Mike- Thanks so much for first, not being annoyed by my post

and thank you for this very clear overview... makes perfect sense to me and I DO understand that this was a "snapshot in time". YES, film is predominant, and the economy has certainly helped slow digital adoption... but the big reason I jumped on to the RED wagon was because of their foresight and boldness (despite much resistance) to pioneer the Digital future...

in many ways, I think this test bodes well for post houses like Lightiron (and ourselves hopefully up in these parts) who want to specialize in RED finishing for bot exisiting and future pipelines. Mainly because we can still offer value to producers...

Like many, I just think its unfirtunate that this TEST, was indeed viewed as a shootout
and it will influence some negatively towards RED. Certainly has a director up here who cant stop telling me how Awesome the D21 looked and how bad the RED ONE looked.

Thanks again, appreciate this viewpoint very much.

 

[David Mullen ASC]

Sure, the ARRI D21 looks great, but does this director want to deal with a below-200 ASA camera for a whole project? Or recording uncompressed 2.8K RAW using data recorders?

 

[Carsten Fenstermacher]

Having talked to quite a few people involved in both production and post for the ASC/PGA program, and also having been at the SMPTE presentation that some here are referring to (sitting next to my friend Blair, as a matter of fact), I have a few comments.

I've heard that there was an separately graded pass of the red footage that looked great but the PGA/ASC does not want to show this. Can anyone confirm this?

Does anyone know what was delivered to the post house that did the final grading? Who was the post house?

 

[M Most]

I've heard that there was an separately graded pass of the red footage that looked great but the PGA/ASC does not want to show this. Can anyone confirm this?

It sounds like you're trying to create a conspiracy theory where there is none. While there were a number of things done with all of the files, the results that are being presented publicly are the ones done through a film targeted DI path because that was the entire point of the program.

Does anyone know what was delivered to the post house that did the final grading? Who was the post house?

The original R3d files were used by Laser Pacific, who did all of the post work. In the case of the R3d footage, the files were converted to 10 bit DPX files using Red's Redlog LUT, and graded by using an ASC CDL power function to bring the values a bit more in line with expected Cineon values, and then doing subsequent color manipulation on a Lustre through a film print preview LUT. Basically, not very different than many facilities have done Red originated DI's for the last 2 years.

 

[DS Williams]

Well said David.

I am a committed Red defender but everything you said makes perfect sense and is fair.

That is exactly our problem in this industry at the moment. We currently have committed "defenders" of formats, many times foregoing rational testing and common sense in the name of protecting their stakes in the current chaotic format/camera war climate. Cameras are tools. And only tools.

 

[David Mullen ASC]

It's an ongoing and unsolvable conflict -- the need for standardization and the desire to implement new technology with new workflows, formats, etc. It's a basic tug-o-war that has only gotten worse with digital technologies that change constantly. Eventually people get tired of the chaos and settle for an imperfect standard for awhile, until that eventually gets tossed out and it's the Wild West all over again. Both sides of the argument have valid concerns. Without standards, we waste an awful lot of time creating new workflows and testing new processes, only to find that we cannot control the process once it leaves our hands and enters the wider distribution world. We end up creating stuff that only works when we physically control it all the way through presentation, which is fairly useless because most of us want our work to be seen by others.

Also, with standards, there is the ability for other technical areas to evolve and advance working within that framework. For example, we started a standard of 4-perf 35mm movie film back before 1900, and look how far the permutations of that standard went, how much quality was eventually wrung from that imperfect standard.

On the other hand, I also admit that the 10-bit LOG 2K/4K RGB DPX standard will have to evolve (or be abandoned) as more and more movies are shot with digital cameras and shown with digital projectors. But we aren't there yet.