ASC and BSC camera tests ...

[Neil W. Smith]

There's an interesting discussion happening in the 'Off Topic' forum which has some relevance to all RED owners bidding for DP gigs or talking to Producers and Directors about the benefits of shooting with RED:

http://www.reduser.net/forum/showthread.php?t=31901

I attended both ASC presentations at the Producer's Guild weekend and at the DGA and have voiced my concerns over the way the test were devised. I didn't see the BCS results but Geoff Boyle, BSC, has also raised concerns over on CML on the way that RED was presented at the BCS test.

Ted and David Mullen were also at the DGA presentation and afterwards we had a brief discussion on the results .... here's a short quote from one of my posts in the Off Topic thread:

"Ted and I discussed the ASC methodology after the presentation at the DGA and both agreed that the RED images could have been made to look a ton better if a different post process had been used. The work we've done with ASC DPs over the last few months prove that RED if correctly exposed and finished in an optimized post workflow can produce very 'filmic' and high-quality images at a great price point.

So, if you hear anything negative coming out of the ASC and BSC tests about the quality of the RED images, you might want to take it with a pinch of salt ... you really have to understand the objectives of the tests and the processes employed before you make an informed decision on the real quality of a RAW based workflow."

The point being that there is now a whole bunch of Producers, Directors and DPs walking around with the misguided idea that it is really difficult to shoot RED and deliver a very high quality finished product ... this is simply not true. When RED is correctly exposed and put through an optimized post pipeline that extracts the most information from the RAW .r3d files, it is relatively easy to get a great looking final product ... certainly a lot easier and more straightforward than the ASC and BSC camera tests would have you believe.

So just be aware that if you're talking to someone who's heard about the ASC or BSC camera tests (and who doesn't understand why the tests were done the way they were) you might need to spend some time explaining the shortcomings of each test and reminding them of how wonderful RED images look and the importance of taking their .r3d files to a post-house that knows how to extract the most from RED RAW.

Neil

 

[Brandon Fraley]

Why WERE they done "the way they were"?

 

[Karl Gustav H.]

AIUI, the test(s) used a standardised workflow for each camera on test. While this can reduce the number of variables, it also fails to reflect the nuances of each camera type.

 

[David Mullen ASC]

Because a 10-bit Log 2K DPX workflow is the most commonly used D.I. approach for both film and digital shot for theatrical release.

The trouble with using some particular post house's special workflow for RED using possibly proprietary or non-standard techniques developed in-house is that then the ASC or BSC would appear to be endorsing a particular post house. So they had to choose a post path that was fairly commonplace for post houses.

Which is certainly a valid thing to test, how these cameras handle being put through the most commonly available processes being used. Because if a producer choose to shoot on the Red camera, he could end up almost anywhere with the footage. So the question is whether to show the producers the best result possible if he is lucky enough or smart enough to find that special place that knows how to get the best out of the Red camera, or to show the producers what would happen on average if he took it to a mainstream D.I. facility that may or may not have a lot of experience with the Red.

Ultimately these sorts of tests tell people where there is room for improvement because technology is not a static thing. Look, there are nearly 5000 Red cameras out there, so it's not like people are not going to be shooting with them because of the ASC and BSC tests... what it means is that producers are going to demand post houses learn to get better results out of the Red camera. And it also means that Red and the post houses are going to have to communicate better; if there is some particular technique that is going to improve the quality of D.I.'s working with Red footage, it's in Red's best interest to disseminate that information.

 

[I Bloom]

What color space was used to generate the 2K DPXs?

IBloom

 

[Gavin Greenwalt]

If there is some particular technique that is going to improve the quality of D.I.'s working with Red footage, it's in Red's best interest to disseminate that information.

I think this is a really important thing to note. The RAW to RGB conversion should be difficult to make a mistake. Not to do right.

It's like what I always say about RAW. "If it's hard then the developers did it wrong."

It's just going to take time since we've spent well over a decade making everything way more complicated than is now necessary. But I'm sure in a few years we'll get there.

 

[Tony Brown]

To be fair to the BSC they did go to some lengths to at least try and explain that they fully understood that the fairest system they could agree on would not be the level playing field they might have hoped for. Its a shame it was RED and SI 2K that suffered the most.

TBH both exceeded 16mm without grain elimination in many respects IMO, which was not the conclusion of many present. The 16mm looked terrible. As for the extract shown on 35mm print of "A Bunch of Amateurs" which had been shot on S16.... I'm amazed they showed it. Inteded to show the benefits of S16 it was actually hard to look at.

It merely highlights the fact that, rightly or wrongly, we're all standing in a minefield being told different ways out.

Being a polite Brit I'll be saying "after you........".

 

[Neil W. Smith]

Camera tests and minefields

Camera tests and minefields

To be fair to the BSC they did go to some lengths to at least try and explain that they fully understood that the fairest system they could agree on would not be the level playing field they might have hoped for. Its a shame it was RED and SI 2K that suffered the most.
....
It merely highlights the fact that, rightly or wrongly, we're all standing in a minefield being told different ways out.

Thanks Tony for the feedback on the BSC camera test .... agree totally that it is very hard to provide a "level playing field" and that pragmatic choices have to be made in order to get these camera tests done ... many talented people contributed their time for free in both tests and we should all be grateful for their hard work and dedication.

Many of us learned a lot from the ASC test (and I suspect the BSC test) about the shortcomings of particular digital cameras but we also know that all the digital cameras in the tests can produce excellent images if exposed and finished in a way that is optimized for that particular camera .... have a look at 'Benjamin Buttons' if you want to see how good Viper can look if done properly.

RAW based workflows are changing the 'acquisition-workflow-output' equation in quite dramatic ways .... we're just in the middle of grading a RED feature shot by an ASC DP (Marvin Rush, ASC) and it looks gorgeous - incredibly filmic. Our colorist is not using the 10 bit log DPX workflow recommended in the ASC camera test but both Marvin and the Director are amazed at how much color information is available in the 12 bit lin .r3d files and how far we can push the images in the grading suite .... correctly exposed RED in an optimized color workflow can look as good as all the other more expensive digital cameras.

The point being that it's not just that we're in the midst of a 'minefield' of different options in digital filmmaking, we're in the midst of a revolution in 'warfare technology' that is shifting the balance of power from the old analog world to the modern digital world. Unfortunately, RAW based 'land-mines' are going to blow a few more people up before the technology is well understood.

In the meantime, from those of us that have been in the 'Hurt Room' for the last few years, know this, its getting easier, its not quite so painful, and the rewards when you've cleared the minefield are definitely worth the blood, sweat and tears. No one said it'd be easy but then nothing of value is generally gained without a lot of hard work - ask Jim how many sleepless nights he's had over the last three years.

Keep shooting well exposed RED images and us RAW post guys will deliver you back wonderful looking movies ... the revolution has only just begun.

Neil

 

[Chris Bell]

Neil,

Having just watched Knowing this weekend, there is far more capacity with the RED than what I am getting from REDCINE and Prores exports. Can you share some workflow advice and how you are grading the RAW files? What would you advise folks working with only a MacPro?

 

[M Most]

Neil,

Having just watched Knowing this weekend, there is far more capacity with the RED than what I am getting from REDCINE and Prores exports. Can you share some workflow advice and how you are grading the RAW files? What would you advise folks working with only a MacPro?

Personally, I would advise them to seek out professionals if they want the quality of a professional finish. Expecting someone with limited experience and a desktop computer to be able to achieve what a highly experienced professional colorist working with well shot footage, high end equipment (in the case of Knowing, a Quantel Pablo system) and a support staff that includes an entire color science department achieved is pretty unrealistic.

Having said that, there are many resources available for learning the craft involved in color correction. What there is not is a magic bullet that allows you to instantly be an expert. Steve Hullfish's book is a good start. The Reducation program will likely give you more specific information in dealing with Red files. Doing a project with a professional facility (like Neil's Hollywood DI, for instance) will let you see and experience what's involved in a professional environment and ask questions as you go. What you should not expect is for those same professionals to offer you free answers over the Internet because, quite frankly, that's not what they're in business for. I would suggest booking a session with them if you want to receive a nice education on what it is they do.

 

[Sanjin Jukic]

ASC and BSC have to precisely elaborate all the tests with written and publicly open "Technical papers" and then we could see where the thing have got wrong with RED1.

A part of that "cynicism sort of test" is that RED1 mostly has got all of us RED Users in Europe with 4K Digital Projection of PJ short movie "Crossing the Line" that first time happened at IBS 2007 in Amsterdam, NL.

NOTHING WAS WRONG WITH RED1 FOOTAGE DIGITAL PROJECTED FROM FROM THE BEGINNING!!!!!

What actually they wanted to show us with that expensive and months long prepared test?!

See BSC test video report here>>>

No comment!!???

 

[Stephen Williams]

NOTHING WAS WRONG WITH RED1 FOOTAGE DIGITAL PROJECTED FROM FROM THE BEGINNING!!!!!

Hi,

IMHO the best footage from RED one today is way better than CTL. CTL looked very good, however it was not without problems.

Stephen

 

[David Mullen ASC]

There's a difference between a demo piece shot to show-off a camera versus a comparative test between multiple types of cameras. Without any frame of reference, most cameras can create fairly slick demo pieces that show off the strengths without revealing the weaknesses.

If you saw the Red footage from the ASC test by itself, you'd think it looked pretty good and 35mm-like without only a few minor issues in one test. Even when compared to some other cameras, only a few other differences become visible in terms of color reproduction and dynamic range.

The ASC test had a couple of day exterior shots, one day interior shot, and two night interior shots. The Red matched the other cameras fine for all the day work, except that some really bright flames during an explosion were a bit clippier than what some other cameras captured.

For the night work under 3200K lighting, the Red found some odd colors in clothing (like a purple shirt next to a burgundy shirt) a little harder to separate than the other cameras, and in the infamous light bulb test, it didn't handle the intense overexposure on the skintones as well as some other cameras. But these were really the only two weak areas and they were not radically weaker by any means, the differences were subtle.

Both the light bulb test and the day explosion test were examples where you had a basic exposure chosen to best capture the overall scene but there were brief moments of intense overexposure, so the issue becomes do you set the exposure to help hold detail in the worst-case moments but then underexpose the majority of the shot? The final exposure was chosen by the DP working with the manufacturer rep. These were valid tests because they challenge the dynamic range capability of the system because you can't set an exposure that maximizes both shadow detail and overexposure detail, you have to decide what to bias the exposure for.

Honestly, even if a different color-correction approach would have slightly improved the Red footage in the ASC test, some of the weaknesses revealed in terms of color reproduction under 3200K light and dynamic range limitations are not new to anyone who has worked with the camera already. It was interesting just to see how the other cameras handled these challenges. But I don't think anyone was really surprised either about the performance of the F35 / F23 / Genesis, because there have already been tests for these cameras. I think more people were surprised at some of the results from the Arri D21 and the Panasonic 3700 because these were less well-known.

I was most interested in how the Arri-D21 behaved because it was the closest camera to the Red, both being 35mm Bayer-sensor cameras. It looked and behaved pretty similarly to the Red, but it had a little better dynamic range and a little less noise, but was much slower in speed, maybe a stop slower.

I believe we saw both a digital version and film print of the ASC test, where were about the same.

Not having seen the BSC tests, I can't say what the issues were with the Red other than people seem to think it was not processed very well.

 

[Chris Bell]

MMost,

What's the point of this forum if not to share technical information and advice? I am not asking him to give away trade secrets, but I'd love to hear some insight on how he is getting the most from the RAW footage.

 

[M Most]

MMost,

What's the point of this forum if not to share technical information and advice? I am not asking him to give away trade secrets, but I'd love to hear some insight on how he is getting the most from the RAW footage.

Except that you are asking him to give away trade secrets. How his company handles Red material is what constitutes his competitive advantage. It is what hopefully sets him apart from others who don't do it quite as well. You are asking him to give up that advantage, and to do it on a public forum. If you want to find out more about what they do and how, I repeat my previous suggestion, which was to call Neil and book a session at his facility, and ask questions and pay attention while you're there. That way, you gain some insight and expertise and he gets to run a business.

 

[Shane Betts]

Do we know what firmware version was used on the tests?

 

[Joe Walker]

I think more people were surprised at some of the results from the Arri D21 and the Panasonic 3700 because these were less well-known.

Hey David, would you mind sharing your thoughts on the Panasonic 3700? I'm actually quite curious how that performed in the test.

 

[Jason Wingrove]

I hope to be catching up with Sue Gibson from the BSC re their (what looked like incredibly intensive and scientific tests) for some feedback soon!
jas

 

[James Brundige]

Personally, I would advise them to seek out professionals if they want the quality of a professional finish. Expecting someone with limited experience and a desktop computer to be able to achieve what a highly experienced professional colorist working with well shot footage, high end equipment (in the case of Knowing, a Quantel Pablo system) and a support staff that includes an entire color science department achieved is pretty unrealistic.

.

I think this points to a real challenge. In the olden days of film post, professional colorists did all the grading. Now, with the tools available, anyone with a laptop and a Red can grade footage. I just finished a show about 40% Red footage. I did the original grade for off line edit. I am a veteran filmmaker with a pretty good handle on Photoshop. I thought the footage looked good. I took the show to a real colorist to print to broadcast tape, and the difference is night and day. There is a real art to color, gained from years of experience and apprenticeship.

A good analogy is editing. Now any bozo with a laptop can cut something in Final Cut, but an experienced editor, with years of knowledge, can make a piece sing. If you want a great show, get a great editor. If you want a great looking show, get a great colorist. Red gives us the tools to do it all, but the skill set can only be acquired through study and practice.

 

[Neil W. Smith]

On getting the most from RED footage ..

On getting the most from RED footage ..

... There is a real art to color, gained from years of experience and apprenticeship ... If you want a great looking show, get a great colorist. Red gives us the tools to do it all, but the skill set can only be acquired through study and practice.

James,

You raise some some very valid points about the challenges involved in getting the most from RED footage .... we were fortunate enough to be involved in grading the first images from the first RED prototype camera way back in the summer of 2006 (in RED years that is ancient history) ... the original .3rd files were 16 bit TIFFS (76 MB each) and Jim would take the TIFFS into Photoshop, adjust the curves until he got the look he wanted and then Graeme would adjust the de-bayering algorithm until it matched Jim's look and then we'd grade the images in Scratch ... don't know if you remember the beautiful clip of the black Oakley watch - well, it didn't look quite so amazing when we first saw the images ;-)

We've graded every version of the camera firmware from many different RED cameras over the last three years and have been tracking the way Graeme has been improving his algorithms ... the key to getting the most out of the .r3ds files is to understand how Graeme is applying his genius to extracting the maximum color information with the least amount of noise from RED RAW. De-bayering is both a science and art ... Graeme is the consummate Renaissance man able to combine high-tech in the service of artistic vision ... know the man, know the technology.

Along with improvements in camera technology we have a grading environment that also helps match the developments in digital acquisition to improvements in color correction techniques:

1) a $100,000 2k DLP projector with deep blacks and wide color gamut
2) an $18,000 spectroradiometer which allows us to measure the RGB values of the projector to within one thousand of any color space standard we choose
3) a special 20 foot DI screen that is non-perforated and reflects light and color without any shift in gamma or chromaticity.

This high-end, tightly calibrated viewing environment means we can absolutely trust the images coming back off the screen and is a stable constant in the ever changing world of digital acquisition and workflow ... we can tell if RED or Assimilate or Apple change any part of the imaging chain and make adjustments (or phone calls) as necessary. Grading the 4k .r3d files in a tightly calibrated grading environment is key to getting the highest quality finish from RED footage.

We don't grade off CRTs or LCDs ... there's real difference between the brain's perception of transmissive light through a small monitor versus reflective light from a large projection screen .... color grading is a subtle art that depends on many cues from the matrix of colors and shadows that are presented to the eye and interpreted in the brain's visual cortex (you actually see with the brain, not with the eyes) ... the 'magic of movies' take place on the silver screen and its ability to trick the viewer into the 'willing suspension of disbelief' that is the hallmark of the theatrical movie experience.

As well as the technological underpinnings of great color correction we have a highly talented colorist who has two degrees in Art and a deep understanding of color science and has been working with RED files since day one .... when you watch a grading session involving a skilled DP and a talented colorist you are a witness to one of marvels of modern filmmaking .... the interplay of the DPs' artistic vision with the colorist's ability to manipulate the digital images on screen is a a joy to behold .... shooting RED with professional color correction in mind is a recommendation for all RED filmmakers - if you haven't seen your images graded in a top-notch grading environment with a talented colorist who knows how to work with RED RAW you are missing out on the real power of the RED revolution.

If you listen carefully to the comments of Dave Mullin and Mike Most (both highly talented and respected practitioners in their respective fields of cinematography and color correction) you get a sense of how rapidly digital acquisition and file-based workflow is changing and how much 'blood, sweat and tears' goes into staying on top of developments and maintaining mastery of the art and craft of filmmaking.

Though we are in the midst of an amazing technological revolution in filmmaking there is one area where there's no real difference .... whether you're shooting with 35mm or RED its still all about the story .... technology should help serve the artistic vision of the filmmaker not hinder it ... we strive wherever possible to make the technology invisible to Directors and DPs which is why we continually seek to optimize the relationship between 'acquisition + workflow + output' at any given budget.

Great RED finishing is dependent on great RED cinematography and smart workflow .... there are no short-cuts in digital filmmaking ... its still all comes down to the hard work of people like Jim, Graeme, David and Mike continually pushing the envelop so the rest of us can take the tools and make great looking movies.

Neil