Pirates of the Caribbean 4

[Gunleik Groven]

The key with a data/file workflow is to make sure that there is enough media so that media is held until reviewed through dailies and editorial until recycled.

Turning over media in shorter bursts helps to keep the loaders pipelines filled so the catchup at the end of the day is minimized. Ideally it's only the last mag set of the
day and that can usually be taken care of by the time everyone else has wrapped.

Pushing overflow footage to the next day's lunch delivery also works well for some show.

This should be made a sticky somewhere...

 

[M Most]

The key with a data/file workflow is to make sure that there is enough media so that media is held until reviewed through dailies and editorial until recycled.

I don't disagree, but try telling that to a production that's turning around 3-5 hours of material every night and is told there's no money available for another set of 20 CF cards. Then try telling them that when they start having data corruptions (that eventually turned out to be caused by using a version of a copy utility with known bugs) and can't supply you with the original cards in order to determine whether the problem was created during the original file write (i.e., on the camera) or during backup operations (all of that, of course, actually happened).

One of the problems with making things "affordable" is that in a very short time frame, producers expect everything connected with them to be even more "affordable", and are unwilling to accept the risk involved in doing that - or the costs involved in preventing it.

 

[Deanan]

I don't disagree, but try telling that to a production that's turning around 3-5 hours of material every night and is told there's no money available for another set of 20 CF cards. Then try telling them that when they start having data corruptions (that eventually turned out to be caused by using a version of a copy utility with known bugs) and can't supply you with the original cards in order to determine whether the problem was created during the original file write (i.e., on the camera) or during backup operations (all of that, of course, actually happened).

One of the problems with making things "affordable" is that in a very short time frame, producers expect everything connected with them to be even more "affordable", and are unwilling to accept the risk involved in doing that - or the costs involved in preventing it.

I do tell productions all the time. Most of the time they adjust in favor of better practices. They may not do all of them but usually enough to get them into a safe zone for their production and budget.

You sound like you're the only one with a handle on reality

 

[M Most]

You sound like you're the only one with a handle on reality

I'll chalk the snark up to your being tired...

I'm certainly not the only one. But that doesn't change the reality, which is that this is still a learning process for many, and that producers don't always listen to proper advice, even after learning the hard way.

 

[David Mullen ASC]

Years ago when I was shooting features on the Sony F900, I tried to get a producer interested in shooting on the Viper, pre-SRW1 days, so it meant going to a data recorder (S-Two in this case). Because the back-ups would have to be done at the post house, I figured I needed about two days worth of recording capability to have some overlap from when I sent the drives away to when I got them back. Between that cost, and the cost of the back-ups to LTO's, and then the post house telling us that we couldn't shoot too much material or else the price would go up and the turnaround time would increase, the producer killed the whole idea and we went back to shooting on the F900 instead.

So far, I haven't been able to afford that much recording media on a Red show, enough for a whole day's worth of shooting, if not a day and a half, to make sure everything is backed-up and checked by editorial and the post house before the media is used again.

 

[Deanan]

I'll chalk the snark up to your being tired...

I'm certainly not the only one. But that doesn't change the reality, which is that this is still a learning process for many, and that producers don't always listen to proper advice, even after learning the hard way.

It was an observation not really a snark. Maybe a little.
Alot of your posts read that way lately.

Producers(UPMs, line producers, etc) listen more closely as soon as they hit the first problem. I'd say almost always. The last thing someone wants is to lose more footage when they could have prevented it.

 

[David Mullen ASC]

Producers(UPMs, line producers, etc) listen more closely as soon as they hit the first problem. I'd say almost always. The last thing someone wants is to lose more footage when they could have prevented it.

Keep in mind that Mike and I have lately been dealing with TV production companies -- I recently had to argue over a simple Log to Rec.709 conversion for dailies for a Genesis show, the company told me that it would cost them $1000/week extra in dailies and wanted me to shoot in Rec.709 in the first place despite all the arguments about how Log would look better, color-correct better, be more archival, more flexible, than baking-in Rec.709 gamma -- but saving themselves $1000/week was more important than all those arguments.

I bring this up because these producers really crunch the numbers so may factor that renting a Red package may save them "x" over a Genesis or F35, but as soon as you throw "y" at them for extra recording media, back-ups, data manager, etc. over a tape-based show, they start questioning why you want to shoot on the Red camera. In their mind, all of these cameras produce a "good enough" image for TV, so it really comes down to cost. You can only win arguments like needing more media for recording if you can prove it will save them money elsewhere, like in manpower or time after wrap for back-ups, etc. Sure, they don't want to lose data, but as soon as you wave that red flag at them, you can scare them off completely from a data acquisition camera and back to HD videotape camera, so you don't to overplay that card. You don't want to bring up the specter of "losing data" until they are well-committed to using the Red camera.

But, as I said, the whole industry and the technology is moving in Red's direction anyway, all of these arguments regarding data versus videotape will soon be moot because videotape will soon disappear as an option and everyone will be dealing with data back-ups. Even Sony has plans to replace the tape drives in their later video camera models with data recorders and whatnot.

 

[paulherrin]

thanks for the perspective, david. always appreciated.

 

[Deanan]

I bring this up because these producers really crunch the numbers so may factor that renting a Red package may save them "x" over a Genesis or F35, but as soon as you throw "y" at them for extra recording media, back-ups, data manager, etc. over a tape-based show, they start questioning why you want to shoot on the Red camera.

Best practices don't always mean things that have to cost more but simply putting efficiency, best practices, and money where it makes the most sense.

Likewise if a production is thinking they can get away with 2 CF cards per camera, there's no getting around that they'll be in a bad spot quickly. With good information, producers can make well educated decisions about how to balance the tradeoffs vs costs. Too often they are being shown half the picture. Lining up expectations with budget options and risks or benefits even sometimes means less money spent with more capability.

I work with a huge range of productions from student productions all the way up the monster production budget of this thread's title so I get to gather alot of intelligence and experience on what's working and what's failing or what the risk factors are.

 

[Detlev Eller]

loosing footage is not an option.

 

[Gunleik Groven]

@ David/Deanan

This is actually a very good reason to allow for a 1080 rec709 recording directly on the Epic, be it prores/redcode RGB or whatever.

It can protect the ability of doing a RAW online over choosing a cam that only shoots REC because of the dailies situation.

My favourite example this autumn is an episodical shot on D20 to kipro REC 709.

It's just such a weird solution, but comes from the exact situation David is giving an example of.

The DIT role veries a lot on different shows, from a simple backup and transcode to keeping the images "good" all the way until online.

Typically on the productions I work or superwise, there is a very tight dialogue between me (the DIT, when that's not me) the DP and the post people and we run a lot of tests and adjust stuff when needed as we go. The DIT has to make the images ready in a way acceptable for all involved, and I pick them up in the online/VFX stage.

But for quite a few productions that is NOT the case.
The DIT is more of a data manager, and the transcode is just considered an expensive and risky hassle. (I don't agree with that, but that's besides the point).

Having a REC 1080 editable-codec module, would give us the best of both worlds.... for those occations.

I just hope they won't immediately insist on recording 5k 3:1 if the 1080 "prores" module options becomes availabe.
But I think that may happen, as it very often are the same people who cuts corners on offlines who insist on "the best".
Like in the D20/KiPro example...

Will this lead to some productoion being finnished from those offline files?
Sure...

But it will make the Epic/Scarlet an option for a wider spectre of productions, and for those who don't handle the RAW workflow, this can be a place to start...

(But I guess the last sentence represents exactly what RED does not want to see.... in a while...)

 

[Stefan Antonescu]

Recording to 1080P ProRes is coming... but what happens when TV goes to 4K? We'll give you what you ask for... but need to tell you that 1080P's days are numbered.

BTW... you can also record 720P and transcode to SD. Every option is available. Ever wonder why YouTube has a 4K option?

Jim

Gunleik, it seems that a ProRes module is comming.
Even though Jim isn't a big 1080P fan, it's nice to see that he'll make it for those who need it...

http://reduser.net/forum/showpost.php?p=698003&postcount=9

 

[Pedro Guimaraes]

Watched the trailer last night during previews for Tron 3D.

I thought it looked great.......I for one, can't wait to watch it!

 

[Palmer Woodrow]

Our so-called "advanced" TV standard allows interlacing and has no provision for upgradeability. That pretty much tells you that 4K will never happen. We don't even have 1080p, there's no provision for it in our current standards, and millions of TVs have been sold with no requirement for firmware updates. Moronic short-sightedness.

This is what happens when you allow ignorant career politicians to dictate technical standards.

Enjoy your interlaced, compressed-to-hell MPEG-2 future.

 

[Larry Price]

Our so-called "advanced" TV standard allows interlacing and has no provision for upgradeability. That pretty much tells you that 4K will never happen. We don't even have 1080p, there's no provision for it in our current standards, and millions of TVs have been sold with no requirement for firmware updates. Moronic short-sightedness.

This is what happens when you allow ignorant career politicians to dictate technical standards.

Enjoy your interlaced, compressed-to-hell MPEG-2 future.

By that logic you might just as well have said, "Our current NTSC TV standards have no provisions for high-definition and progressive scan. That pretty much tells you that HDTV will never happen."

Standards can be revised, extended, expanded or replaced. ATSC replaced NTSC as the TV standard for the U.S. I have no doubt that eventually there will be a 4k(+) standard that will replace ATSC. We'll one day consider ATSC to have been merely a transitional standard, serving until technology evolved beyond it.

As for TVs not being updateable via firmware, I'm not certain how short-sighted it really is. With most current display technologies, you can't increase display resolution via some kind of firmware update. The maximum number of pixels the panel can display is fixed. Just as we had to replace old SDTVs with HDTVs, we'll have to replace HDTVs with 4k models.

I don't even believe that 4k is the be-all, end-all of standards either. Someday, something will come along that surpasses even that. 8k? 16k? 28k? Holographic projection? Direct neural interface? Who can say? But I'll bet that whatever it is, Jim and RED will be involved in shaping it! :beer:

 

[geoffboyle]

Years ago when I was shooting features on the Sony F900, I tried to get a producer interested in shooting on the Viper, pre-SRW1 days, so it meant going to a data recorder (S-Two in this case). Because the back-ups would have to be done at the post house, I figured I needed about two days worth of recording capability to have some overlap from when I sent the drives away to when I got them back.

On Mutants, which followed this work flow, we had 12 D mags which covered about 3 days shooting with a single camera and we had major problems at one point because of missed flights.

We didn't recycle any drive until one of the two LTO's we made on location had got back to the post house and been restored to a disk system and checked.

We go to the point where one afternoon we were about to stop shooting when a call came in clearing the first 4 D mags.

What I'm trying to say is that we thought 3 days would be enough on-site storage, it wasn't.

On Dark Country we ended up with 3 guys doing nothing but transfer and backup data so we could have enough media to shoot on.

On Wallander we had 2 cameras and 32 8Gb CF cars as well as 4 RED Raids, these had to be recycled daily or we ran out of storage. It was barely enough for a daily requirement that sent everything back to base twice a day.

I tell every producer I work with that they cannot overestimate the amount of "digital stock" they will need, yes they can recycle it but only after backups and checks and there are all kinds of problems with that.

A major movie being shot on Epics at the moment has to wait 5 days for clearance to recycle.

How many SSD's does a major movie need to cover 5 days shooting and if it takes 5 days for clearance you really need to carry 8 days worth of SSD's...

Oh and bear in mind that 3D doubles all your data requirements...

 

[Gunleik Groven]

Good points Geoff...

The single most common user bug/complaint the last 2 years or so on these boards is due to recycling mags before verifying copy.

 

[Gunleik Groven]

(Taking on the voice of my main employer the last year as a response to Kayas question for RED arguments...

RED is way cheaper than SR, which to us is the relevant comparison...
Any "HD-cam with pro 35mm adapter" racks up a bit of costs, too. Not at least in lighting...

The results are way better.
The 1080 we get out of the RED beats any other 1080 cam we've seen on all measurable results except DR.
As we're usually using the REDs in controlled light, that is less of an issue.

For VFX, the ability to run out 3k & 4k files when neccesarry to get more precise keys and compositting results is something we really don't want to lose, after getting the taste for it.

Love the re-fraiming and vari-speed capacities as well and direct editorials off set.

If the image workflow is thoroughly tested and followed up with a competent person from pre to post, we get results we've never seen before on any camera... In a good way.

Would love a proresmodule for offlines, though.

yes, and we run 1080 for offlines and uncompressed 444 log for onlines, EXR for VFX. with the EXR workflow suggestedby Graeme/Deanan, we have improved speed and color-consitancy between Grade and VFX with a wide factor. It really is THAT efficient and precice.

Concidering EXR for online now, and will test over X-mas...

:/Voice of employer off...

(And where the heck did Kaya's question go... ?)

 

[M Most]

Our so-called "advanced" TV standard allows interlacing and has no provision for upgradeability. That pretty much tells you that 4K will never happen. We don't even have 1080p, there's no provision for it in our current standards, and millions of TVs have been sold with no requirement for firmware updates. Moronic short-sightedness.

This is what happens when you allow ignorant career politicians to dictate technical standards.

Enjoy your interlaced, compressed-to-hell MPEG-2 future.

You really need to do some homework on this subject before making such statements.

The current standard was not developed by "ignorant career politicians." The ATSC document was the product of what was called the Grand Alliance, a consortium that was tasked specifically with developing standards for terrestrial broadcast of digital television. The consortium consisted of manufacturers (Zenith, Thomson, Philips), corporate research entities (David Sarnoff Research Center, AT&T) and educational institutions (MIT). If you have a better way to do something like come up with a new standard than getting the best minds in the industry (and out of the industry) together to do it (and actually keeping those "ignorant career politicians" out of it in the process), I'd like to hear what that would be. There is no 1080p in the current version of the standard because (1) essentially all monitors at the time were CRT's or projectors, and (2) there was no such thing as 24p. That didn't come along until the late 1990's, and only at the behest of Laser Pacific, which pushed Sony to develop it as a way of posting film based material in a more appropriate format. Firmware updates were not even a consideration at the time because consumer electronics devices - including televisions - were not based on computer technology. That didn't happen until at least the late 1990's. And finally, the standard is not permanent and unchangeable. There is an ATSC 2.0 project already well along that will amend the existing standard and is likely to include additional codecs (H.264/AVC most prominently at the moment), additional frame rates, and additional frame sizes. The difficulty is in retaining backwards compatibility with the millions of devices already sold, not an easy task.

What you call "moronic short sightedness" I call your inability to understand that there was a time before HDTV, before iPhones, before satellite navigation systems, before Playstation 3's, and, yes, before the Internet. Back in those dark times - i.e., anytime before 2000 - things still existed, things still got done, society still survived, and technology still moved forward. It's always very easy in hindsight to say that something shouldn't have been done the way it was, especially when it's easy to forget that technology you now take for granted has only existed for about 5 or 6 years. One could say that the US Postal Service was short sighted because they should have known that eventually there would be email. Or one could say that inventing videotape was short sighted because they should have known that there would eventually be high speed computers that could create movie files. Personally, I consider the ATSC standard a very forward thinking document simply because it provided for 18 different formats - something absolutely unheard of in the many decades of analog broadcast. It also provided for a continuing development of the standard - hence the ATSC 2.0 proposal.

As I said, you really should do some homework before coming to these conclusions.

 

[Steve Freebairn]

On Mutants, which followed this work flow, we had 12 D mags which covered about 3 days shooting with a single camera and we had major problems at one point because of missed flights.

We didn't recycle any drive until one of the two LTO's we made on location had got back to the post house and been restored to a disk system and checked.

We go to the point where one afternoon we were about to stop shooting when a call came in clearing the first 4 D mags.

What I'm trying to say is that we thought 3 days would be enough on-site storage, it wasn't.

On Dark Country we ended up with 3 guys doing nothing but transfer and backup data so we could have enough media to shoot on.

On Wallander we had 2 cameras and 32 8Gb CF cars as well as 4 RED Raids, these had to be recycled daily or we ran out of storage. It was barely enough for a daily requirement that sent everything back to base twice a day.

I tell every producer I work with that they cannot overestimate the amount of "digital stock" they will need, yes they can recycle it but only after backups and checks and there are all kinds of problems with that.

A major movie being shot on Epics at the moment has to wait 5 days for clearance to recycle.

How many SSD's does a major movie need to cover 5 days shooting and if it takes 5 days for clearance you really need to carry 8 days worth of SSD's...

Oh and bear in mind that 3D doubles all your data requirements...

On Pirates we shot 4510 Mags, 99% of those were 16GB CF Cards using RED Stations hooked via E-Sata, all archival of media was done within 24 hours of shooting, that included 8 Copies of the movie, 4 of which were long term storage that were SHA 256 Checksummed. We had 1 person doing the backups. The only reason we had a several day turn around was because they decided to use a post house to process the SbS footage.

On the unnamed project that Geoff mentioned, we are creating all the SbS editorial media in REDCineX. The turnaround is no longer 5 Days, it is less than 48 hours now. It will probably be even less as others adjust their mindset.

I'm a big fan of having enough media, 5 shooting days is very comfortable, but, it's very possible to have less as long as you have a good workflow and people.

The important thing to remember is that Editorial should get a chance to watch through the footage before recycling media. That can happen within 24 hours of shooting.

The equipment to work with RED and even S3D Epic footage is available all over the place. Any computer capable of opening a file in RCX is a possibility, anything from a laptop to a Mac Pro, and lots of things in between. The BIGGEST RED SHOW EVER IS USING RCX and a ROCKET. Think about that. You have the ability to use the same tools on your shoot.


A rough way to calculate out the number of cards is to look at your shooting estimate with some basic math.

TC = Total Cards
W = a number between 1 and 4 (1 being super low budget and 4 being expensive non repeatable action)
X = Days before Reuse
Y = Minutes of Footage Shot every day End to End
Z = Minutes of footage that will fit per card


TC = W* X * Y/Z

TC = 3 * 3 X 120/8 So, 135 cards, We shot 2 Units, which lines up with the 260 CF Cards that we used.

on a project with another very well know actor that we worked on earlier this year, we had 40 CF cards for 3-4 cameras. That was on the low budget side when it came to media, it forced us to a 1 day turnaround and some days we had to re use media in the afternoon as soon as we could verify it Data wise and Visually.

W is one the craziest variables in the formula, most shooting days are similar, and after a few days on a project you can start to see patterns. The W variable (that's not an official term) represents how full the card will be before it's pulled. On a really low budget project you might shoot every card til the camera stops, on really expensive stunts, you might pull the media after every take. THIS GREATLY CHANGES HOW MUCH MEDIA YOU NEED.

I'll get off my soapbox, but if you take anything away from this post, it should be that you can work on a wide range of shows with easy to get equipment but the single most important factor is the team of people and a basic understanding of how shooting RAW works and the steps that are needed to verify footage, both Visually and with Checksums. If all you do is copy over footage and never look at it you're going to run into issues, and likewise, if all you ever do is scrub through the footage but don't do checksums, it will come back to haunt you. At the very least make double copies through two paths.

If you compare apples to apples, it is cheaper and quicker to archive RED media than to do the same thing with Tape.