4K Storage for Features - How Much is Enough?

[Dan Blanchett]

I've been shopping around for good RAID options for my next Mac workstation, but the decisions don't stop at what brand to buy.

I plan to edit a two-hour feature in some proxy size (like 1K) from the 4K source material, do VFX in Shake or AE, and grade in Apple Color. Final product will be 2K master for film out. All on the same box and RAID.

So I am wondering how much storage I will need. 6TB, 9, 12, 16? Assuming a 10 to 1 shooting ratio, I imagine there will be 20 to 30 hours or more of 4K footage to cull from. Then there's the rendered 2K master and everything involved in the whole editing/VFX/grading process.

I admit it's hard for me to wrap my mind around the workflow that gets all this 4K footage from the RED (shot on CF) to the RAID, along with onset dumping to MBP and making back ups to LTO 3 tape.

I'm eagerly waiting for Mike Curtis to post his definitive workflow for features shot on RED, along with updated configuration recommendations with Silverado systems. But for now I will be satisfied with answering my RAID question. :biggrin:

 

[Edgar Pitts]

We are planning on at least 12 TB for a similar workflow. Our shooting ratio will probably more in the 20:1 range. We are looking at a small shared SAN setup as we will be doing everything except the final grade in house prior to filmout on multiple workstations.

 

[Mike McCarthy]

Basically it is all a big math equation. I hear that 4K at 24fps in Redcode is supposed to be 27.5MB/s. That is 1.6GB/min, and 100GB/hr. 20 to 30 hours will require 2 to 3TB of storage space for the source footage.

Then you will have to select an offline format, unless you plan to use the Quicktime reference files, which doesn't seem like the best plan yet. I don't recommend using DV or HDV, but I will use those numbers for the same of estimates. 20 to 30 hours of footage would require 250-360GB at HDV. DVCPro-HD would require four times that much space, 1-1.5TB. ProRes or Cineform would be bit more than that. 1080p doesn't make much sense as an offline for a 2K conform since it is very close. 720p might give you a bit of a performance boost, and SD will work if you already have the tools available. I definitely recommend editing in a native 24p environment, instead of adding and removing 30 frame pullldown. Regardless, budgeting 2TB for the offline gives you the flexibilityto do whatever you need.

Once you have an offline, you will likely want to export uncompressed 2K versions of the shots you used. Uncompressed 2K 10bit RGB at 1.85 aspect ratio is 200MB/s (DPX or Blackmagic 2K) , which is 12GB/min, and 720GB/hr. For a two hour movie, with short handles, you are looking at 2TB. From there, ideally your color system can take those files directly, and export directly to DPX for filmout, which is another 1.5TB. If you need uncompressed versions of a texted and textless, that will be one more set of uncompressed files requiring 1.5TB.

That is 3-5TB for the source and online, and 5TB for the finishing at uncompressed 2K. Note not all of these files need exist at the same time, especially if you have LTO-3 backups. Using a compressed finishing codec for finishing would lower the requirements as well. A 14 disk array with 1TB spindles, split into two 6TB Raid-5 arrays, would give you 12TB. 10TB that we discussed, plus 20% to minimize fragmentation etc. You want to avoid filling any drive to more than 90% of capacity, unless you are just using it to transport data. Working on that data with no free space will cause excessive fragmentation.

The lowere end solution would be a single array of 8 1TB disks, 7TB usable after Raid redundancy. Delete your offline source after you move to 2K, delete your (BACKED UP) source footage after your 2K conform is done, before rendering colored versions.

I make a lot of assumptions, and this is only one of many ways to look at it, but I hope this helps, and gives you a few ideas.

 

[Dan Blanchett]

Thanks, that is very helpful. For my budget, I was looking at 8TB, but now I may see about going up to 12. G-Speed XL was my first choice, though there may be better deals to be had.

 

[Mathieu Ghekiere]

Basically it is all a big math equation. I hear that 4K at 24fps in Redcode is supposed to be 27.5MB/s. That is 1.6GB/min, and 100GB/hr. 20 to 30 hours will require 2 to 3TB of storage space for the source footage.

Then you will have to select an offline format, unless you plan to use the Quicktime reference files, which doesn't seem like the best plan yet. I don't recommend using DV or HDV, but I will use those numbers for the same of estimates. 20 to 30 hours of footage would require 250-360GB at HDV. DVCPro-HD would require four times that much space, 1-1.5TB. ProRes or Cineform would be bit more than that. 1080p doesn't make much sense as an offline for a 2K conform since it is very close. 720p might give you a bit of a performance boost, and SD will work if you already have the tools available. I definitely recommend editing in a native 24p environment, instead of adding and removing 30 frame pullldown. Regardless, budgeting 2TB for the offline gives you the flexibilityto do whatever you need.

Once you have an offline, you will likely want to export uncompressed 2K versions of the shots you used. Uncompressed 2K 10bit RGB at 1.85 aspect ratio is 200MB/s (DPX or Blackmagic 2K) , which is 12GB/min, and 720GB/hr. For a two hour movie, with short handles, you are looking at 2TB. From there, ideally your color system can take those files directly, and export directly to DPX for filmout, which is another 1.5TB. If you need uncompressed versions of a texted and textless, that will be one more set of uncompressed files requiring 1.5TB.

That is 3-5TB for the source and online, and 5TB for the finishing at uncompressed 2K. Note not all of these files need exist at the same time, especially if you have LTO-3 backups. Using a compressed finishing codec for finishing would lower the requirements as well. A 14 disk array with 1TB spindles, split into two 6TB Raid-5 arrays, would give you 12TB. 10TB that we discussed, plus 20% to minimize fragmentation etc. You want to avoid filling any drive to more than 90% of capacity, unless you are just using it to transport data. Working on that data with no free space will cause excessive fragmentation.

The lowere end solution would be a single array of 8 1TB disks, 7TB usable after Raid redundancy. Delete your offline source after you move to 2K, delete your (BACKED UP) source footage after your 2K conform is done, before rendering colored versions.

I make a lot of assumptions, and this is only one of many ways to look at it, but I hope this helps, and gives you a few ideas.

Wow, thanks, very helpful post!
Nice to get an idea of how much it would take to finish a feature in this workflow.

 

[Dan Blanchett]

It's kinda crazy that uncompressed 2K takes up so much more space than the original 4K it's created from (which is obviously not uncompressed, but is "good enough"). I guess that's a testament to the REDCODE RAW codec and what makes this whole workflow feasible.

 

[Mike McCarthy]

You could use a compressed finishing format, like Cineform2K, and cut that number by a lot, but eventually you will need to render to dpx for a filmout. I just finished a conform with Cineform, and I have four full exports of the movie at 150GB each, (uncolored, colored, textless, texted) and then a 1.5TB folder of final DPXs for filmout, That's a 10:1 storage savings with compression, and good for archiving as well.

 

[David Newman]

You could use a compressed finishing format, like Cineform2K, and cut that number by a lot, but eventually you will need to render to dpx for a filmout.

Lasergraphics have confirmed they can do a film out directly from CineForm MOV (tested up to 4K,) saving you the DPX render/export. Other film-out houses are likely moving to support media files wrapped as AVI or MOV as it make good business sense, partularly at 4K where the data size is so large. CineForm 444 4K for a feature would likely fit on a single 500GB drive (as compared with 5 terabytes for DPX.)

 

[M Most]

Lasergraphics have confirmed they can do a film out directly from CineForm MOV (tested up to 4K,) saving you the DPX render/export. Other film-out houses are likely moving to support media files wrapped as AVI or MOV as it make good business sense, partularly at 4K where the data size is so large. CineForm 444 4K for a feature would likely fit on a single 500GB drive (as compared with 5 terabytes for DPX.)

To me it makes very little sense unless the material has been color corrected and screened with proper color management for a film preview. This is rarely the case when Quicktime files are sent to a film recorder. What is done in that case is that the material is put through a LUT in the film recorder that will attempt to simulate the look of the video on the film recording. The problem, however, is that the look of video is usually not what you want on film, and usually not the best rendition of the material on film. Not to mention that the look of the video usually hasn't been confirmed on any kind of industry standard monitoring in the first place.

Film recording is a step that should only be undertaken if it is necessary, and if it is, it should be done right.

 

[David Newman]

You are assuming standard video curves are applied to Quicktime files. There is nothing special about DPX file (other than their size), you can applied the same color processing to a 4:4:4 RGB QT. Add light compression compression and you have the convience of size. So "unless the material has been color corrected and screened with proper color management for a film preview" is assumed. So use the new generation of low cost DI tools that now support RAW/flat QT/AVI input and export through compressed QT rather than DPX.

 

[M Most]

You are assuming standard video curves are applied to Quicktime files.

No, actually I'm not. I'm assuming that many, if not most, people here and elsewhere read what you stated and immediately think "great, now I don't have to go to a DI room and go through all of that uncompressed stuff - I can just simply give them my Quicktime file that I generated in Final Cut and it will look perfect on film because it looks great on my Apple Cinema Display." Only it won't.

 

[David Newman]

Well then don't shoot down a perfectly good workflow, just state that a DI color pass it still required (well highly recommended), I'm just stating DPX is not required. This was in reference to Mike's (McCarthyTech's) post which says you need DPX for film out, when you don't.

 

[M Most]

Well then don't shoot down a perfectly good workflow, just state that a DI color pass it still required (well highly recommended), I'm just stating DPX is not required. This was in reference to Mike's (McCarthyTech's) post which says you need DPX for film out, when you don't.

David, I'm not shooting down anything. But the fact is that very few DI facilities I know of use a compressed, wrapped movie file workflow. So if you bring such a file into one of those facilities, it will likely be converted to a DPX sequence or something similar prior to recording, and you will likely be charged something for that conversion. That's not technical necessity, but it is practical fact. Now, if you're talking about the do-it-yourself approach, and what benefits are available by prepping the show the way you describe, I wouldn't argue that point. But it's also why I said what I did about proper color correction.

 

[Mike McCarthy]

My knee jerk reaction to David Newman's post was no state that anywhere you take a CF MOV file to be filmed out, will probably convert it to DPXs anyway before they print, same as scanning to CF 2K is usually actually done with DPX as an intermediate. But on closer examination, it looks like there is now a hardware Filmout solution that can natively accept compressed MOV files directly. Technically, anyone could print a file exported from their FCP timeline, but as always, it will only look as good as the time and effort that you put into the workflow.

I think it should be assumed that any suggestion made on these forums will always have a "provided that..." disclaimer of one form or another. CF MOVs can be filmed out PROVIDED THAT you use a lasergraphics recorder.
It can look good PROVIDED THAT you calibrate your LUTs and preview devices correctly, etc.