Basic information about drive speed

[Steve Kennedy]

I've been searching for ages for just some really basic information about drive (read/write) speeds required for working with RED footage, and I'm just finding little scraps of information here and there around the place. Does anyone know of a good resource for understanding exactly what kind of performance one needs from HDs depending on the various flavours/usages of RED one might be using?

In other words, what R/W speeds are required for:

1. Editing Scarlet/Epic footage in native R3D?
2. Grading Scarlet/Epic in Resolve etc.?
3. What different levels of debayer quality mean for drive speed?
4. What is the bare minimum at which one can edit R3Ds without an HD-induced lag? (the minimum)
5. What is the drive speed beyond which going faster makes no perceivable difference? (the maximum)

I'm finally switching from editing everything on a late-2011 Macbook Pro laptop, to having a nMP set up, and I'm having a hard time understanding exactly what my HD requirements are going to be.
Does anyone know of a good resource that actually lays out all the above information, specific to dealing with RED footage, as opposed to getting into one specific set of questions related to one individual's setup?

Thanks!

S

 

[BigLu]

1 what software for editing are you talking about, and you can option for offline online workflow which is a far more efficient way of editing.
2 graphics card is more important
3 RRX is the solution here not necessary hard drive speed if you want to skip the RRX then your kinda dependent on again graphics card
4 what software again and again working in an offline online workflow will give you much better speed and performance for your editing experience
5 uhhhhhhhhhhhhh lets get threw 1-4 then you can play with 5

I will do what i can to answer what i can, I'm sure as you answer these questions you'll get more answers.

 

[Steve Kennedy]

(I just wrote a long reply to this, and it disappeared...? So apologies if it reappears and this is a duplicate!)

1. I'm using Premiere Pro CC, latest version. The offline->online workflow would definitely be my preference, but I don't think it's possible in this case. I have around 30TB of material here – I decided to transcode a percentage of it in order to calculate how long it would take to transcode the whole lot. I don't remember the exact figure, but it was going to take something like 40-50 days (yes, DAYS) of continuous processing time on my computer to transcode everything. I need to get on with the edit immediately, so that's not an option. Next time, I'll make sure the footage is being converted by the crew as they're shooting...

2. I understand the importance of the GPU, and hopefully Adobe et. al. will be continuing to improve the way their software takes advantage of the GPU, and especially multiple GPU configurations. However, I have heard that aside from CPU/GPU tech, there is an HDD speed bottleneck to be aware of, and I am trying to understand what that is so that when I get my nMP in a couple of weeks I'm not hit by the sudden surprise that despite its awesomeness (3.0Ghz 8-core, dual D700s, 64GB RAM from OWC) the editing of native R3Ds is still laggy from drive speed. What I have been using up to now are 8TB G-RAID TB drives, but since editing raw is slow on my computer for CPU/GPU reasons (it being a 2011 model) I can't know how much of that is affected by the drive. Those drives give me about 200 MB/s write and 180 MB/s speeds according to AJA Datacalc. My question is, really, is that fast enough? Or do I need something like a Promise Pegasus2 R6, something with hardward RAID etc.? But beyond just a simple, "yes, you need that" I want to understand why – what speeds are required, what are available etc. etc.?

3. It would be a budgetary stretch to get a Rocket, so I'm hoping to find a way to avoid having to get one if at all possible.

4+5. Again, I'm just trying to understand the whole picture, the parameters involved etc. Offline-online is not possible right now.
Software: PP CC + Resolve.

Thanks so much for your time and help.

S

 

[BigLu]

PP allows you to play back at a reduced debayer for realtime ability, still might struggle not sure with your system.

Just a test try this and it will give you an idea if its worth the wait for you.
Take a 1 GB file exactly 1 GB take it into RCX Pro ver 30 make sure you go into your preferences and optimize your settings in system to take advantage of your computer configuration.

Transcode a file to and be sure to do the following settings
1080p ProRes 422LT
1/2 Debayer good

Take that speed into account and do your math to see if its something worth you doing for your offline online workflow.

Atleast put that idea to bed so you know its not worth it or perhaps worth it if you can sit on it for a few nights.

I do fear your not gonna be fully happy with the performance of your laptop working from the direct RAW files, maybe it will be acceptable to you but perhaps not.

What city and state are you located in, if you have over 30TB of footage to crunch threw it sounds like a job for RRX which are out there in the wild and you can find people whom will help you. If your close to me ill help you.

 

[Steve Kennedy]

OK, I ran the test, using RCX 30.
A 1GB file took 744 seconds to process to the codec you described. I was wrong about my total data amount, because the interviews have already been transcoded but I still have around 13.5 TB of video. Here is the math:

13.53 terabytes = 13854.7 GB
* 744 seconds = 10,307,896.8 seconds total
/ 60 = 171798.28 minutes
/60 = 2863.3 hours
/24 = 119 days

Which essentially means that I would still be transcoding long after the edit is due to be finished. I think we can safely put that workflow to bed for this project.

I do fear your not gonna be fully happy with the performance of your laptop working from the direct RAW files, maybe it will be acceptable to you but perhaps not.

In two weeks time I will no longer need to edit on my laptop, since our nMP will have arrived. I'm trying to make sure the new setup is going to be fast enough, specifically the drives.

What city and state are you located in, if you have over 30TB of footage to crunch threw it sounds like a job for RRX which are out there in the wild and you can find people whom will help you. If your close to me ill help you.

I am in Berlin, Germany for the post of this film. I certainly appreciate the offer, but think it would probably be impractical to do something like this? It seems like what I need is a fast enough system to work with the files myself... but perhaps I am wrong about that.

How fast, approximately, would one expect a 1GB file to transcode using a RRX, to the same codec as your test?

 

[Vico Martin]

Steve, before your nMP arrives, you can down the debayer Quality in RCX, transcoding to 1/4 Quality is enough for offline edit, and you can divide the total amount of days by 4 or more.

with 1/4 debayer and a nMP with a speed raid you can eat this 13.5 Tb in near 24h for sure.

 

[Blair S. Paulsen]

Since you are working in PremPro, and if you have little or no requirement of interchanging elements with VFX/etc, then why not just stay native? As others have noted you can just reduce the decode resolution and work in a "virtual" offline manner without having to burn the time to actually make low res offline files.

IAC, as the OP suggests, drive/bus speed is obviously desirable - but does it speed things up enough to justify greater investment? It really depends on several factors besides what format your material is to begin with. Thanks to RedCode compression, sustained read speeds in the 180-200MB/s range are usually sufficient for source. Will you be working in compressed formats like ProRes/DNxHD or (nominally) uncompressed file types like DPX/TIFF sequences? Is the hero deliverable 1080/2K or UHD/4K? Will you often be reading/writing simultaneously, like when transcoding? Keep in mind that when reading and writing to the same drive over the same bus, your speeds can drop significantly below spec due to bus arbitration and other overhead issues. If available, I often use a USB3 port for the drive(s) with the R3Ds and TB for the RAID I plan to write to - cleans up bus issues and performance is more linear.

Cheers - #19

 

[Jeff Kilgroe]

I would also suggest remaining with a native edit workflow when in Premiere. Offline / proxy editing is a valid solution, but also creates more work in this situation where the NLE handles your native files just fine.

Going back to the drive speeds, you really need 150MB/s+ for each stream of 5K~6K footage that you will be reading simultaneously. With Dragon and 6KFF, 200MB/s+ is preferred. It makes no difference which resolution or quality/ debayer level you play back the footage at, you still have to read the entire clip data. If you recorded Epic 5KFF @ 24fps and 5:1 compression, that is 130MB/s. Anything less than that and your playback is going to stutter. That is faster than any single mechanical HDD can consistently sustain. Most will burst faster and can come close to pulling it off if you start with them fresh and empty, but you really need a couple drives striped together to pull that off. If you have two streams going at the same time for real-time playback, perhaps dissolving between two clips, then you need to double that MB/s requirement, or be reading from separate drive sources. And that continues on, the more clips or streams you add. The system and software will do its best to cache and buffer where possible to help smooth this along.

Blairs questions/ suggestions above are all things you need to consider. And for a new Mac Pro, I recommend the 6-drive or 8-drive Pegasus2 Thunderbolt2 RAIDs or the 8-drive Areca Thunderbolt2 RAID. I have not been having very good luck with G-Tech products lately and their Thunderbolt offerings are terrible.

USB3 is 1/4 the speed of Thunderbolt2, but still has plenty of bandwidth for external drives for some uses. Output drives or reading/writing a couple simultaneous clips/ streams. All too often it's not the interface itself that's the bottleneck, but the drives specifically. Generally speaking, you can figure 85~120MB/s per drive compound performance for RAID-0 configurations. RAID-5 or RAID-6 configurations are more complicated and speeds vary with the number of drives and actual implementation. Speeds of your drives can plummet well below what you would expect if you have multiple operations of read/write coming from several places on the same system.