Any recommendations for hard drive configurations?

[James d Page]

Hello,
I'm building a Dual Xeon E-5 PC based Premier editing suit typical for short films and the like. So what hard drive configuration can you recommend.


Thanks James

[Benedict Heinzl]

a raid 50 would be the best I can think of.
fast and secure.

[James d Page]

I was thinking of windows/program disk, SSD scratch disk and a Raid data array. How useful is seperate SSD to set as scratch disk on a system with 48GB of RAM?

[Jeff Kilgroe]

I was thinking of windows/program disk, SSD scratch disk and a Raid data array. How useful is seperate SSD to set as scratch disk on a system with 48GB of RAM?

Personally I'm using a fast SSD for OS and applications and an array of 4 or 5 drives (RAID-0) as my primary workspace/scratch. Larger RAID storage connected externally is typically RAID-5 or RAID-6. A lot depends on bandwidth needs, budget, etc.. If you can get all your capacity with the internal RAID-0, then that's a great way to go. Make regular backups and you're good. RAID-5 (or compound like RAID-50, as in 5+0, or a stripe set of two RAID-5 volumes) is fast, but expensive and rarely more reliable than a simple RAID-0. Forget all the hype and IT doublespeak. RAID-5 will bite you in the ass if you start believing it's actually more reliable. And in situations where it is, I've found it to be much faster to restore my data from backup to a RAID-0 in most cases than it is to properly let a RAID-5 volume rebuild itself.

As for the 48GB of RAM... You don't want 48GB of RAM in a Xeon E5 system. The platform uses a quad-channel memory architecture, you will drop to dual-channel and cut your memory bandwidth in half if you install 48GB of RAM. You need to install 4, 8 or 16 DIMM sockets with identically matched RAM modules to get full quad-channel performance. Try to go to 64GB via 8 x 8GB. That's a good amount to start with and will give you full performance. Your 16 cores will thank you in apps that can actually use them.

[Alexander Mejia]

Personally I'm using a fast SSD for OS and applications and an array of 4 or 5 drives (RAID-0) as my primary workspace/scratch. Larger RAID storage connected externally is typically RAID-5 or RAID-6. A lot depends on bandwidth needs, budget, etc.. If you can get all your capacity with the internal RAID-0, then that's a great way to go. Make regular backups and you're good. RAID-5 (or compound like RAID-50, as in 5+0, or a stripe set of two RAID-5 volumes) is fast, but expensive and rarely more reliable than a simple RAID-0. Forget all the hype and IT doublespeak. RAID-5 will bite you in the ass if you start believing it's actually more reliable. And in situations where it is, I've found it to be much faster to restore my data from backup to a RAID-0 in most cases than it is to properly let a RAID-5 volume rebuild itself.

As for the 48GB of RAM... You don't want 48GB of RAM in a Xeon E5 system. The platform uses a quad-channel memory architecture, you will drop to dual-channel and cut your memory bandwidth in half if you install 48GB of RAM. You need to install 4, 8 or 16 DIMM sockets with identically matched RAM modules to get full quad-channel performance. Try to go to 64GB via 8 x 8GB. That's a good amount to start with and will give you full performance. Your 16 cores will thank you in apps that can actually use them.

In my expierence Raid-5 has been reliable for me. Even if You run into a situation where you have to rebuild it, It's still up and you still have good enough performance to play back video. Just make sure you get more speed than what is needed.

I used a Atto r380 attached to 8x 2tb 7200RPM drives in Raid5. It's been very solid and rebuilds only have to happen when I have bad drivers from my editing hardware to lock up the system. We edit 720P 60fps Uncompressed video, so 500MegaBytes per second is much larger than the the 140 MegaBytes per second needed to stream it back, so we still have RT playback during rebuilding.

[Jeff Kilgroe]

Yes, most good RAID systems are still somewhat functional during rebuilding. However, on heavily-used workstations this can be a huge time killer if you need the bandwidth to get the work done.

I've deployed so many RAIDs in the past few years, I lost count way back when. Have bought nearly 250 hard drives this year alone and have put all but a few reserved for spares into service. More often than not, I see larger RAID-5 and RAID-6 systems bite the dust because it's not always just a drive that fails. Even the ultra-reliable ATTO host controllers, which are my favorite ones, can still suffer catastrophic failure. I pity anyone who trusts a RAID-5 as a secure backup, seen that backfire too many times. For smaller arrays that are to be used for immediate online fast storage, I prefer RAID-0. If we're talking 5 drives or less in an internal array, RAID-0 offers the best performance with strong reliability. At the 6 drive mark, the advantage does begin to tip the scales toward RAID-5.

Another factor is the use of the system specifically. For server operations, I almost always go for RAID-5 in an attempt to afford the least amount of down-time. Backups still performed religiously. In performance workstations it'a all about just that, performance. If my primary workspace was crippled for the next 10+ hours for a rebuild where I was doomed to only working at 100~150MB/s, I would probably hurt someone. Nope, screw it, just swap the drives, set up as a new RAID-0 volume and transfer the latest backup of the stuff I need right now. Typically, all my current work is contained on my primary storage and on an external RAID as well. If one goes down, my other copy of it is never more than a couple hours old anyway.

When we have RAID failures here, it's due to failed drives about 60% of the time. That's a rough guesstimation, but I'm sure not too far off. The rest of the time it's due to other factors like a power supply blowing or other power anomaly. Not that unusual to have an HDD go bad and it can generate an EMF or electrical pulse that can damage the controller and/or adjacent drives or other components.

RAID-5 is an option and everyone should consider all available options against their needs. For our datacenter purposes, RAID-6 rules the roost. For our workstations, all our internal RAID volumes are RAID-0 because it's about performance and what we can get back up and running at full performance in the least amount of time.

All our external RAIDs on workstations are RAID-5, a couple large ones are RAID-6. Most of them with more than 6 drives are reserving at least one drive as a hot spare.

[James d Page]

Thanks for the advice,
The 48GB is because CPU 0 can have 2 banks and CPU 1 has 1 bank of quad DDR3 at so (8+4)X4GBs of RAM, all the same brand type and spec.
For the main Media Array I think Raid0 4 to 6 internal disks, another advantage of RAID 0 over 5 is reduced latency.

So given an editing system with plenty of RAM, what are the pros and cons of SSD in practice . Clearly there fast but also small.
For program disks is the advatage in boot time, what sort of advantage do you see in operation.
Allthough I have heard people go on about SSDs scratch disks, Is there much advantage when there is loads of RAM? inparticlar when the the SSD is not good few times larger than the amount of RAM?

[Alexander Mejia]

Lets say that your Raid-0 gets hosed and you need to restore from the religious backups you're making: How long is that restore going to take if your copying over 5TB of data? Maybe 8TB? Even if you're doing a network restore, 1gbps is not that fast for transferring that much data. Do you have to unplug your red rocket or Black magic card to make room for another raid controller so your not down for some 3+ hours?

[Jeff Kilgroe]

Of course, restoring a backup takes time as well. The advantages of the RAID-0 for smaller RAID volumes, primarily internal workstation storage, is the increased speed and faster response times. Drive replacements don't require lengthy rebuilds that slow your storage space to a crawl. If your backups are easily accessible and nimble enough to select what you need when you need it, then recovery times are fast in critical situations. I can restore from LTO-4 tape at about 100~120MB/s, LTO-5 is twice that fast. Most systems here have both internal and external storage, so recovery time is usually as fast as moving from one RAID to another, typically 450MB/s + or faster.

Mostly I was pointing out that RAID-5 often gives people a false sense of security. Using a system like RAID-5 gains increased redundancy, but at the expense of performance and capacity. That redundancy only works to a certain extent, typically one drive out of 3~5 drives can fail and the system can still recover. In the event of failure, if more than the safe proportion of drives are affected, you're no better off than you would have been with RAID-0. In many instances, failures are due to issues with controllers, cabling and/or power. Drive failures can commonly lead to cascade damage. We see lots of instances and reports of consumer or "prosumer" RAID systems where people are constantly fighting with regular drive failures. The drives are most often blamed, but if the issues are persistent, there's probably something else going on. Only a matter of time or the stars aligning before serious data loss or corruption sets in.

Corruption also lends itself to another topic of discussion. RAID-5 systems are nearly as susceptible to intermittent data corruption as a RAID-0 volume. If you want increased error control, it's better to move to RAID-6 or comparative RAID-1+x.

FWIW, we have encountered far fewer failures with our RAID-0 than with smaller RAID-5 systems. Many of the cheaper RAID-5 systems via onboard host controllers, units from CalDigit, G-Tech, Promise, etc.. often generate failure notices and lockouts or forced rebuilds from what amounts to controller errors and SMART monitoring false alarms. We still prefer RAID-5 for small external units, even with these issues. RAID-0 is almost always reserved for internal RAID volumes with the occasional use in non-critical volumes having 4 drives or fewer.

Depending on which system we're working with, the better options for recovery will differ. There have been times in the past where swapping PCIe cards for various tasks was a requirement with the Mac Pro and some i55xx series Xeon PC's. They just don't always have enough lanes/slots to go around. Swapping cards and rebooting is normally a trivial process, or we aim to keep it that way. Although for some users or configurations, it's not always so. Last card-swapping workstation I had here was my Mac Pro being used for Resolve. When time came to make a full backup of the system -- more than what was automatically backed up over the network, I had to pull the second Quadro 4000 GPU card and swap it for an ATTO SAS controller so I could dump the internal contents to LTO tape. That was only a temporary solution though that was necessary after I sold off my Cubix PCIe expander box and a few other things. My primary Resolve station is now an HP Z820 and I have everything installed that I need in that box.

[Alexander Mejia]

I just wanted to add that a battery backup is ESSENTIAL to having a raid. Its maybe only 5%-10% the cost of your raid setup, but it will save you so much TIME, due to power outs that could hose a day trying to rebuild a raid during a power outage.

We just had a power out and I was still running, so its really worth the extra few bucks.