Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Networked 10gbps RAID 60 for back

E

Eric Stracke

Member
Joined
Apr 7, 2015
Messages
9
Reaction score
0
Points
0
I'd like to upgrade my backup solution to something... a little more robust than the spinning disks I have sitting on shelves. I'd love to get anyone's thoughts on this route I'm thinking about taking.

My idea was to run a 4,1 or 5,1 Mac Pro tower with an Atto R680 with a 14 drive RAID 60 with 4TB WE SE drives. This machine would connect to my network via 10g Ethernet and live in my server room (closet with a thermostat controlled exhaust fan).

This storage pool would be my onsite backup. I'd like to hold out for LTO 7 for my off site backup which is currently just spinning disks on shelves.

Does anyone one have any words of wisdom for a backup storage pool like this? Is raid 60 a good idea for something like this? I'm really just after a large, redundant, fast storage pool. I won't be editing off this networked storage pool but I would like very fast transfers.

Im all ears for any words of wisdom before I pull the trigger. Thanks.
 
According to these probability formulas from DELL, each 7-disk segment of your RAID60 would have an annual probability of failure of 2.57% (assuming 10% annual drive failure rate at 100% duty cycle). Since the failure of RAID0 is the probability product of the elements, your 14-disk RAID60 would have an annual probability of failure of 5%, or a 23% chance of failing within 5 years. On the other hand, if you mirrored the RAID6 segments, then your chances of a failure drops to 1.3% per year, or 6.2% in five years.

So, are you feeling lucky?

Here's something counter-intuitive (to me): a simple RAID6 with 14 disks has an annual expected failure rate of nearly 15%. Over a five year lifespan, you have better-than-even odds (nearly 58%) that the RAID will fail unrecoverably.

Now, perhaps DELL is using overly pessimistic numbers about intrinsic drive failure rates. I've certainly replaced my share of disk drives over the years, so I know that drives do fail. I would think that if you are serious about backup, err on the side of mirroring rather than striping your RAID6 sub-arrays.
 
Thanks Michael, that it some great insight on RAID reliability. I'll have to consider mirroring instead of stripping. I'm not wild about the loss of storage space but I agree it is safer.

Here is a question - is it possible to stripe 3 RAID6 arrays? Say, three 6 drive arrays? I've never heard of anyone doing that before, is there a reason for that?

Can anyone speak to real world transfer speeds over a 10g network in a scenario like this?
 
Eric Stracke said:
Thanks Michael, that it some great insight on RAID reliability. I'll have to consider mirroring instead of stripping. I'm not wild about the loss of storage space but I agree it is safer.

Here is a question - is it possible to stripe 3 RAID6 arrays? Say, three 6 drive arrays? I've never heard of anyone doing that before, is there a reason for that?

Can anyone speak to real world transfer speeds over a 10g network in a scenario like this?
Click to expand...

In principal, and when it comes to software-defined arrays in Linux and OSX, you can have as many striped elements as you wish. (I have no experience with Windows.) Getting stuff to be fast over 10Gbe is all about the OS. Linux has been doing it for years (thanks to the scientific supercomputing community). Back to the subject of UREs (unrecoverable read errors), read this blog posting about how you cannot just rely on RAID anymore, you need integrity-checking filesystems. Remember, backups to tape are not just about tape media. Competent backup solutions do comprehensive checksumming and support pretty detailed data recovery (provided you don't actually shred the media). RAID is simple and fast, but it needs help for the super sizes of disk drives coming down the pike. Again, Linux leads the way in filesystem technologies and storage systems that sit higher in the stack. If all you want is backup, that's overkill compared to using BRU with LTO6. But if you want an active archive, read up on CEPH and GLUSTER to get a handle on how Amazon, Facebook, Google, and others think about large-scale storage reliability.

I should also add that when it comes to creating robustness, there's something to be said for complementary layers vs. Just More. Concrete is strong under compression, but weak under tension. Steel is strong in tension, but not as strong in compression (it can buckle). Steel-reinforced concrete gives you the best of both worlds. If you are trying to address a fundamental weakness in disk drive technology, you can only do so much by having more and more disks. Having a complementary layer like tape backup strengthens the solution where it's weakest.
 
Last edited:
Having built about 750tb or so of networked storage in the last year, I highly suggest you use ZFS, which you can run nicely on Linux. Also, you can do this all with very inexpensive hardware. Don't get a Mac. Get some inexpensive hardware with lots of bays, a half-decent CPU, but lots of ECC RAM. ZFS is perfect for lots of drives, and easily-expandable down the road.

With ZFS, you can have one giant "pool" of storage, but it's made up of several smaller arrays (vdevs). You wouldn't just slap 16 drives together, nor would you mirror them, nor would you do both. You'd have a pool with several vdevs, each one being around 4-6 drives. Each one of those vdevs would be its own RAID5 or RAID6 (raidz1 and raidz2 in ZFS land). So even though you may have a total of 12, 14, 18, whatever hard drives in total, you're much better protected than any traditional RAID level or stripe+mirror, because you can now suffer 1-2 drives of loss from each of your smaller vdevs, not in total. Performance is still extremely good, even when using raidz2 (RAID6), which typically has a bad reputation for speed. Beyond that, you get all the fun ZFS benefits, like never suffering from data corruption, on-the-fly compression, and drive rebuilds happening in a fraction of the time it takes on other RAID systems.

Oh, and another great thing: Some of it sounds complicated if you haven't used ZFS or something like it before, but it's actually remarkably easy to use, set up, and maintain. A single command can set up your array, and a single command can tell it to checksum your entire pool, and fix any errors it finds. The cost and difficulty of a system like this are remarkably tame, and the reliability should put you at ease. I built my first ZFS storage server knowing almost nothing less than three years ago, and to this day it's handling multiple people editing RED footage off it over 10GbE simultaneously and has never suffered an issue.

Other tips: If you're using ZFS, you don't need (and actually it would be bad to have) a RAID controller. You're best off with a couple fast HBAs, and even the nice ones should be cheaper than your RAID cards.

10 gigabit -- specifically 10GBase-T -- is the way to go. The Intel X540 cards aren't too expensive. There's far cheaper options out there these days, too, but I haven't bothered to try them. Keep your existing ethernet cable. You can do a direct connection to the machine(s) that need access, or if you need a switch, there are some inexpensive ones that can handle 10GbE that still don't suck.

Haven't used those WD drives. Lots of WD Red drives, but mostly the 4TB Seagate NAS drives in fast storage systems, and recently built a giant rig with 288TB worth of Seagate's new 8TB drives. They aren't very fast, but it's for infrequently used storage -- and despite that, given the nature of ZFS and how many drives are in that server, it's still managing to read/write at a couple gigs/second. But whatever drives you want, as long as they're known to be reasonably reliable and not overpriced.

This reply is becoming a sizable book now, so just feel free to ask about anything else if you feel so inclined.
 
I've long lusted for a 10GigE network, but when I ran the numbers a couple years ago the entry cost was high and sys/admin had a rep for being a pain. I've been hoping that ThunderBolt networking would become well supported and work with things like laptops that don't have a slot for a 10GigE NIC. Perhaps I need to do my research again based on current products and admin tools...

Cheers - #19
 
10Gbe was only high because of the cost of switches. Current the Netgear switches are far cheaper than the industry standard manufacturers for Corp IT. Even the 24 Port Netgear 10Gbe switches are 5K. Compared to Fiber 10Gbe is far cheaper to implement now. TB networking will never offer the expand ability and distance options. It's really just meant for small workgroups and limits other expansion. BTW Dell's numbers on raid 6 are not accurate. I have yet to see an entire raid 6 collapse from corruption in my entire time in the tech industry as long as the parity verification are run weekly or biweekly. You do want Enterprise drives though if you go that route. NAS drives have a much higher error rate and likelihood of bad blocks from the manufacturer. If you use the NAS drives make sure you write zeroes to every disk before you build the raid which will take quite some time. If you go the ZFS route makes sure you have a 2nd system ready to go in case the primary host fails.
 
With 10GbE, you can use regular CAT6 or CAT5e [within reason] ethernet cables, and the NICs cost under $500. As Eric has mentioned, there are now cheaper switches now, such as this Netgear, that work pretty well. If that's still too expensive, and if you only have a couple machines that need to talk to the server, you don't even need the switch -- you can run a direct line from the server to each client. This is often even preferred, but obviously if your number of clients grows, this can get unmanageable very quickly.

I disagree about buying enterprise drives. They're quite expensive, and they do last longer, but NAS drives are designed to have very long lives as well (compared to standard desktop drives) and I am not convinced that the savings in the long-term are way better. Sure, they're nice, but they are by no means essential. If you have a good RAID solution set up (such as the multiple raidz2 setup mentioned above), then rebuilding after a failed drive is a slight inconvenience, but not something to fear or spend thousands of dollars on drives to stave off a bit longer. Here's a nice hard drive reliability post from Backblaze, a storage company that goes through hundreds of thousands of drives and thus have some pretty good stats. (They like the Seagate 4TB and HGST 4TB drives for the cost to reliability ratio. I mostly use the former.)

Most of the time, the cost to build a storage system like this is the biggest hurdle for people, so I like to recommend the least expensive path to a great system that will fulfill their needs and have room for growth, even it it's the absolute best stuff on earth. Most peoples' needs just don't come close to what is available put there.
 
I really like the idea of ZFS on freenas. A buddy of mine suggested I go that route but I got cold feet.

Do you guys have any recommendations for builds for about 48TB?

I don't plan on editing off it but I would like transfer speeds of 500MB/s or more over 10gbe Cat6a.

On the lookout for any good resources. Thanks.
 
Do you have a budget in mind? Or any specific needs not mentioned? ZFS is totally the way to go. Definitely an abundance of hardware options out there, just like when building a desktop PC, so anything that might narrow it down. I've been meaning to write some guides for this sort of thing.
 
Daniel Keywan Hollister said:
With 10GbE, you can use regular CAT6 or CAT5e [within reason] ethernet cables, and the NICs cost under $500. As Eric has mentioned, there are now cheaper switches now, such as this Netgear, that work pretty well. If that's still too expensive, and if you only have a couple machines that need to talk to the server, you don't even need the switch -- you can run a direct line from the server to each client. This is often even preferred, but obviously if your number of clients grows, this can get unmanageable very quickly.

I disagree about buying enterprise drives. They're quite expensive, and they do last longer, but NAS drives are designed to have very long lives as well (compared to standard desktop drives) and I am not convinced that the savings in the long-term are way better. Sure, they're nice, but they are by no means essential. If you have a good RAID solution set up (such as the multiple raidz2 setup mentioned above), then rebuilding after a failed drive is a slight inconvenience, but not something to fear or spend thousands of dollars on drives to stave off a bit longer. Here's a nice hard drive reliability post from Backblaze, a storage company that goes through hundreds of thousands of drives and thus have some pretty good stats. (They like the Seagate 4TB and HGST 4TB drives for the cost to reliability ratio. I mostly use the former.)

Most of the time, the cost to build a storage system like this is the biggest hurdle for people, so I like to recommend the least expensive path to a great system that will fulfill their needs and have room for growth, even it it's the absolute best stuff on earth. Most peoples' needs just don't come close to what is available put there.
Click to expand...

The Timeout feature is really the primary reason to get the enterprise drives. The network series such as the WD Red drives have that as well however the cheaper Red versions are variable spin rate versus 7200RPM. The Red Pro's are the same price as the Enterprise drives so you might as well get the enterprise. If you dont have the timeout feature on the drive firmware and the block repair takes longer than the controller expects for the drive it will mark the drive out of the array as bad. ZFS wont change this. Be careful how many drives you use without the timeout feature in any parity or block based redundant raid even software. The amount of times you will deal with rebuilding squarely depends on the amount of bad blocks those drives have and how long it takes for those drives to repair those areas or mark them out after failed repair. The drive manufacturers ship new drives with up to 1% bad blocks. Considering a 1TB drive has millions of blocks that number is large. The article you link does not reference bad block raid situations and I have been tracking drives failures for 8+ years and seen the fluctuations in the market. If you build a ZFS array you are on your own on support.
 
Last edited:
Thanks all.

Daniel, I'd love to hear some of your thoughts on a build. I was hoping to keep it around 4-5k including 48tb worth of WD SE drives.

Eric, from your last post it sounds like you wouldn't recommend a ZFS based freenas. What else would you recommend for a 10gbe networked backup storage pool?
 
ZFS has it's benefits and I dont recommend not using it. I am just stating there are some considerations if you use it. The benefit of Hardware Raid 6 if handled well is the migrate-ability to other systems if you have hardware failure and the ability to scale significantly with SAS and SAS expander units. Raid 6 if maintained correctly via parity checks with enterprise disks that have the timeout feature are extremely robust to corruption and I have yet to see an NTFS raid volume completely collapse. I have seen many raid 5 volumes collapse from corruption and even one raid 6 HFS+ volume on a Mac Pro. However the organization was not maintaining their raid volume when the HFS+ volume collapsed. If you go the ZFS route drive failure is less of a concern with the software raid. However migrations becomes more difficult if you have system hardware failure occurs that brings the system down. And you have to know what you are doing with the ZFS raid config where the SAS hardware raid is far easier to setup and maintain without having such knowledge. Expanding ZFS raid volumes also involves another ZFS box once you exceed the first storage node. So long term the ZFS system will be more expensive than SAS.
 
Thanks Eric. Do you have a good resource you can share about properly maintaining a raid system like that?
 
The reference would be the manual for the raid management software since each manufacturer has the parity verification option or schedule parity check listed listed differently. For example the Areca controller management has a schedule parity check option that lets you select daily, weekly, monthly but not a specific time. You have to select how long the computer is idle before it starts. LSI/Intel raid management software lets you select the day and time besides how often the check occurs. Both have the option to manually do that instead but then you are reliant on someone doing it themselves ever week or 2 weeks which should be the standard time used by everyone. You want to view the logs that report parity errors after the verification completes. If the log shows any more than a few errors found and repaired then at some point you want to run a low level check disk in Windows to verify the integrity of the partition info, index, and data. OSX has a similar utility in disk utility. If a drive starts dropping out of the raid more often due to bad blocks then you want to replace that drive relatively soon with the spare drive on hand and rebuild. I would suggest a parity check though before you replace the drive unless that itself forces the drive to drop out again. If you have time you have to have your replacement spare drive already wiped clean with zeros to mark out any bad blocks from the factory. When you replace the drive and start the rebuild write zeros to the drive that was dropping out to bad blacks. The drive may still be fine but just had more blocks go dark over time. That should give you an idea of what to do overall for maintenance. The final item is drive roll over. Desktop drives reliability start dropping after 3 to 5 years. Enterprise drives should be 5 years but in reality the reliability is 3 to 5 as well. Do not use drives beyond 5 years for any raid holding critical data that is used on a daily basis ie production storage. Backup storage may be ok after that time but I would not store anything that is critical and not backed up elsewhere like LTO tape if the drives are beyond 5 years.
 
Super helpful, thanks Eric.

I've got a follow up question, is it possible to move whole RAID systems between computers? Or even from a Mac over to PC? I'd like to build this system now but just add it to main edit station and build a computer to have on the network to host the raid at a later date.

Currently I'm using 8 drives on the atto R680. Could I add a sas expander in there so I can accommodate my new raid60 and not break my current 8 drive raid?

thanks for all the help! I'm really eager to start on the backup system!
 
Yes you can move the raid controller and storage between systems as needed. However if you move the Raid volume from a Windows System to an OSX system then you have to have software in the OSX system to read and write to the NTFS partition. If you move from OSX to Windows then you need a software to read and write to the HFS+ volume. I would suggest though that if you do this you remove the data from the drive and reformat to the native partition type for the host system. That would be better for long term data integrity.

The SAS expanders are normally built into the storage units if they are external. If internal then yes you have to add your own. However with the amount of drives expanding beyond 8 I would not recommend putting them internal with the system due to power and heat generation in a workstation also used for HPC/editing work. It's also easier to move the storage to a secondary system and maintenance drives in the future if the SAS is external. So if you plan to expand storage later an external SAS storage unit would be more ideal.
 
You must log in or register to reply here.
Back
Top