Need advice about 96TB RAID storage

[Tom Lowe]

I need to install 96TB of RAID storage here at my house, with fiber connections running to six different rooms in the house.

What type of storage hardware would you guys recommend? Can I just buy a 96TB server from Dell or something? What RAID config should I use? RAID 5?

How will the server connect to my PCs via fiber? Is there some kind of PCI card for fiber?

Thanks for any advice you can offer! I don't know much about this subject at all.

 

[Cail Young]

Hey Tom.

If this is for editorial, fiber might be overkill if you are not working with uncompressed DPX files.

We've been using Editshare systems for editorial, which scale up to that size and run over Gigabit Ethernet or 10Gig Ethernet.

Alternately there are many other providers who offer Ethernet and Fiber-Channel systems.

Sounds like you may not be 100% sure what you need exactly, so here are some questions:

1) How many seats (active simultaneous users)
2) What throughput per seat / total is required? (i.e. what codecs and how many streams per seat / overall)
3) What level of share control is required? (Free for all, managed storage, some sort of hybrid)
4) What's your budget? How much redundancy do you need?

5) Why 96TB?

 

[PatrickFaith]

....
We've been using Editshare systems for editorial, which scale up to that size and run over Gigabit Ethernet or 10Gig Ethernet.

If you can swing it, the 10g is way better then the 1G. 10g has really come down in price and almost all the software works with it just like it was normal ethernet ( i have a small 6 server system that runs it, is amazingly fast, and all the software just "works" as if it is a normal ethernet so you really don't have to think too hard when managing it). 10G in many cases runs faster then fiber-channel (a $15k 10 g switch runs at 600 gigabits per second ... is so so nice, also has options for 40g).

 

[Tom Lowe]

1) How many seats (active simultaneous users)

Maybe four at one time.

2) What throughput per seat / total is required? (i.e. what codecs and how many streams per seat / overall)

4K DPX, 4K Cineform, and R3D 5K and 6K mostly.

3) What level of share control is required? (Free for all, managed storage, some sort of hybrid)

I'm not sure about that. People need to be able to contribute and to read, with maybe two admins who can delete?

4) What's your budget? How much redundancy do you need?

We have money. We also want to keep an identical RAID at our office, so then one of the quetions is, how to keep them synced? Our home internet connection is only 30 megabit, so the files will need to be hand carried between the two servers.

5) Why 96TB?

Well that is a number I sort out picked out of the air. Just a guess, but eventually, I am sure we will need more.

 

[Tony Lorentzen]

You should definitely check out the CalDigit HDPro24, which has 24 drives configurable up to 72 TB per unit. Their SuperShare solution allows connecting many clients as a SAN. The great thing is that their solution has 20Gb/sec connection (up to 100M) between clients and the drive - that's quite a bit more than the usual 10Gig ethernet (which has quite a lot of overhead) and the 4 or 8 Gbit/sec fibre channel (with almost no overhead). Fibrechannel allows for really long "cables" though. Each HDPro24 can deliver up to 1600MB/sec which should be enough for 4K DPX (which needs around 900MB/sec), but it won't be fast enough for more users to work on 4K DPX - for that you would need to add another HDPro24.

 

[Mathieu Marano]

I just installed a 48TB Qnap NAS for a similar purpose. We have 3 designers working on the same 4-5K RED footage and sharing projects. The Qnap TS-1679U comes with 4 GigE ports but I also installed 10GigE cards in it. MacPros are direct attached to the QNap. They have Smell Tree 10Gig cards.

I know you van get to 64TB with a single unit but you can eassyly add mor thru expansion or eSATA 6 ports. Oh, and il costs less than 10 000$.

http://www.qnap.com/useng/index.php?lang=en-us&sn=862&c=355&sc=703&t=2330&n=10235

 

[Tony Lorentzen]

Yeah, Mathieu. That's a nice solution unless you need to work with 4K DPX files, which is what Tom needs to as far as I understand. I'm not into QNAP myself, but rather Synology who has a similar solution for that too.

Tom, if you want to save a buck or two (and who doesn't?) then you might want to take a look at the iStoragePro hardware. It's basically the same as Maxxdigital and Light Iron is using in their products except you can build a system like theirs for almost half the dough in some cases. The Enterprise systems from iStoragePro has two 4 Gbit fibrechannel ports which should be able to provide you with the ability to work with 4K DPXs. Check it out: http://cidesign.com/products/special-order/Enterprise/Enterprise-Fibre-Subsystem/

 

[Ketch Rossi]

Maxx Digital, has so far done the best in both Pricing/Quality of products as well as standing behind its products for service, as they did so in their most recent sale at Sony Entertainment, going out of their way to bring extra Drives and working overnight to make sure their client had what they needed.


This to me is more important then having you save few bucks, which I know is the the issue with your employers, so I'll go with Ron, and have Maxx Digital build a couple of Raid 5 or 6 towers for you guys, so that you can add them to the one I brought to Dubai last November.

 

[Tony Lorentzen]

If you (or one of your guys) want to build something yourself then you could get one of the cases from XCase in the UK: http://xcase.co.uk like the RM 414 which is a 24-bay rackmountable chassis with a SATA-3 6Gb/sec backplane. Drop in a good quality server grade motherboard, an LSI RAID controller and a couple of 10GigE or Fibre channel cards. Maybe throw in a couple of Red Rocket cards and use it as a MONSTER render machine!? :-D

Drop me a line if you need someone to build something like this for you ;-)

 

[Tony Lorentzen]

Hey Ketch - no doubt buying the Maxxdigital has the major advantage that it comes with great support. Some need that and some of us are techies to a certain level that we can live with saving the 50% and configure stuff ourselves. I have absolutely nothing against Maxxdigital at all, just providing valid information.

 

[Mathieu Marano]

The Qnap would have about the same performances as the CiDesign. The QNap has beefier CPU and can be loaded with RAM. I think 10GigE could easily compare with dual 4GB fiber.

 

[Matt Gerard]

We are ordering a test unit of one of these. Right now we have a 48TB facilis Terrablock with 4g fibre running to 4 edits suites. Its been working well, but the volume sharing feature drops the overall speed a bit to much for our tastes. Right now we use it as just a big hard drive, and divvy it up in 8TB chunks to the rooms as they need it. Its been pretty darn stable, we replaced a couple of drives when they reported errors, and we don't even notice the rebuild process. But, we needed more space. So we will probably be selling the terrablock if anyone is interested We've got about 65k into it with upgrades and the service contracts.


This one is 10g ethernet. and quite expandable. We will need new HBA's, and they aren't cheap either. But we will see what the data rate tests come back in as. The biggest this with anything e-net is that it is open sharing, no file locking to deal with. As long as we keep our workflows in line we won't have any trouble with that. We will see how it goes...

http://gblabs.com/products/Space/

 

[Jeff Kilgroe]

Speed is important. Anyone who tells you it's not important is delusional. Most all SAN infrastructure is IP based or operates off a similarly structured topology -- fiber channel, infiniband, ethernet, etc.. So it's easy to balance loads, but latency is going to be much higher than local storage, and overall efficiency of speed is quite a bit lower. Unfortunately the down-side to it all is that real speed costs real money. As speed increases linearly, the price increases exponentially.

For "off the shelf" solutions, the CalDigit SuperShare is great. For 4 connected stations and ample storage, it's not too bad cost-wise and you can sustain 5Gbps simultaneously to each of the 4 stations from one SuperShare switch, even more if they're not all hitting the SAN at the same instant. Theoretically, anyway. Each switch is 20Gbps internally and all connections are 20Gbps. CalDigit's implementation is PCIe based, similar to InfiniBand or Thunderbolt, but it's proprietary and uses fiber cabling. I really like CalDigit stuff, but their RAID management isn't quite as robust as some of the others out there. In my experience on setups with 3 or 4 workstations, it's pretty fast.

4Gbps fiber is another alternative and not a bad way to go, host cards are getting cheaper, but the switching hardware is still expensive. Lots of companies handle this sort of stuff -- LaCie, MAXX Digital, Dulce', and a million others... Most all of them use off the shelf controllers and what you're paying for when you decide on a provider is not so much the hardware (as it's the same from everyone other than the style of box they put it in), but you're paying for their support and service. To second what Ketch has said, MAXX Digital is pretty good. The advantage to going with fiber is that you can get switches with much higher internal bandwidth than what you get with CalDigit. Some switch implementations also use load balancing and cascade overflow design, so you can offload bandwidth from one switch to another with only a small penalty in latency. There are fiber switches out there with tons of ports and 80 or 100 Gbps internal bandwidth.

10G ethernet is another good connectivity option, but a few companies have stranglehold on the patents and licensing for it, keeping the prices very high. This is why we haven't seen 10GE ports integrated into server motherboards, adapters are still $475+... Also why Intel has bought Infiniband and launched Thunderbolt, 10GE is expensive and difficult to implement due to market "politics". You'll find it's not so much the adapters that are expensive, relative to fiber, but the switching hardware prices are in the stratosphere.

10GE, Fiber and Infiniband all let you bond or run aggregate links. With fiber it's very common to run dual link for 8Gbps per node connection.


Being in Dubai, I'm not sure where you would do best to look. The ones I listed might be a starting point, but I would think that in the giant futuristic playground that is Dubai, someone somewhere should be able to set you up.


I haven't looked at Dell's or HP's options for this lately, but they do have solutions -- more expensive than the companies that do only this.

For drive configuration, with that amount of storage, you will be running RAID-6 most likely and will probably keep a few drives as "hot spares" so they can automatically be brought online in case of a failure.


Be sure you have a good backup and archival solution in place too. No matter how secure you think your RAID is, I've seen too many of them vaporize all their data if a power supply goes and cooks a few drives. Or a controller goes and cooks a few drives. Or a drive goes and takes two more with it, along with the controller.... Yeah, nothing is ever as secure as we would like it.

 

[Björn Benckert]

You should definitely check out the CalDigit HDPro24, which has 24 drives configurable up to 72 TB per unit. Their SuperShare solution allows connecting many clients as a SAN. The great thing is that their solution has 20Gb/sec connection (up to 100M) between clients and the drive - that's quite a bit more than the usual 10Gig ethernet (which has quite a lot of overhead) and the 4 or 8 Gbit/sec fibre channel (with almost no overhead). Fibrechannel allows for really long "cables" though. Each HDPro24 can deliver up to 1600MB/sec which should be enough for 4K DPX (which needs around 900MB/sec), but it won't be fast enough for more users to work on 4K DPX - for that you would need to add another HDPro24.

We got SAN / fiber here.. but playing back 4k DPX at multiple stations simultaneously will not be done without framdrops, lag etc... Other than that it's a nice system all machines run fine on it together, PC, MAC and Linux.

I think we paid about 60k USD for a sever and 7 cards / seats.

 

[Peter Moretti]

I have no experience build or buying servers. But a name I hear thrown around a lot here is Omenon.

 

[Tony Lorentzen]

We got SAN / fiber here.. but playing back 4k DPX at multiple stations simultaneously will not be done without framdrops, lag etc... Other than that it's a nice system all machines run fine on it together, PC, MAC and Linux.

I think we paid about 60k USD for a sever and 7 cards / seats.

Do you have CalDigit or something else?

 

[Jon Thomasberg]

FWIW, We have multiple QNAP NAS units with the Intel X520-T2 dual-10Gbe cards. I am about to try iSCSI on them but I have found out the hard way that Windows has a issue with handling single-threaded file-based storage via CIFS shares, AFP on macs is much better, but nonetheless has issues sustaining 4k or 5k footage without dropping frames.

Also, I bought the 1st 1679 in the US and while nice, it had problems with creating a RAID6. The problem on it was not resolved, but QNAP support was excellent. I told them that we were on a tight deadline and was impacting my biz. They worked with me on getting a replacement EC1279 unit out next day. Synnex (the distributor I went through) on the other hand -- not so much. They took over a month to RMA the defective unit.

But I can attest that going the NAS route, even over 10Gb with jumbo frames enabled has its share of issues. I personally will be looking for a better solution shortly. Like I said, I will be rebuilding the NAS to use iSCSI rather than NFS now. If that doesn't work I will be going the block-level storage route to bypass the less-than-stellar TCP stack in Windows.

Bottom line, as of now I would not suggest going NFS over 10Gb. And by the way, for those that don't know: port-channeling 2 10Gb ports together using something like LACP works fine for switch link aggregation if you have multiple workstations hitting the storage, but doesn't do anything for per-workstation I/O. I still only end up getting ~800Mbps network utilization (1Gbps - overhead) over my 10Gbp link. Tried 2 cat6a crossover cables direct from WS to NAS and configured LACP on both sides = no diff.

If anyone like Tony L or Jeff K have insight into a balls-blazing fast solution to this, please post response or PM.

 

[Ketch Rossi]

Hey Ketch - no doubt buying the Maxxdigital has the major advantage that it comes with great support. Some need that and some of us are techies to a certain level that we can live with saving the 50% and configure stuff ourselves. I have absolutely nothing against Maxxdigital at all, just providing valid information.

Tony,

Ron at Maxx Digital is way too humble, but that have been suppling some of the biggest Storage needs to major movies, and major Studios, like the tower system below to Sony, again, for me is a factor of great importance to be able to count on them to call me even overseas to insure everything is okay, the fact that they can custom build a unit for you, the fact that they can send a tech with the gear shipped any were in the world and do the installation for you and test the unit to insure everything works, and the continuos support a quick phone call away on Ron's cell, this to me are top winners in this, and it is not because Ron is my friend that I choose them for all my projects, but because I need reliable Storage for my footage.

 

[Tony Lorentzen]

Hey Ketch,

All I'm embracing is the fact that we all have the freedom to choose. I think it's great that I can get a system like the Maxxdigital for about half the price because I know how to service the thing myself, but I totally realize someone else might not and will be willing to pay the extra money for service and support. The power of choices

Best

Tony

 

[Ketch Rossi]

Hey Ketch,

All I'm embracing is the fact that we all have the freedom to choose. I think it's great that I can get a system like the Maxxdigital for about half the price because I know how to service the thing myself, but I totally realize someone else might not and will be willing to pay the extra money for service and support. The power of choices

Best

Tony

Yes, absolutely Tony...