Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Need advice about 96TB RAID storage

computer moron

computer moron

Hi -
unless you are have lots of free time on your hands, I don't suggest building your own system. You will find that even when you have the right pieces, it takes a lot of
trial and error to get everything setup correctly. For example, you just don't plug in a 10gig ethernet card into a Mac, and it just works, without tuning files. You don't get
400MB/sec + speed just by plugging in a card into a Mac. Anyone that tells you that has simply not done it.

When you buy a generic drive array, it requires a disk drive host adaptor, which also has lots and lots of setting that have to be correct, for it to work properly. This is why when
you buy from Maxx Digital, Sonnet, Dulce, etc. you get a WORKING SOLUTION, and not some piece of crap that you have to post on user forums "gee, how come everything is running
so slow".

You say that "you have the money" - so you should buy a working solution. Solutions from Maxx Digital, AVID, EditShare, Facilis, Studio Network Solutions, Cal Digit, etc. are all excellent
but you will need support for setup, and on going maintenance with any of these solutions. Let's take Cal Digit for example, and SuperShare, which is a great product. When you buy
SuperShare, you DO NOT get SAN manangement software. You need XSAN, MetaSAN, or Command Soft Fibre Jet to get this to work. Did you know this - who is going to set this up
for you - who is going to teach you how to use MetaSAN (for example). That is my point. This ain't no firewire drive that you just plug in. You need SUPPORT from an excellent company, or a
dealer that is willing to help you every step of the way. Even the best systems have setup problems, and you will at some point (particularly at the beginning) need help from someone.

Maxx Digital Final Share is a very simple shared storage solution that will give you 90 - 100MB/sec per client using standard 1GbE copper ethernet cable, or over 350MB/sec using
10Gig Ethernet direct connect. Redcode 42 at 4Kp24 is 36.9MB/sec, so you can see that even standard 1GbE ethernet will work for you. This just doesn't apply to Maxx Digital Final Share - it applies to any copper ethernet solution on the market. But let me assure you, when you go out to the local store to buy a generic Netgear 1GbE 24 port switch - NOTHING WILL WORK.
You need special equipment, setup the right way, or nothing is going to work.

If it was easy, then Maxx Digital, EditShare, Facilis, AVID, Cal Digit, etc. would not be able to stay in business.

Bob Zelin
 
I forgot to comment on the dual system identical data scenario. It ain't gonna happen unless it happens manually. Even with a 100MB/sec uplink pipe, you are not going to be transfering Terabytes of data from array # 1 to array #2 over the internet. The technology just isn't there. You can do low res proxy files (and I mean h.264, not ProResProxy), but you can't have RedCode or ProRes422HQ files transfering over the internet to two different drive arrays, when you are talking about Terabytes of data.

Bob Zelin
 
Tom you should contact this guy : Christopher Noellert: chris dot noellert at mac dot com

He is my old college, now working / living in LA area. Chris has set up more digital labs than most on both side of the atlantic and he for sure knows what to get and especially what not to get, and he is not in the market of selling you something. but will consult you in getting the best possible solution, help you install it and so on.

And when I write "digital labs" that is digital film labs with 70 or so people on staff multiple flames, scanners, Q machines, edit bays, film printers etc so he will for sure know how to get your house wired.

I think you and him would get along in a good way, he is also one of the very best flame artists on this plantet so maybe you have more use of him than just setting up the gear... So say hallo from me and you will be in good hands.
 
Tony Lorentzen said:
Yeah, Mathieu. That's a nice solution unless you need to work with 4K DPX files, which is what Tom needs to as far as I understand. I'm not into QNAP myself, but rather Synology who has a similar solution for that too.

Tom, if you want to save a buck or two (and who doesn't?) then you might want to take a look at the iStoragePro hardware. It's basically the same as Maxxdigital and Light Iron is using in their products except you can build a system like theirs for almost half the dough in some cases. The Enterprise systems from iStoragePro has two 4 Gbit fibrechannel ports which should be able to provide you with the ability to work with 4K DPXs. Check it out: http://cidesign.com/products/special-order/Enterprise/Enterprise-Fibre-Subsystem/
Click to expand...

Just an FYI. The link is for an older discontinued product and iStorage pro has no plans to continue in that market. They are a sheet metal company that sells to may system builders and i can assure you that there is much more that is done after leaving there hands. Also CID the parent company only sells to system builders. there are many good choices for shared storage and we at Maxx Digital offer amazing service to our clients and that is all we can do to keep them coming back. As Jeff said always have a good archive system as well. We like our LTO 5 solution as it is good for 30 year plus shelf life. Our FINAL SHARE for PC or Mac is a great cost effective (supported) solution for shared storage and it is easy to use.
 
Here if you guys like Power and Speed try linking together 3 of this really Bad A^%$#$ 32GB Thunder RAID systems... I know I will... ;)


maxxthunderraid060.jpg
 
Tom Lowe said:
I need to install 96TB of RAID storage here at my house, with fiber connections running to six different rooms in the house.

What type of storage hardware would you guys recommend? Can I just buy a 96TB server from Dell or something? What RAID config should I use? RAID 5?

How will the server connect to my PCs via fiber? Is there some kind of PCI card for fiber?

Thanks for any advice you can offer! I don't know much about this subject at all.
Click to expand...

Hi Tom,
I just saw your Smaug post and remembered this post.

Recently I stumbled across LSI SAS Switches. Jupiter from OWC will take advantage of this. Faster than Fibre, and supposedly cheaper.
I hope it helps.

Here's their 64TB model, and easily expanded. I can't wait to see what it's capable of.

http://eshop.macsales.com/item/OWC/ER1630064TE/
 
Tom,
The company I work for is looking at doing something similar but more machines and less bandwidth per seat. Anyways was wondering what you decided to go with and how its working out for you?

Thanks,
Rudy
 
Probably too late for the OP but the cheapest/highest quality to me seems to be doing it yourself if you have the know how (like me) and supermicro chassis/motherboards with hitachi disks. Hitachi disks have proven to be the most reliable by far eve at the data center I used to work at where we had 10s of thousands of disks in production. Here is what my rack setup at home is:



about 200TB of storage.
 
What do you use to connect the 200TB, and then To other client computers?

Are you using multiple raid cards, sas expanders, and then shared via a 10GBE switch, or?
 
ExaSAN PCIe based very fast RAID storage

ExaSAN PCIe based very fast RAID storage

Tom Lowe said:
I need to install 96TB of RAID storage here at my house, with fiber connections running to six different rooms in the house.

What type of storage hardware would you guys recommend? Can I just buy a 96TB server from Dell or something? What RAID config should I use? RAID 5?

How will the server connect to my PCs via fiber? Is there some kind of PCI card for fiber?

Thanks for any advice you can offer! I don't know much about this subject at all.
Click to expand...

Tom,

If you're primarily running Mac clients you might want to seriously consider an ExaSAN based Xsan RAID configuration - very fast and much cheaper than fibre channel based topologies ... here's the reasons why it's a good option to consider:

1) Apple's Xsan client licenses are FREE in every copy of OSX Mountain Lion and Mavericks - your SAN management software choice is actually the most important decision you have to make - it's the "glue" that makes it all happen and allows you to share data between multiple users at the same time without corrupting files or destroying directories.

2) you'll need a dedicated MetaData controller to run the Xsan SAN software - a Mac Mini is cheap, robust and works incredibly well ... you can upgrade from ML to ML Server for a whopping $19.99 ... twenty bucks for a sophisticated SAN management software solution is an incredible deal - also, Apple is pouring a ton of engineering resource into Xsan so it's only going to get better.

3) ExaSAN RAIDs are based on the PCIe architecture - PCIe is the fastest I/O bus on your motherboard ... an ExaSAN RAID delivers 1200 MB/s to 1300 MB/s to the desktop from a 12 x bay enclosure ... you easily saturate the 800 MB/s bandwidth if you're using a Thunderbolt to PCIe expansion box like the MAGMA expansion chassis that we recommend.

4) With ExaSAN RAIDs you can start small with say 36 TBs to 48 TBs in one A12 RAID enclosure but easily add three more A12s giving you 4 x 48 TBs = 192 TBs ... and then if you need more storage, you can easily add 3 x JBOD enclosures to each of the A12 boxes giving you up to 768 TBs ..... with your 4K time-lapse photography it won't take you long to fill that up ;-)

5) You can have up to 12 clients (Mac, PCs and Linux) attached to one SW16 switch which is attached to the A12 RAIDS.

In terms of pricing, a single ExaSAN A12 costs $6000 and with 4 x !2 desktop Hitache drives you can have 48TBs of 1300 MB/s high speed RAID storage for under $10k! .... another 48 TBs using a JBOD for an extra $8k .... total of $18k for 96 TBs of the fastest bus you can get ... noting is faster than PCIe.

You'll also need a switch, cabling, HBAs and the MDC Mac Mini but it's by far the cheapest and fastest shared storage option out there - we've tested the alternative FC, SAS, NAS and 10GbE options and its the approach we recommend for all 4K workflows. Buying a simple DAS RAID is easy and cheap (there's hundreds of me-too products out there) but if you want to share your data between multiple users you should carefully consider a Xsan based SAN if you're an Apple guy.

More details of the ExSAN approach are available on our website - I'm an Apple Certified Consultant specializing in OSX Server and Xsan so if you have any questions on best practices for an Apple based SAN, call me on 323 850 3550 and I'll be happy to talk through what else is involved.

http://www.lumaforge.com/page10/index.html

Cheers,
Neil
 
Richard Tran said:
What do you use to connect the 200TB, and then To other client computers?

Are you using multiple raid cards, sas expanders, and then shared via a 10GBE switch, or?
Click to expand...

Most of the storage is DAS hooked up to my main machine. 138 TB of the disks is on one machine using two raid controllers and external storage via SAS expanders. This is also my beefiest machine with dual 8 core CPU's and 192 GB of ram and a geforce gtx 670 4GB. For the other machines I just use gigabit ethernet for now. I don't run Mac OS X so if I wanted faster networking I would go the infiniband route probably although honestly I don't really have any free expansion slots ATM with video card, sound card, two raid controllers and 2 BBU's.
 
SAS does not speak PCIe

SAS does not speak PCIe

Richard Tran said:
If you have one available SAS port,

You might be able to share all that data via an LSI SAS SWITCH.
http://www.lsi.com/channel/products/storagecomponents/Pages/LSISAS6160Switch.aspx

The OWC jupiters are definitely built around this system, and reading the ExaSAN site, it appears it is also.
Click to expand...

We run both Jupiter and ExaSAN RAIDs here and are a LSI Channel Partner for the 6160 SAS switch - we're a beta tester for all three companies and have put all of the different topologies through their paces ... you're slightly mixing up several things:

1) the RAID controllers in the actual Jupiter and ExaSAN enclosures may or may not be LSI RAID controllers ... things change all the time.

2) the RAID controllers in both boxes talk to SAS to the physical drives drives.

3) the Jupiter box talks SAS to the Jupiter/LSI 6160 switch

4) the ExaSAN box talks PCIe to the ExaSAN SW16 switch ... totally different protocol to SAS.

So, no you will not be able to connect a Jupiter box to a ExaSAN switch and likewise you will not be able to connect a ExaSAN A12 box to the LSI 6160 SAS switch.

Sames goes for a fibre channel topology ... different handshaking protocols that are not interchangeable even though all hard drives at a very low level speak SCSI which SAS does (Serial Attached SCSI).

You pays your money - you picks your choice ... which is why I was recommending above that Tom looks closely at the ExaSAN PCIe based configuration .... SAS at the moment is only 6Gb/s versus 20 Gb/s for PCIe 2.0 ... that gap will only get wider in the fall with the with the wide scale adoption of PCIe 3.0.

The ExaSAN RAIDs delivers a sustained 1200 MB/s to 1300Mb/s throughput to the desktop ... 48 TBs for under $10k.

In terms of 'bang for buck' for 4K capable fast storage you can't beat PCIe based ExaSAN RAIDs at the moment ... Thunderbolt 2 might move the goal posts but at the moment if you want a fast, cost-effective SAN for multiple Apple clients you should evaluate ExaSAN and Xsan closely ... if you are primarily a PC house then a SAS based topology with MetaSAN might be a better alliterative (albeit more expensive).

Neil
 
Neil W. Smith said:
You pays your money - you picks your choice ... which is why I was recommending above that Tom looks closely at the ExaSAN PCIe based configuration .... SAS at the moment is only 6Gb/s versus 20 Gb/s for PCIe 2.0 ... that gap will only get wider in the fall with the with the wide scale adoption of PCIe 3.0.

The ExaSAN RAIDs delivers a sustained 1200 MB/s to 1300Mb/s throughput to the desktop ... 48 TBs for under $10k.
Click to expand...

Keep in mind that SFF-8087/SFF-8088 which you use to connect to disk enclosures are 4 SAS lanes so its 24 gigabits. Also with multiple links/multipath disk I/O which a lot of controllers/OS support you bump that up to 48 GB so there is a lot of bandwidth there.

On my 90 TB array (84 TB usable) I can read at 2000 MB/sec and write at 1100 MB/sec. This is raid controller bottle necked though from a raid controller that is now two generations old. The newest ones could easily do 3000 MB/sec read and near 3000 MB/sec write.

I built this DAS system 2 years ago almost exactly to the dot. It cost me about 4.5k for 84 TB usable. Honestly I can't say under 10k is all that impressive for only 48 TB at today's prices when right now. Right now I could build a 176 TB usable for around 10k which would have local write speeds of multiple gigabytes/sec. Now adding 10 gig to the mix might put me a bit over 10k though.
 
You get what you pay for ...

You get what you pay for ...

houkouonchi said:
Keep in mind that SFF-8087/SFF-8088 which you use to connect to disk enclosures are 4 SAS lanes so its 24 gigabits. Also with multiple links/multipath disk I/O which a lot of controllers/OS support you bump that up to 48 GB so there is a lot of bandwidth there.

On my 90 TB array (84 TB usable) I can read at 2000 MB/sec and write at 1100 MB/sec. This is raid controller bottle necked though from a raid controller that is now two generations old. The newest ones could easily do 3000 MB/sec read and near 3000 MB/sec write.

I built this DAS system 2 years ago almost exactly to the dot. It cost me about 4.5k for 84 TB usable. Honestly I can't say under 10k is all that impressive for only 48 TB at today's prices when right now. Right now I could build a 176 TB usable for around 10k which would have local write speeds of multiple gigabytes/sec. Now adding 10 gig to the mix might put me a bit over 10k though.
Click to expand...

Wonderful system you have there and I agree, we could all build our own 'roll you own' storage which would probably be a lot cheaper than an off the shelf commercially available and tested RAID configuration with a five year warranty on parts. However, I was assuming that Tom was not looking for a 'science project' but something that will work reliably and consistently straight out of the box and require the minimum amount of support, hassle and down-time.

With your 176 TB array for $10k, who would I call at 4am in the morning when my project has to be delivered at 8am and I'm having problems writing 5K files to the drives? I know, your system always works perfectly and there's never any technical problems.

I'd also like to know where you're getting your components from ... you'll need 44 x 4TB drives to achieve 176 TBs .... assuming you're using Hitachi or Seagate desktops and you can get them for around $130 each that puts you at $6k for hard drives alone ... a decent LSI or ATTO RAID controller is around $900 each .... so that leaves you with around $100 total for your enclosures, power supplies and fans.

I totally get that enthusiasts build their own 'Hackintoshes' to save money ... however, professionals typically buy Macs from a reputable Apple dealer with a good support infrastructure .... I wonder why? At the professional level, time and long-term reliability are more valuable than short-term cost savings.

Tom is a highly skilled professional cinematographer and businessman .... his data and Red files are his livelihood - his storage system is literally worth its weight in gold ... are you seriously suggesting that Tom bets his most valuable asset on a 'roll you own' system with no Tech QC, warranty or technical support?

And btw, the ExaSAN A12 RAID system does indeed have a theoretical maximum read and write speed of 2400 GB/s .... however, I've never seen that in real life ... we can actually get 1400Mb/s sustained read/write to a MacPro Tower with a single PCIe card with a single A12 RAID enclosure running desktop drives (not Enterprise SAS) ... I always quote the more conservative figure of 1200 MB/s to 1300MB/s .... customers are always delighted when they run the BMD or AJA Disk Speed test and actually get more.

Earlier this year, we were asked by Blackmagic to thoroughly test the ExaSAN A12 RAIDs so it could be certified for DaVinci Resolve usage ... Peter Chamberlain (Product Manager for BMD Resolve) has a brutal real-life disk storage and data fragmentation test that he puts his storage through before he certifies if fit for use with Resolve ...we ran Peter's killer stress test for several days and even at over 90% full capacity the ExaSAN A12 RAID still performed over spec ... sustained 1200 MB/s to 1300 MB/s read and write speed to the desktop - reliably, consistently and effortlessly.

Neil
 
I live in the IT world and I say knock it out ...the cost of upgrading a lower end system later will even out to what you will spend today. You should at lease have a 5 year lock with all that you are installing and stay away from the cloud world because their key is for you to own nothing and track your life. Hard drive crash I can replace it...forget your password on the cloud prove that its you or they can lie and say your stuff got deleted by mistake now what? films, ideas, personal docs are gone so you think.
 
Neil W. Smith said:
We run both Jupiter and ExaSAN RAIDs here and are a LSI Channel Partner for the 6160 SAS switch - we're a beta tester for all three companies and have put all of the different topologies through their paces ... you're slightly mixing up several things:

1) the RAID controllers in the actual Jupiter and ExaSAN enclosures may or may not be LSI RAID controllers ... things change all the time.

2) the RAID controllers in both boxes talk to SAS to the physical drives drives.

3) the Jupiter box talks SAS to the Jupiter/LSI 6160 switch

4) the ExaSAN box talks PCIe to the ExaSAN SW16 switch ... totally different protocol to SAS.

So, no you will not be able to connect a Jupiter box to a ExaSAN switch and likewise you will not be able to connect a ExaSAN A12 box to the LSI 6160 SAS switch.

Sames goes for a fibre channel topology ... different handshaking protocols that are not interchangeable even though all hard drives at a very low level speak SCSI which SAS does (Serial Attached SCSI).

You pays your money - you picks your choice ... which is why I was recommending above that Tom looks closely at the ExaSAN PCIe based configuration .... SAS at the moment is only 6Gb/s versus 20 Gb/s for PCIe 2.0 ... that gap will only get wider in the fall with the with the wide scale adoption of PCIe 3.0.

The ExaSAN RAIDs delivers a sustained 1200 MB/s to 1300Mb/s throughput to the desktop ... 48 TBs for under $10k.

In terms of 'bang for buck' for 4K capable fast storage you can't beat PCIe based ExaSAN RAIDs at the moment ... Thunderbolt 2 might move the goal posts but at the moment if you want a fast, cost-effective SAN for multiple Apple clients you should evaluate ExaSAN and Xsan closely ... if you are primarily a PC house then a SAS based topology with MetaSAN might be a better alliterative (albeit more expensive).

Neil
Click to expand...


Thank you so much for the clarification Neil! Appreciate it.
 
Yeah I get what your saying but my point was that wasn't really a bargain is all.

I actually was calculating for more than 6k of drives but with expanders and stuff you can run all that off a single raid controller and there is not a need for two $900 controllers and I typically use Areca. You could run it off a $600 areca controller instead of two $900 LSI controllers. Also even with the stuff you mentioned you said $100 for chassis and fans when alt he parts you listed so far was only around $8k. I was going by a $10k budget.

I get that building your own machine is not for everyone. I am somewhat of a raid/storage expert so its not really an issue in my case.

The company I work for is all about storage actually. Its about the support/development of an open source/clustered file-system than the founder of the company created.

I have just seen a lot of DAS/SAN/NAS systems that are a lot cheaper than 48TB for $10k and disagree about how much of a bang for your buck that is.
 
You must log in or register to reply here.
Back
Top