Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

NEED help with this DIY 16TB RAID

J

J Dub

New member
Joined
Jan 30, 2009
Messages
4
Reaction score
0
Points
0
Summery:
So I bought a 16bay SATA II external case put 16 x 1TB Samsung F1 in, with a arc-1160ML RAID card on an old G5 dual 5GB RAM.

Pros:
* After setting up a RAID 6 I have about 14TB Volume.
* HUGE File Store Library of Video
* Transfuses of around 700-800MB/s read and 300write

Cons:
* Using SMB (SAMBA) i get about 60-90MB/s on a Gigabit network Full Framed. (I'm also using FC server [meta])


What can I do to pump this up much faster over the network... at the moment I just want to use it to store my library of Havested Video... When ready to edit I will copy what I need... I have looked into iSCSI but every one says to be wary of such. When running 'top' while coping from network it looks like the overhead is fileserver/smb its self on large transfures.

Ideas :ninja:s ?
 
10GbE would indeed speed things up. The problem is that GigE has a max throughput of about 118MB/s in real-world numbers and SMB has a significant amount of overhead, so 90MB/s SMB over GigE is about as good as it gets.

10GbE or Infiniband would probably be a better solution and then look to a SAN implementation. Depends on what sort of compatibility is needed though what platforms will this storage volume be accessed from? How many workstations are being connected. It often becomes a lot cheaper to put large RAID volumes on multiple workstations than to link all your systems to a single high-speed storage node.
 
Jeff Kilgroe said:
10GbE would indeed speed things up. The problem is that GigE has a max throughput of about 118MB/s in real-world numbers and SMB has a significant amount of overhead, so 90MB/s SMB over GigE is about as good as it gets.

10GbE or Infiniband would probably be a better solution and then look to a SAN implementation. Depends on what sort of compatibility is needed though what platforms will this storage volume be accessed from? How many workstations are being connected. It often becomes a lot cheaper to put large RAID volumes on multiple workstations than to link all your systems to a single high-speed storage node.
Click to expand...

ok, looking at http://en.wikipedia.org/wiki/List_of_device_bandwidths

iSCSI over 10G Ethernet (Very few products exist) 10,000 Mbit/s 1,250 MB/s
iSCSI over 100G Ethernet (Planned) 100,000 Mbit/s 12,500 MB/s
Click to expand...
 
I think I understand now.

Gigabit Ethernet is not Gigabyte Ethernet. In other words, Gigabit Ethernet is 1,000 Megabits per second which is only about 120 Megabytes per second? Do I have that right or is there some other reason for the 8-fold drop-off? That would mean 10 Gigabit Ethernet would have a max transfer speed of 1.2TB per second assuming no overhead from SMB and only two computers on the 10 Gigabit switch?

What is the cheapest way to get 10 Gigabit Ethernet switch and cards and cabling on a network of 3 macs and 3 PC's?

Is there a cheaper way? Fibre Channel?

-shooter
 
this works: if the storage system is managed by a HOST (i.e. windah's etc) and the client system is accessing it simply as a NAS (which it sounds like) ad GbE is you bottleneck using CIFS/SMB where 120Mbit/sec is generous, then consider..

using a COTS (ISV) UDP application. I believe RORKE DATA et al (pr companies like them) have something that really kicks this content in. The RORKE DATA ImageLAN™ (not ImageSan™) or equivalemts that are simlar to it is not so expensive per seat.

The idea is (OR WAS) that the agents/clients sits in the file system and aside form using propreitary UDP to push the packets around (not TCP) their implemntatio has the advantage of making the NETWORKEd attached file system appear like a DIRECT ATTACHED FILE SYSTEM ATTACHED.

Other oprions are trying NIC TRUNKING.. this works too. Need an app or some specialised NIC s/w for this.

Sadly to say that serving large objects over a TCP/IP network will never be easy , especially with GbE.

Other considerations as most I.T. guys will state, is the saturation and packet distuption you get = latency.

FWIW, if you DONT have a need for a NAS AND you need good r/w efficiency, you will need to go proprietary.. especially to lug around HUGE objects...

in a 'usual' busy network, SMB/CIFS is less than USB2 on a good day. Other will disagree so you can easiy test it out between two GbE NICS and a few QUICKTIME movies to see how well they pay until you hit a wall.. this is te effective limit of the network path.

No rocket science here.

Option 'B' is to may be look at using JUMBO FRAMES.. good luck there.. some increase in data rates at 9KB packets but now you have a very bust network path and additional cost to upgrade all your switched etc to JUMBO frames.. (agin will stand corrected).

10Gbe is certainly not a 10 x fold over Gbe .. when TCP/IP apps are used from my experience.

fwiw.
 
If you are getting 90MB/s on Gigabit, you are doing great. If you need more than that, you have few solid options besides Fibre Channel. Fibre Channel is much more expensive than ethernet, but 1500MB/s is achievable with that interface. You will need a Fibre Array, a Fibre Switch, Fibre cards, and SAN software, none of which are cheap. We have a dual channel 4Gb setup, and can get 600MB/s to any system from a single 16disk array. With more disks, and the new 8Gb fibre and/or Quad channel, you could expect to double that if needed. You are looking at $30-50K in initial costs though, as opposed to like $5K for sharing a local array on Gigabit.
 
J Dub said:
ok, looking at http://en.wikipedia.org/wiki/List_of_device_bandwidths

iSCSI over 10G Ethernet (Very few products exist) 10,000 Mbit/s 1,250 MB/s
iSCSI over 100G Ethernet (Planned) 100,000 Mbit/s 12,500 MB/s
Click to expand...
Click to expand...

Problem is... 10G Ethernet is rather expensive and as the Wiki link says, few products are in existence.

100G Ethernet (Planned) is fine and dandy, but it doesn't exist yet. It's a tentatively planned spec. No products exist. It theoretically (or in lab tests) requires multiple bonded CAT-6E STP cabling. There is no single-cable standardized interface at this time. Maybe a few more years.

Currently Infiniband is the only solution out there that can reach these sorts of speeds. But we're talking a lot of money here, especially for quad-channel (QDR).

Fiber channel is a good solution too. A lot cheaper than Infiniband, but still terribly expensive compared to GigE ethernet. But it's the best realistic option for SAN storage at the moment unless you have the needs and budget for Infiniband.

iSCSI pretty much sucks, BTW...
 
An sort of easy way to increase data speed is to have multiple network cards on the server machine. If you have 4 edit stations you can use one 4x slots gigabit ethernet card and connect cable directly with no switch or hubs in between. many switched that handle gigabit have poor bus backbone and just have small ARM cpu's and don't handle large volumes of data good. If you use NFS (network file systems) you probably get better speed also. I'm using this setup, and have worked very good, and much better then pulling teeth with SMB. ..and much lover cost then infiniband, 10G-ethernet or fiberchannel.
 
J Dub said:
What can I do to pump this up much faster over the network...
at the moment I just want to use it to store my library of Havested Video... When ready to edit I will copy what I need... I have looked into iSCSI but
every one says to be wary of such. When running 'top' while coping from
network it looks like the overhead is fileserver/smb its self on large transfures.

Ideas :ninja:s ?
Click to expand...
You could set up link aggregation in OS X (basically striping both ethernet ports).
Do the same with all computers on your network and you should be
able to get at least 125 MB/s of read and write from any machine.
 
Dustin Cross said:
You can also get inexpensive Netgear GigE switches that can have two 10GbE ports. - Dusty
Click to expand...

Thanks Dusty. Where do I find the inexpensive 2-port Netgear 10GbE switches? Model number, price?

Also, if I just want 10GbE between my RAID server and my primary MacPro, can I just buy (2) 10GbE cards and connect directly without switch?

-shooter
 
Shooter,

Talk to the guys at Small Tree. I am pretty sure you can direct connect two 10GbE cards.

Any of the Netgear switches with GBIC ports will work. Looking at Netgear website I don't see any CX4 GBICs any more.
http://www.netgear.com/Products/Switches/SwitchModules.aspx

Something like this switch for $1800 would work:
http://www.netgear.com/Products/Switches/FullyManaged10_100_1000Switches/GSM7328S.aspx


Talk to the guys at Small Tree they are easy to work with.

Why to you need more than 100MB/s?

AFP is faster than SMB if you are going Mac to Mac, but even AFP has limitations. I think about 300MB/s is the fastest I have ever heard of AFP getting even over 10GbE which should be able to do 1000MB/s.

Dusty
 
Jon is right you may well be able to bond the NIC's and get 2Gbit. Jumbo frames will also help but there will be limitations on the network in so far as you can connect machines.

What sort of speed to you require to support ?
 
You must log in or register to reply here.
Back
Top