Strage Array Performance and Connectivity Notes

[Tim Jones]

Hi Folks,

(Of course, that title SHOULD have been "Storage Array Performance and Connectivity Notes")

In light of the discussion about eSATA and SAS storage arrays and the way they need to connect, I thought I'd share some further details and get you all up to speed with what and how these devices work.

When you deal with eSATA/SATA and SAS connectivity, there are two ways to connect devices:

Point to Point

Each drive is attached to a single SATA / SAS channel​

Multi-device Expander

Each SATA / SAS channel can connect to multiple drives​

The differences are as follows:

PtP:

You are limited to the number of channels available on your host -

A single eSATA/SATA channel can connect to 1 SATA drive
A single SAS channel can connect to 1 SAS or SATA drive​

Therefore, an 8 channel SAS HBA would allow you to connect 8 devices. This could be 8 disks, 7 disks and a tape, etc. But the total number of physical devices cannot exceed the number of available channels.

Expander:

The limits change depending on the expander chipset (manufacturer) and type of expander used -

There are 2 types of expanders defined:

Edge Expander

An edge expander allows for communication with up to 255 SAS devices, allowing the SAS initiator (computer) to communicate with these additional devices.​

Fanout Expander

A fanout expander can connect up to 255 sets of edge expanders, known as an edge expander device set, allowing for even more SAS devices to be addressed.​

If you do the math, the result is a very bit number. However, 65,536 is not somewhere an average storage solution would want to head.

Pluses -

• Because PtP is a single device per channel, you can expect to see the full bandwidth potential from each device. This means that if you are using 6G connectivity with an 8 channel HBA, your aggregate (combined) potential bandwidth is 8 x 6G, or 48Gb/sec (~5GB/sec cooked).
• Because Expanders allow for many more devices, you can provide much higher storage capacity points when compared to PtP.

Minuses -

• Because PtP limits you to the number of physical channels available, your resulting capacity will be limited by the capacity sum of your drives and any RAID level overhead.
• Because Expanders share channels for multiple devices, you will be limited to the maximum bandwith per channel for an attached collection of drives. This means that if you are using a 6Gb HBA and you have an array of 8 drives, you will still be limited to the maximum of 6Gb from those drive (around 600MB/sec cooked).

Hit me with any questions / clarification.

Tim

 

[Charlie Perera]

Hi Tim,

Thanks for that very useful breakdown of information.

I was wondering if you could provide any feedback for a issue I am having regarding copying data to multiple eSATA devices via an Atto H680 in a Magma 3T or Sonnet Echo Express Pro chassis.

I am only able to copy to one eSATA device, when copying from a Promise R6 or SAS Storage via a 10GigE network adapter (currently in a Sonnet Echo Express SE). Finder crashes if I attempt to copy to more.

I have tried a couple of miniSAS to 4x eSATA fanouts and have the latest Atto drivers. Retina MBP, OSX 10.7.5.

Your thoughts would be much appreciated.

Regards,

Charlie

P.S. I am able to copy from RED Mag SSD reader(s), to 2x eSATA drives all on the same fanout attached to the H680.