Why the Thunderbolt - eSATA speed limit?

[Elliott Balsley]

Calling Jeff Kilgroe or other IT wizards...

A single thunderbolt port should have a theoretical speed of 2000MBps (4x PCIe 2.0). Right?
So why can't anyone get more than 800MBps on an external eSATA card? Here's an interesting test done with a bunch of thunderbolt RAIDs daisy-chained, where they maxed out around 800MBps on one port, and 1300MBps using both ports on a MBPr.
http://www.tomshardware.com/picturestory/623-12-thunderbolt-performance-5big.html
I realize there is overhead in any bus, but 800MBps is such a huge hit! I'm just curious to know the explanation for this.

 

[Dimitri Boschma]

The stuff tested is with Lacie drives.. and I think the Lacie stuff is crippled to only Sata 3G. (At least my Lacie sata/thunderbolt hub is) And I think I read somewhere the Big drives from Lacie are also.
If you would use a External PCe thunderbolt box with for example an Atto Card you would get a bit higher speeds.
Correct me if I am wrong but I think that is the case here...

 

[Jeff Kilgroe]

The test is a waste of time. First off, they used the LaCie 5Big TB RAIDs. They're garbage. These RAIDs top out at 500MB/s on a good day, due to limitations of their internal RAID controller. They're Thunderbolt 1 (not TB 2) devices, meaning that the actual max bandwidth allowed to each RAID device is 10Gbps or really about 800MB/s. Thunderbolt ports are 20Gbps total, distributed into two 10Gbps allocations, each allocation having a combined 10Gbps up and down stream channel.

By chaining two of these LaCie units together on the same TB port, each can have their own 10Gbps channel, both are running about 420MB/s sustained, you get about 800MB/s after overhead and software RAID between the two. Meh… You have less scheduling overhead and can get a little bit of a boost by splitting them across two different Thunderbolt ports. However, the maximum bandwidth available to each still sits at 10Gbps and the bottleneck is still their onboard RAID controller in those LaCie boxes.

My Promise Pegasus R6 RAIDs can sustain right about the 800MB/s maximum. Easy to sustain 1.5GB/s with two of them in a RAID-0 stripe.

I appreciate a lot of the stuff Toms Hardware tries to demonstrate with tests like this, but there are times you just have to scratch your head and wonder why they don't do any research before the testing to actually make a worthwhile test. And why they don't take the time to explain what's going on and why and what could be done to make it better.

As for the ATTO card in a PCIe expander, yes it works great, as do other good RAID controllers from Areca, HighPoint, etc... However, it tops out at about 800MB/s as well because of the Thunderbolt channel limitations.

Thunderbolt 2 brings full port bandwidth availability shared amongst connected devices. So Something like the Pegasus R8 RAID that is coming soon is said to deliver well over 1GB/s from a single 8-drive RAID box when connected to a Thunderbolt 2 port. PCIe expanders that connect to Thunderbolt 2 would be limited more to about 1.6GB/s. Of course, that only happens with Thunderbolt 2 devices. And as soon as you connect a Thunderbolt 1 device to a Thunderbolt 2 port, the port and all other devices will fall back to Thunderbolt 1 operational mode -- 10Gbps max per device.

The biggest push for Thunderbolt 2 from the perspective of Intel and getting peripheral card makers (primarily GPU makers) to integrate Thunderbolt was to move to 20Gbps port-wide ability so that proper 4K/UHD at desired frame rates could be supported. Use of the extra bandwidth for storage or PCIe expansion is nice, but a smaller niche of the market. Thunderbolt 3 will arrive in late 2015 or early 2016 and it will increase the actual bandwidth of the port as a whole. No one knows for sure what the final spec will allow, there have been claims of 40, 80 and 100 Gbps per port. Intel has demonstrated 100Gbps working in a lab environment and I know they are working to that goal.

 

[Elliott Balsley]

You may not have seen in that test they also daisy chained 4 units on a TB port. No improvement of course. But the bottleneck is clearly the interface, not the drives.
The real question is why is 800MBps the limit? TB is supposed to be 10Gbps each direction, which should equal 2500MBps by my calculation.

 

[Jeff Kilgroe]

You may not have seen in that test they also daisy chained 4 units on a TB port. No improvement of course. But the bottleneck is clearly the interface, not the drives.
The real question is why is 800MBps the limit? TB is supposed to be 10Gbps each direction, which should equal 2500MBps by my calculation.

Didn't notice that about the 4 drives in my initial skim-through. So I was thinking that something was definitely wrong there and it may have to do with how they chained or set up the drives, or perhaps how LaCie has implemented their Thunderbolt connectivity. As I stated above, each Thunderbolt port has two 10Gbps channels (bi-directional composed of one 10Gbps up and one 10Gbps down stream per channel). There is an aggregate of 20Gbps bi-directional in each Thunderbolt port with the ability for each device to attach itself to one of the two channels, not both. Thunderbolt 2 changes that, but that's another discussion. By attaching 4 drives, they should have been able to see the full bandwidth of the port, but it seems as if they were still locked to a single 10Gbps channel and I would question the LaCie RAID in that regard.

Then, I took a couple minutes and went back to read through all of it. It seems they muddled around and were bitten by crappy Thunderbolt hardware, as suspected. OTOH, they still realized nearly 1400MB/s when doubling up LaCie RAIDs on both ports. Given the throughput available in that scenario and the craptastic performance of the LaCie drives, that's about what I'd expect to see there.

Yes, I've worked with the LaCie 5big units myself. Bought two of them when they came out, against my better judgement (LaCie stuff is usually shite). Sent them back when their 5-drive units couldn't hold pace with my 4-drive Pegasus R4.

10Gbps is 10 billion bits per second, which equates to 10000000000/(8+2)/1048576 = 953.67MB/s. Real-world performance due to bus latency and additional interface overhead of the secondary electronics and signal conversion, etc.. leaves the top speed at about 800~850MB/s per channel. 1.5GB/s is perfectly do-able on a single Thunderbolt port and I have done it myself, as noted above, with the Promise Pegasus RAIDs. To put it bluntly, if these guys couldn't get more than 800MB/s using more than two of the LaCie 5crap RAIDs, then there is something really wrong with LaCie's RAID boxes (my first suspicion). Actually looking at the results, it seems that the LaCie units don't span to the secondary Thunderbolt channel on the port -- a somewhat common and undesirable limitation amongst Thunderbolt devices on the market. Placing two or more on one port means they continue to aggregate on channel #1 and saturate that channel to the tune of 800MB/s, rather than load-balancing the bandwidth on the two available channels by alternating their channel allocations as more devices are added. A lot of early Thunderbolt devices would only attach to the first channel, many didn't even include a secondary port for chaining other devices.

To me, this is a rather poor test of Thunderbolt. It is somewhat telling of the LaCie drives, although it begs more control and information. Not sure if it's worthwhile to pursue it more or if anyone would really care. LaCie is already replacing these drives with newer Thunderbolt 2 capable units and other than a couple mid-range PC motherboards, the misfit 2013 iMac and long in the tooth Mac mini, no one is shipping Thunderbolt v1 ports anymore.

 

[Elliott Balsley]

I get what you're saying about LaCie units not properly spanning.
Wow, I totally messed up on that calculation. *Forehead slap*
But I am wondering, why do you divide by 10, aren't there 8 bits in a Byte?

Do you have any thoughts on the speed of Sonnet's Echo Express III with TB 2 upgrade? I was thinking of buying one for a new Mac Pro and putting inside:
Atto H6F0 (x8)
Red Rocket-X (x16)
Decklink SDI (x1)
LTO drive
Blu-Ray burner

That makes a hell of a computer in a 2U package. But will it choke on TB 2 bandwidth?

 

[Jeff Kilgroe]

I get what you're saying about LaCie units not properly spanning.
Wow, I totally messed up on that calculation. *Forehead slap*
But I am wondering, why do you divide by 10, aren't there 8 bits in a Byte?

There is a 2-byte overhead so the interface actually works on a 10-bit byte. Same with firewire, USB and most other intermediate interfaces.

Do you have any thoughts on the speed of Sonnet's Echo Express III with TB 2 upgrade? I was thinking of buying one for a new Mac Pro and putting inside:
Atto H6F0 (x8)
Red Rocket-X (x16)
Decklink SDI (x1)
LTO drive
Blu-Ray burner

That makes a hell of a computer in a 2U package. But will it choke on TB 2 bandwidth?

That might be a bit nuts for one PCIe expander box or to pipe through a single Thunderbolt port. It all depends on how you use the hardware. The Blu-Ray and LTO are trivial and already would occupy two of the ports on the ATTO controller. The ATTO controller or the Rocket can saturate the TB2 port on their own if you push them. Why would you want that specific SAS controller? Seems like if you want that many internal ports, you would go for the RAID version. One Thunderbolt 2 port is 20Gbps or about 1800MB/s maximum throughput. It's the same total bandwidth as an original Thunderbolt port, however it's not separated into two allocations. I think that was just too complicated for most peripheral makers to work their heads around and this makes more sense to allow full use of the bandwidth if a device can do it. If you could span all that across two or three TB2 ports, then you would be doing great. There have been some rumors of upcoming expander boxes that connect via more than one TB 2 port, but I haven't seen anything officially announced. The Thunderbolt 2 spec does allow for bonded links in the same way ethernet or infiniband can be doubled up for increased throughput. A Thunderbolt 2 port (or Thunderbolt 1, if used properly) is effectively the same as a PCIe v2 X4 slot.

 

[Elliott Balsley]

Is there no way to use both channels of a single TB2 connection for 40Gbps, in lieu of daisy chaining a second device? Hopefully by the time Mac Pro ships, we'll see some more expansion options.

What I really want is a SAS card with 2 ports inside, and 2 outside, but I can't find one, so I'm looking at that Atto. I would just sneak the cables in somehow . I need one SAS for the LTO, and one SATA connection for the Blu-Ray inside. The one SAS for an external JBOD (software RAID) and one eSATA fanout. So I would not be pushing the Atto card all that hard I guess.

 

[Eric Bowen]

"There is a 2-byte overhead so the interface actually works on a 10-bit byte. Same with firewire, USB and most other intermediate interfaces."



Thank you for that info. I always wondered why it was listed as 20Gbit/s instead of 16.

 

[Mike Nagel]

in my understanding:

TB uses a 8b/10b encoding (same as PCIe v2). So u get 80% maximum effective bandwidth with 20% overhead.

TB 1
w/ overhead - max. total bandwidth: 10 Gbps = 1.25 GB/s | w/o overhead - max. effective bandwidth: 1 GB/s = 1024 MB/s

TB 2
w/ overhead - max. total bandwidth: 20 Gbps = 2.5 GB/s | w/o overhead - max. effective bandwidth: 2 GB/s = 2048 MB/s

Anandtech was able to achieve 1002 MB/s throughput in a test setup with SSD's in a Promise Pegasus R6 on TB 1, so very close to the theoretical limit of 1024 MB/s. See here for more details.


- M