The fastest super speed RAID setup

[Christoffer Glans]

I am trying to find the fastest possible RAID setup without blowing a hole in my bank account. The issue is that people don't have their facts straight when talking about this.

Like...
* Confusion about Gbit/s or GB/s - Mbit/s or MB/s
* SATA 1 (1,5Gbit/s), SATA II (3Gbit/s), SATA III (6Gbit/s)
* USB3.0, eSata 3Gbit/s/6Gbit/s, Thunderbolt
* Disk speed 5200rpm, 7200rpm, 10000rpm
* SSD's speeds
* Continous read/write vs buffer/cache read/write

Now, I'm looking for a very fast RAID setup that would max out the USB3.0 (5Gbit/s) transfer rate and in addition to that (because I don't have thunderbolt yet), a RAID setup that maxes out the Thunderbolt (10Gbit/s). This is intended for continous read/write, playing and rendering video media (5K, 4K, 1080p and so on)

What I'm looking at is something like this 2xHDD or 5xHDD.
I'm trying to figure out the math here, but I need some help. If I were to put five 7200rpm drives into the 5xHDD Raid and set it to RAID0, Would I max out the 5Gbit/s USB3.0 or would it never reach that speed? In addition to that, if I were to put five SSD's into a RAID0 config with SATA III and Thunderbolt, what speeds would I reach?

Do I count the read/write of the SSD or HDD in addition with the SATA performance and multiplying it with the number of drives in the RAID config and then take that number and compare it to the theoretical speed of the connection? (never mind the computer, let's say the motherboard can handle the full speed of both Thunderbolt and USB3.0).

So if I have five SSD's that read/write at around 450Mbit/s it would be like this:
SATA III (6Gbit/s) = 600Mbit/s - 450Mbit/s = All good for each SSD internal.
450Mbit/s x 5 = 2250 Mbit/s (22,5Gbit/s)
22,5Gbit/s in Thunderbolt = Overkill?

What I'm looking for is something that maxes the speed when working in DaVinci Resolve or editing high res files in other software like Adobe Premiere etc. (5K uncompressed for example).
Now, to be realistic, how do I max out my USB3.0 speeds? (5Gbit/s). The 2xHDD I linked to earlier features SATA III and two slots. If I were to put two 500gb SSD's that possess a read/write of 500Mbit/s / drive I would get a RAID0 of 1tb with a speed of 1000Gbit/s, which is a lot more then the USB3.0 data rate.

So what is the best option? Might it be more economical to put five regular HDD's at 7200 RPM in a RAID0 config with USB3.0? Most 7200rpm max out at 80Mbit/s with SATA II. So I would not need SATA III in the RAID itself.
So 5 x 80Mbit/s = 400Mbit/s. It does seem like the best config for USB3.0, but it's still a big leap from that to 1000Mbit/s with the two SSD's.

Now looking at the speeds needed. I got the AJA calculator on my phone and it says that uncompressed DPX's 16bit RGB 4K at 25fps produce a needed speed of 1.9Gbit/s.



How do create the best possible RAID for 1.9Gbit/s? Anyone got any good tips?

 

[Craig Parkes]

A huge factor in all this for real world usage is the RAID controller and how full the drives are etc. Transfer speeds of the drives will not scale linearly.

For reference, a new highpoint 2322 card and 8 bay storage array in RAID 5 I got about 3 years ago was running at around 600MB/s read/write when empty. That's about 4.8 Gbit/s and so technically even that 8 bay afraid wouldn't have maxed out USB3.0 'technical' max speed with Redundancy, on SATA II drives. The thing being I had 8 of them and a proper Raid card at my disposal.

Five 7200 RPM drives in RAID 0 should provide significant performance, but how is the raid setup on the USB3.0 enclosure? Part of the hardware, or software (OS) raid? How the RAID is set up and talking across USB 3.0 will potentially be a bottleneck in practical terms, you really need to ask questions about a particular group of products, not a theoretical set up, to get a useful response.

 

[Christoffer Glans]

Alright, thing is, I'm looking for just the right product for this
Do you have any recommendations to storage arrays that might achieve what I'm trying to achieve?

 

[paul schefz]

pegasus r6 12tb raid 5 (10tb available) runs at about 600mb/sec read/write....putting in ssd would speed things up even more but i am not sure how much more speed you need (epsecially for large continuous r/w) and the price/space value would drop considerably....
pegasus seems to have a special sauce because none of the other thunderbolt raid are that fast....
usb 3 maxes out at 500mb/sec, i doubt you can get much faster then 400r/w in real world....and that is only one direction, thunderbolt has twice that up and down....

 

[Craig Parkes]

Alright, thing is, I'm looking for just the right product for this
Do you have any recommendations to storage arrays that might achieve what I'm trying to achieve?

Most people recommend dedicated raid cards plus an enclosure if you are looking at RAID 5 because the raid cards generally offer the best ability to successfully rebuild and monitor array - enclosure only builds are a little bit harder to troubleshoot diagnose problems with, especially the 'fully automated' ones with just dipswitch settings. ATTO comes highly recommended, I've had success with Highpoint.

Price point dictates what 'just the right' product is. In the real world there is very little opportunity for actual 'overkill', generally there is always a bottleneck somewhere but you still want the best combination of speeds and reliability you can afford rather than trying to make everything as 'efficient' as possible based on technical specs, because reality never matches up to technical specs.

Two 500GB SSD's in Raid 0, are probably very reliable but you are probably going to run out of space very quickly - 1 TB of usable space isn't all that much if you have projects overlapping - which is why most people opt for more cheap spinning drives with some sort of redundancy.

Spinning Drives in RAID 0 are great for speed, but require you to have a robust regular (i.e daily) backup because for each additional drive you are creating an additional opportunity for all the data to be lost. 5 drives in Raid 0 may be very fast, but in the real world across a couple of years the chances of 1 of those drives failing would make me prefer 8 drives in Raid 5 - likely to be similar if not greater speed and higher redundancy with a proper raid card.

Seeing as you need both USB3.0 and potentially want thunderbolt, you would need to look only at enclosures that have both, and capability of the enclosure is a far bigger factor than the drives (because there are a far greater array of drives all at similar price points then there are enclosures.)

Off the top of my head I don't actually know any multi drive enclosure that has both at the moment.

Also a real world factor is that you are basing your numbers off 1 stream of DPX, generally you want to have significant overhead above the video playback stream you are using to ensure smooth playback as the drive gets more full, the Pegasus r6 offers that for thunderbolt in Raid 5 or 0 according to these comparisons: http://reviews.cnet.com/external-hard-drives/promise-pegasus-r6-6tb/4505-3190_7-34855279-2.html

The G-RAID Thunderbolt Enclosure offers lesser speeds, but would get the job done: http://www.storagereview.com/hitachi_graid_with_thunderbolt_review_8tb but is likely to slow down more dramatically as the drive gets more full.

I haven't seen any two drive USB 3.0 enclosure with multiple ports, and none that could exceed 200MB/s on one port - which is what you say you want. You are going to need a raid card of some sort and an enclosure to match OR one of the thunderbolt options -

This article talks about a HighPoint card that might work for your needs: http://www.barefeats.com/hard142.html where you could attach multiple USB 3.0 drives and raid them together, OR if you could find an enclosure with multiple USB 3.0 ports (one for each drive) that would achieve what you want, but is a bit of hokey set up in terms of connector robustness for me.

What matters is not the speed of the drive alone, or the connector alone, or the raid controller alone, but how all three work together - and figuring that out does unfortunately require real world benchmarking of the entire system. Best to buy a solution that has been benchmarked and is in your price range, then try and cobble together a 'mythical' cost effective solution based on specs and theoretical limitations, there are too many points of bottleneck between or within each device for that.

It seems a single connection USB 3.0 device isn't going to cut it no matter how you shake things out in real world performance, so either get a raid card and a recommended enclosure along with it that's benchmarked (heaps of options there, just depends on budget), or get a machine with Thunderbolt and buy one of the thunderbolt enclosures (limited range, but well benchmarked), or maybe try and hack together a solution using that highpoint USB 3.0 card or something similar (no enclosures that I know of designed to work with that particular card - it's a bit of an outlier but an interesting option.

 

[Colter Ripley]

I would recommend having your OS and programs on the SSD so that is crushing fast. This will allow the program and OS to really hum.

Next to have an internal SSD for the project that you're working on is second priority. Our projects are often up to 1.5TB and that SSD RAID array cost us 1700 or so.

Third priority is that external enclosure. There are a bunch of options but it depends how much storage you need. What comes to mind for me is that you want to choose the fastest connection - Sounds like USB3 and Thunderbolt are the current fastest. Get a controller for either (avail. Mac/ PC) to start. Then build out that box. G-Tech has the most boss stuff that i've seen and ultra reliable too.

I've had RAID's go down before so make a good choice on data management.

PS. Premiere plays back 5k footage at 1/4 resolution no problem on pretty much any computer... 1/2 if rendered.