Storage vs editing drives/workflow vs cost

[Rui Guerra]

Hi all,

I'm running (again) out of space in my external raid drives, and need a solution fast because of new editing and more shooting I have in hands.


What do you guys use regarding the dilema of disk speed for video editing vs disk space for storage (i.e. not currently been edited). Since Thunderbolt II (that's what my iMac have ports for, besides USB3) drives are much more expensive then USB3 drives, does it make sense to have cheaper USB3 drives with lots of TB just for storage and, when needed for some editing, copy the needed footage for faster RAID5 drives? I know that this possible solution means that I always need extra space available in my RAID5 drives and the added time to make the files copies from the slower drives to the fast ones.

What appeals to me is the thousands of dollars that can be spared if (in theory) we just have one big RAID5 drive for editing and all other are regular USB3 drives for storage.

Just for reference, I have two 12TB in RAID5 Promise Pegasus2 R6 Thaunderbolt II drives (and some WD 8TB RAID1 thunderbolt drives). Each of the Pegasus2 R6 cost around $2.400 USD and will have only 10TB od effective disk space. If I buy a simple 4 TB USB3 drive it will cost about $150 USD, so, for around $500 USD I will have the same 10TB of disk space and almost $2.000 USD extra in my wallet

Looking forward for your inputs.

Thanks!

 

[Misha Engel]

First buy a USB 3 SATA Dock (something like a Sharkoon Quickport XT USB 3 V2 for around € 25)
Replace the disks in one of the Pegasus 2 R6 with something like 6x WD Red Pro NAS Drive 8 TB (6 x € 310 = € 1.860) and use the old disk from the pegasus for USB storage via the USB dock.

NEVER HOT SWAP THE DRIVES FROM THE USB DOCK

Look here to see if raid5 is save enough for you or that it would be better to configure it to raid 6 https://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/

When you need more data storage you can convert the other Pegasus or just buy some Seagate Barracuda compute 3,5" drives of 8 TB for around € 200/drive.

 

[Maciek S]

Also just to add an option

people are going as high as 10/12 TB drives with these raids successfully incase that's an option for you.

https://forum.promise.com/thread/10-tb-ghst-he-drives/

 

[Tim Gerhard]

Have you considered archiving to LTO tape? Once you buy the tape drive, the tape cartridges are cheap going forward and keep lowering in price.

LTO8 12TB - $150ish

LTO7 6TB - $68ish

LTO6 2.5TB - $25ish

 

[Antony Newman]

If you don't need rapid access to old data - LTO tape currently has the best cost : Approx $10 per TB

If you would rather have a simple (as Misha suggested) USB dock solution.
Backblaze give an insight as to how the costs of HDD is reducing according to capacity : www.backblaze.com/blog/hard-drive-cost-per-gigabyte/
In the US, HDD prices are down to $19 per TB (for 3-4 TB).

AJ

 

[Rui Guerra]

Also just to add an option

people are going as high as 10/12 TB drives with these raids successfully incase that's an option for you.

https://forum.promise.com/thread/10-tb-ghst-he-drives/

This looks like a very cost efective solution...although I wonder how safe is to open the Promise Pegasus2 raid system and replace the HD with another ones (different brand, different capacity).

I guess that to be able to make that transformation, I need a spare big drive to first copy all the files in my Pegasus and only then I can replace the drives. SO that means I'll have to buy an additional 10 TB drive to make that temporary copy...

LTO tape does not appeal to me.

 

[Simon Dunne]

So you need tiered storage. Fast to edit, and slow for archive. Some boxes can manage this for you, promoting archive to the faster cache and vice versa. For the sort of budget you're discussing, i'd highly recommend looking into this:

https://www.qnap.com/en-uk/product/tvs-1282t3

Fast, expandable, USB-C, 10Gb, install windows/linux and use the CPU as a render farm.....lots of options. 8x10Tb in raid 6 will give you VERY decent speeds and lots of redundancy.

 

[Misha Engel]

How about this solution Rui?

A thunderbolt 2 dock with 2 slots for 2.5"/3.5" slots OWC Drive dock € 250
In one slot you can put the spinning (storage) drive 8TB seagate barracuda 3.5" for roughly € 200 (you won't find much cheaper than this).
https://www.seagate.com/www-content/datasheets/pdfs/3-5-barracudaDS1900-10-1802GB-en_GB.pdf
And in the other slot you put a SATA SSD of your choice for editing/scratch drive let's say crucial mx500 or Samsung 860 EVO 2 TB for roughly € 450

You can daisy chain this dock with your pegasus to store your work in progress (When finished off-load it to a cheap storage drive "even better on 2").

https://www.owcdigital.com/products/drive-dock-thunderbolt-2

This is the cheapest flexible solution I can think of and it is still pretty fast.

 

[Misha Engel]

One more thing.

You can store the spinning drives in something like this

https://getprostorage.com/hard-drive-storage-cases/

 

[Michael Wahba]

Misha, do you know of any TB3 solutions that are similar to the one you just posted? Interesting....

 

[Noah Yuan-Vogel]

The important question is what speeds do you need? Do you definitely need more than 300MBps? And how much storage do you need? And how quickly will you need to expand your storage?

Your problem seems to me to be that your current storage system is based on 2TB drives. If you just upgrade the storage in one of your Pegasus RAIDs you have the best of both worlds... Lately the best value are the 8TB WD drives I've been getting for $130-160. Drop as many as you need in your Pegasus and call it a day. For about $900 you could upgrade to a 48TB RAID and call it a day, or perhaps put only 3-4 10 or 12TB drives in and add more when the prices drop later if you want the option for more than 48TB in a few years without having to trash/repurpose all your 8TB drives.

I would NOT recommend LTO unless you have a budget of at least $5k for archiving and expect to need over 100TB of archived storage or need a more robust long term media-on-the-shelf storage solution. Tape is more trustworthy on the shelf long term than hard drives but it's also a painful medium to deal with if you are retrieving data frequently, near-line online storage in my opinion is much more convenient and cost-effective in my opinion. I would also recommend not using a pile of separately formatted and casually cataloged single drives sitting on a shelf. Frankly if you are coming from using 2 10TB volumes and just need an upgrade from that, then LTO makes no sense at all. There are so many options for you under $1k that will give you plenty of speed and keep all your projects centralized.

 

[Misha Engel]

Misha, do you know of any TB3 solutions that are similar to the one you just posted? Interesting....

Sort of, a dock compatible with Thunderbolt 3 USB-C port, USB 3.1 (10Gbps) Standalone Duplicator Dock for 2.5" & 3.5" SATA SSD/HDD Drives - with Fast-Speed Duplication up to 28GB/min.

https://www.startech.com/HDD/Duplicators/standalone-duplicator-dock~SDOCK2U313R

 

[Bob Zelin]

I am surprised that Misha has not recommended 45Drives, that uses FreeNAS. After all, you could simply hire Misha to set this up for you, and pay for his administration costs !

Bottom line - if you are successful, you will ALWAYS require more storage, and ALWAY need to purchase more storage. You can do this with Promise, G-Tech, QNAP, cloud storage, and so many others. The bottom line is that if your real question is "how do I not spend any more money, as I continue to grow my business, and need more storage space"- the answer is simple. You can't. If your business expands, you need to INVEST MORE MONEY into your business to accommodate the amount of storage that you need. This means that instead of buying that new boat, or new car, you need to INVEST in your company, so that you can have storage capacity for the business that you require - so that you can STAY in business.

There are countless wonderful solutions out there - all it takes is for you to get a crowbar into your wallet, and spend some of those profits !

Bob Zelin

 

[Misha Engel]

You can spend your money only once Bob.

Those 2 pegasus2 R6 are pretty fast https://www.macworld.com/article/2090833/pegasus2-r6-review-six-hard-drives-thunderbolt-2-and-a-lot-of-speed.html when they are empty and he already invested in them.

P.s. I don't use 45drives

 

[Rui Guerra]

(...), or perhaps put only 3-4 10 or 12TB drives in and add more when the prices drop later if you want the option for more than 48TB in a few years without having to trash/repurpose all your 8TB drives.

(...)

Noah,

If in a RAID 5 6x 2TB system I replace just a few of them with bigger drives, those bigger drives will only have 2TB of it's storage capacity used by the RAID 5, right? I mean, RAID 5 is based in equal amounts of space used between all the drives, which means the the smallest drive in the RAID is the only one that gets 100% used. I'm I right? If so, how can I replace just a few of the drives in the PEagsus2 and have more storage available?

 

[Blair Thornton]

Misha, do you know of any TB3 solutions that are similar to the one you just posted? Interesting....

I use an Akitio Thunder3 mini https://www.akitio.com/portable-storage/akitio-thunder3-quad-mini and backup daily.

See my portable workflow here: http://www.girafxmedia.com/index.php/2017/11/26/red-4k-portable-online-workflow/

 

[Rui Guerra]

(...)
Look here to see if raid5 is save enough for you or that it would be better to configure it to raid 6 https://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/
(...)

Misha,

I may be wrong but I think Pegasus2 RAID5 cannot be configured to RAID 6 (I think the options are RAID 0, 1, 5 and 10). Am I mistaken?

How about this solution Rui?

A thunderbolt 2 dock with 2 slots for 2.5"/3.5" slots OWC Drive dock € 250
In one slot you can put the spinning (storage) drive 8TB seagate barracuda 3.5" for roughly € 200 (you won't find much cheaper than this).
https://www.seagate.com/www-content/...02GB-en_GB.pdf
And in the other slot you put a SATA SSD of your choice for editing/scratch drive let's say crucial mx500 or Samsung 860 EVO 2 TB for roughly € 450

You can daisy chain this dock with your pegasus to store your work in progress (When finished off-load it to a cheap storage drive "even better on 2").

https://www.owcdigital.com/products/...-thunderbolt-2

This is the cheapest flexible solution I can think of and it is still pretty fast.

That can be a good solution but it takes a lot of copy work/time between drives. Or am I missing something here?
That Thunderbolt2 dock could be very useful to continue to use the smaller drives that are now inside the Pegasus2.

Right now, I'm thinking in following direction, but still have some doubts:

1 - Is it possible to replace the drives inside the Pegasus2 (6x2TB) with much bigger ones (lets say 6x10TB) without reliability issues?

2 - I think the answer is "no", but anyway, is it possible to replace not all, but just a few of the drives inside the Pegasus2 with a bigger ones and been able to use the added storage capacity of those bigger drives ant still have the RAID5 working?

3 - In any of the options above, am I correct assuming that first of all I need to buy some big drive to make a copy of the entire Pegasus2 before proceeding to the drive replacement procedures. For that, maybe the Thunderbolt2 dock with one 10TB drive is the best solution?

4 - Looking at the 8 to 12 TB drives out there, in the 7200 rpm type, which ones are better option: WD or Seagate? Or another ones?

5 - Any additional advice?

Many thanks!

 

[Misha Engel]

Misha,

I may be wrong but I think Pegasus2 RAID5 cannot be configured to RAID 6 (I think the options are RAID 0, 1, 5 and 10). Am I mistaken?

The Pegasus2 R6 supports the following RAID levels: 0, 1, 5, 6, 10, 50

That can be a good solution but it takes a lot of copy work/time between drives. Or am I missing something here?

It depends on how you do it and how your workflow is. The main goal was cheap and flexible hence no LTO but harddrives. The drawback might be that somethimes you have to some extra transfers between the different storage solutions you have.

That Thunderbolt2 dock could be very useful to continue to use the smaller drives that are now inside the Pegasus2.

Yes it gives you flexibility.

Right now, I'm thinking in following direction, but still have some doubts:

1 - Is it possible to replace the drives inside the Pegasus2 (6x2TB) with much bigger ones (lets say 6x10TB) without reliability issues?

What I know of that it is tested with drives upto 6 TB, I don't see a reason why it wouldn't work with bigger drives.

2 - I think the answer is "no", but anyway, is it possible to replace not all, but just a few of the drives inside the Pegasus2 with a bigger ones and been able to use the added storage capacity of those bigger drives ant still have the RAID5 working?

No.

3 - In any of the options above, am I correct assuming that first of all I need to buy some big drive to make a copy of the entire Pegasus2 before proceeding to the drive replacement procedures. For that, maybe the Thunderbolt2 dock with one 10TB drive is the best solution?

Well since you need to back up all important data (just raid is not a backup) and you have 2 pegasus2 R6 storage devices I would opt. for the lowest price/TB drives of which the Seagate Barracuda Compute 3,5" 8 TB is one of them €25/TB. With an 8 TB drive you have about 8 x 1,024^3 = 7,45 TB formatted capacity (don't put more than 7 TB on it). With 3 of those drives you should be able to backup both of your Pegasus2 R6 storage devices.

4 - Looking at the 8 to 12 TB drives out there, in the 7200 rpm type, which ones are better option: WD or Seagate? Or another ones?

Well they are all noise because of the 7200 rpm. The most appealing to me are the Seagate Ironwolf 8..12 TB. Purposed build for NAS/DAS upto 8 drives in 1 housing and the best price/perf drives for this purpose.
The 8 TB will set you back around €225, 10 TB around €300 and the 12 TB around €400.

Regarding drive statistics, you have lies, f*cking lies and drive statistics.[/QUOTE]

5 - Any additional advice?

Many thanks!

 

[Michael Wahba]

I use an Akitio Thunder3 mini https://www.akitio.com/portable-storage/akitio-thunder3-quad-mini and backup daily.

See my portable workflow here: http://www.girafxmedia.com/index.php/2017/11/26/red-4k-portable-online-workflow/

Blair, is this quick enough to edit 5k RAW off of your MBP 15inch? I have the same setup basically and this drive solution looks solid.