File Server Crossroads

[Todd Sansom]

disclaimer: This is not my area of expertise. I know enough to be dangerous through the years of research mixed with trial and error. I realize we do things that may not make sense or might be antiquated compared to the over-built workflow and lifestyle we have created with some of our clients. So I'm open to thoughts and wisdom for sure. With that said I'll proceed.

We have had an 8Gb XSAN backbone for many years supporting 6 edit bays. It uses one E830FD Promise enclosures and one J830 with another E830FS sitting in the wings for travel.

In January purchased a JellyFish from Lumaforge that has 10GbE. We have worked with them on tweaking things and have really loved it. We travel with it and just two weeks ago I had 5 WEAPON 8K and 1 PhantomFlex 4k feeding it 35TB in 3 days. I had multiple editors editing with native 8K R3D files. While my DIT was offloading and making copies. The JellyFish is an amazing machine.

Back at the studio, I have a problem where I like to always have rapid access to everything in the archives. We often go to archive footage for a host of needs. Our current XSAN (active storage) is supported by an army of Pegasus R6, R8 v2 and v3 (cold storage) detached and waiting to spin up and pull from. I would prefer to have it all active and not need or use the Pegasus army.

So my options I feel work best or that are best known (tried and tested) are:

1- buy a 45drives s45, use a box of old 6TB drives to populate it with 144TB. Offload the XSAN to the s45 and then rebuild the XSAN with 10TB drives and all three E830s
PROS:
- 600TB working space on the XSAN plus the s45
- use mostly all in house hardware
- cost 40k (36k in 10TB drives)
CONS:
- be limited to 8Gb fiber
- stay in a semi stale XSAN environment
- possible struggles handling 8K workflow and beyond

2- buy a second 10GbE JellyFish with 200TB Raw, 117TB working and also build a s45 with 200-500TB of additional storage and migrate all the footage assets.
PROS:
- superb performance and excellent throughput
- built with 20GbE and 40GbE upgradeability
- less switches and no MacPro tower metadata controllers needed (less crap in the rack)
- leverage massive amounts of RAM and SSD caching for better performance
- no IO boxes with computers that have 10GbE, like the newly announced MacPro and iMac Pro or most any PC
CONS:
- cost is 50k - 80k
- new hardware
- pull CAT7 to all the edit bays
- smaller storage capacity

3- Do as we've done but rebuild XSAN with current hardware and buy more Pegasus or G-Tech shuttles, dump footage on them
PROS:
- a workflow we know and have had success with
CONS:
- feels like we are putting a bandaid on a punctured artery

I should also mention that every ounce (bit) of footage we create lives on 2 hard drives (lacie rugged) as a master and clone, as well as LTO all in different locations. So we have between 4 and 5 copies of everything we film. (cue everyone rolling their eyes)

Thanks to all in advance. I appreciate everyone's knowledge and time offered.

 

[Simon Dunne]

Ok, so you've got your online storage sorted. The Jellyfish. You've got a nearline server too, the Promise, and you've got your archive as well, being a mix of drives and LTO. Nice.

So the question here is, what's the best nearline replacement for your needs. You've moved away from fibre to 10GbE, good move.....continue with that. No one likes MDC's!

So, you've got a few options here. You could build a tape archive, with a robotic arm, and dump back the files you need to a 'smaller nearline' to then pull onto your online as and when required.

Or, you can buy a larger nearline that's not as spec'd out at the Jellyfish, so it'll be cheaper. Loads of options out there to rival in price.

Or, you get another Jellyfish. You use you current one on location it sounds like. Getting anohter wouldn't really act as a DR solution, as it would be full of data that would need to be moved when you really need it. Find out if you can rent one for DR purposes. For a permanent solouton though, you just need a 10GbE NAS. No need for 40Gb back bone or Cat7. All that money need to be directed to your online storage pool budget. Nearline storage just needs to be fast and reliable. Steer clear of Qnap for now. Nice idea, but too many failures. That's too cheap, you want something in the middle. EVO, Infotrend, Tiger, but i'd alway reccommned Space as they KNOW what they're doing and their OS is solid.

 

[Misha Engel]

I kind of know now what you have, but what do you want? Try not to think in products but in specs like: How redundant, what dataspeeds at the workstations, distance between storageserver and workstations, size of the storage, etc...

 

[Todd Sansom]

Simon, Great insight. Thank you for sharing you intro and knowledge. In the end it will come down to funds. The idea you have listed here give some great options for that and I really like expandable and future use in mind.

Cost sure it might actually be that I build a nearline storage for Uber 20k and then rebuild the current can with hardware we already have. And in the next 24 months bring another jelly into the fold.

Misha, thank you as well for your thoughts. I think I have that pretty well figured out. No run will be longer than 35 meters. We will build groups that are RAID 6 with one hot spare too. Drives will be in groups of 10 to 15. I'd prefer to have 10GbE + to each station. 8Gb fiber is working fine right now. We need to be able to achieve 6000MB/s read speeds to saturate all the workstations and ideally 1Gb/s 2k even better for write speeds.

The Jellyfish handles this need. The xsan seems to be at it's limit, in is current build.

 

[Misha Engel]

Count on 88..105 USD per TB in the 400..1000 TB storage.

We have choosen for ZFS raidz3 + 1 hot spare.
With our test storage server (XEON-D, 128 GB-RAM, 12x RED PRO 8 TB, 64 TB usable storage, dual 10Gb RJ45 ethernet) which runs about a year now we are happy.
It is fast because of the RAM and de XEON-D, we can saturate with both read and write 1 10 GB line.
The reason we went for ZFS is that raid 5 is dead already, raid 6 is dying and ZFS solves bit-rot on the HD's and runs on almost any hardware.


The change that you might loose your data because of disk failure(s) with 10 TB drives (MTBF Best estimate real world 250k, Nonrecoverable Error Rate 10^15 + hot-spare in use in al cases).

RAID Level 1 Year 2 Years 3 Years 4 Years 5 Years 6 Years 7 Years 8 Years 9 Years 10 Years
RAID 0 53.66306 % 78.52888 % 90.05094 % 95.38991 % 97.86383 % 99.01016 % 99.54134 % 99.78747 % 99.90152 % 99.95437 %
RAID 1 0.00413 % 0.00826 % 0.01239 % 0.01652 % 0.02066 % 0.02479 % 0.02892 % 0.03305 % 0.03718 % 0.04131 %
RAID 5 0.80775 % 1.60898 % 2.40373 % 3.19206 % 3.97403 % 4.74968 % 5.51906 % 6.28223 % 7.03924 % 7.79013 %
RAID 6 0.00635 % 0.01271 % 0.01906 % 0.02541 % 0.03176 % 0.03811 % 0.04446 % 0.05081 % 0.05716 % 0.06351 %
RAID-Z3 0.00001 % 0.00002 % 0.00003 % 0.00004 % 0.00004 % 0.00005 % 0.00006 % 0.00007 % 0.00008 % 0.00009 %

 

[Simon Lok]

I second Misha Engel's comment. I use ZFS RAIDZ2 volumes served via vanilla hardware (generic PC, usually a Supermicro motherboard in a fat chassis with as many 8 GB drives as you can fit in the chassis / you have SATA ports for, 10 GbE onboard Ethernet, 64 GB RAM for ZIL ARC etc. caching). It works wonderfully.

 

[Todd Sansom]

Misha and Simon, brilliant insight. RAIDz3 + hot spare sounds like a solid solution from what u can tell. And zfs with it's self healing properties. Why not! We have 256 GB of RAM in our jellyfish and it is a hound of a machine.

45 drives box has significantly less RAM but they say it works perfectly as a nearline (cod storage) and we c an send and store media on. But serve up archive footage much faster than an LTO library.