Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

100TB+ of storage... suggestions

Ivan Kovax

Ivan Kovax

Well-known member
Joined
Nov 19, 2009
Messages
975
Reaction score
0
Points
0
Location
Tokyo, Japan
Hey Guys,

Looking for to trade notes here.

In about a month I am scheduled to commence a pretty major feature film and my early calculations are showing around 100-130TB of data for the whole feature film. I need to make 1 master and 2 clones of the data, so we are currently in the process of deciding on the most ideal solution for this.

My thoughts were (for the master) 5x 20TB raid HDD (possibly using them in JBOD/standalone drives mode as we will have multiple copies nonetheless).

I decided to not try and build a MASSIVE system simply because I need to be at least relatively mobile as there is a lot of location shooting, where having a huge 100TB system is simply unfeasible. So having the whole shoot spread over 5-7 20TB drives (per copy) seems like the most practical solution as of now. Back in japan all the rushes will be coallated onto the one massive 100TB+ storage solution

Additionally, we will need to get dalies back to Japan from Thailand on a regular basis, but I am not confident that the Thai broadband internet is capable of the kinds of data we will be burning. Are there perchance any post houses in thailand that anybody has worked with before that deal with ftping rushes via a fiber network? Otherwise we will be ferrying rushes by plane every few days to japan from our locations.

Effectively on this shoot, we will need to back up 300TB+ of data (master and two clones). Fun...

Cheers, and look forward to hearing everybody's notes and how others have handled shoots of this magnitude.

Ivan
 
I don't have workflow advice for anything of that scale, but I'd suggest calling Park Road Post in Wellington NZ and seeing if you can consult with one of the guys there.

They are a friendly bunch, and if anyone knows what you are about to go through, it would be them.
 
It is very hard to calculate Data, without knowing all the ins of the project, Michael Bay has it's closing in to it's first "Peta" currently I believe working with 850TB, form Maxx Digital.

It depends as you know n so many variables, number of cameras, resolution, FPS, HDRX, Compression chosen and so on...

But I think that your calculations should be affording you a good solid path, just make sure that if need to you can expend those Bays, and or add to, and you are ok, if you can get your storage on a moment's notice there is really no need to load up all upfront.

Best of luck with your project.
 
If I had to plan for that much data, I would use small (16tb) raids in conjuction with an LTO-6 tape library.
The raid would only need to be large enough to get through a day of shooting and then have the tape library create the multiple backups.
This would minimize the equipment needed for acquisition, and make it safer and easier to get the data back to the main system. I would rest easier at night FedExing tape than disks.

As far as finding fast Internet, that reminds me of an old I.T. Saying:
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.
Computer Networks, 4th ed., p. 91
Which would work out to about 500 gig per second, if we don't get a flat

 
Ivan, Im currently using supermicro SC847E16-RJBOD1 chassis, with 3TB seagate drives in them, running freenas (ZFS file system).. I have one online fileserver, and one backup server that clone eachother every hour or so.. Its working really really well.. For the last 1.5 year, I've had 0% errors. I've run Windows storage stuff before that, and it was a constant battle against weird failures as soon as I went over 64TB of data.. Now you can probably go with nice 4TB drives instead giving 180TB minus the drives you need for redudancy.. Since you build a new storage system, two SC847DE16-R1K28LPB servers will be the optimal ones giving you 72 drives in a 4U unit! It doesnt get more compact than that.. Just build a nice cart for em :)

I currently use for the online storage zpools of 6 drives, where two are for redudancy (ZFS-2 (or RAID6 in other systems)), so I add 12TB of data everytime we want to expand. The nice thing is that every time we expand we actually increase the fault tolerance having two redudancy drives for each pool...
 
Ivan Kovax said:
Hey Guys,

Looking for to trade notes here.

In about a month I am scheduled to commence a pretty major feature film and my early calculations are showing around 100-130TB of data for the whole feature film. I need to make 1 master and 2 clones of the data, so we are currently in the process of deciding on the most ideal solution for this.

My thoughts were (for the master) 5x 20TB raid HDD (possibly using them in JBOD/standalone drives mode as we will have multiple copies nonetheless).

I decided to not try and build a MASSIVE system simply because I need to be at least relatively mobile as there is a lot of location shooting, where having a huge 100TB system is simply unfeasible. So having the whole shoot spread over 5-7 20TB drives (per copy) seems like the most practical solution as of now. Back in japan all the rushes will be coallated onto the one massive 100TB+ storage solution

Additionally, we will need to get dalies back to Japan from Thailand on a regular basis, but I am not confident that the Thai broadband internet is capable of the kinds of data we will be burning. Are there perchance any post houses in thailand that anybody has worked with before that deal with ftping rushes via a fiber network? Otherwise we will be ferrying rushes by plane every few days to japan from our locations.

Effectively on this shoot, we will need to back up 300TB+ of data (master and two clones). Fun...

Cheers, and look forward to hearing everybody's notes and how others have handled shoots of this magnitude.

Ivan
Click to expand...

http://offhollywoodreporter.com/blog/category/180tb-of-storage-for-under-11000
 
My advice... do not put all egg in one basket... especially not if you have a lot of eggs.

So I would actually make 2x1tb lacie drives x 100 to 130 or what ever amount you are about to shoot. kind of one disc per shooting day with a duplicate. From there I would create proress proxy for editing and dailies in HD and also something small like 720x404 for web loads and such.

The though of keeping 130TB material on one raid is not vice when shit hits the fan, and with that amount it will. I would not trust the guys selling those kind of raids. I bought from people mentioned in this thread and yes they had a misshap when doing serivce and lost all my material.. that was when they where handeling a 4 tb raid... and the power supply broke for no reason... And with our flames, infernos etc we use the stone and wire fiber drives that cost about 100k USD for a few terabytes... and you know what, they break all the time. I can not even count how many times I lost all material on them. No big deal because I allways work in a way so I can have that happening without loosing to much time before I'm reset and running again. But then again then I normally do not do130TB more like 10TB...


When you are done with editing it's quite simple to suck out the used clips out of the big lot of 1tb drives and make a r3d trim down to one drive and do your grade / online from there. No need to have access to 130TB of data while edting. When you get to those numbers things will for sure start to act up on you. I tried it many times and it's never been a happy ending to it.



So simply.

1.Get a huge discount on your portable drive supplyer and buy a 100 of those. so you can make a vault copy (an I would atually do a second vault backup copy) and a working copy... So you have double or x3 of all shot r3ds.
2. Get two smaller raids for editing. Like 5TB each or such. just for your proress files. Same there one that is a backup of the work disk and one that you work form.

Do number / label your disk correctly and keep a nice shelf to store them in while you edit then it's quite easy to extract what ever is need from 100+ drives. At least thats far easier than trying to problem solve a 130TB raid solution that just does not want to wake up one morning :)

Thats my advice, no matter how you decide to slice it, please let us know how it goes.
 
With the price of USB 2 x 256GB so low ($5.79) - wouldn't it be great if:
+) Someone created a small board with a USB3 (or TB) to and array of 20 x 20 USB 2.0 connectors
+) Each socket had a small Red/Green LED indicating if the USB was connected/ or not working.
+) 256 GB x 400 = 100 TB. Cost = $2520


If the board cost $500 (with profit) ... This would create a book sized 100TB drive for $3000.


With a nice little driver - this would be a shocking performer.


http://www.dhgate.com/product/free-...ive-rotation/163439549.html#s1-1-1|1583429565


You could even start with a few USBs ... and add to it as your need arose.
Then in 2016 start replacing 256GB drives with 512GB on a FIFO basis.


AJ
 
I also think LTOs are a better choice for more than a couple days at a time of your storage..
 
As Todd T. mentioned, I would also recommend using smaller incremental devices and RAID striping and then LTO tape for archival/backup and getting the data between sites.

I don't want this to come off sounding like an ad, but I can only speak from my experience - as an example, we offer 2U rack mounted, 32TB SAS Expander units in our ArGest line that can be daisy chained up to 8 units deep on a single SAS port (16 units per SAS HBA). That comes out to .5PB in 32U of rack space. This would allow you to start with a couple of units and then expand easily as your data collection grows. This can also be paired with a 4U, LTO-6 48 tape library to provide archival and interchange for up to 120TB per load out. In our case, this can all be managed from a Mac Pro, or any Thunderbolt-capable Mac Book Pro, Mac Mini, or iMac. Most of our customers are simply striping these arrays since they are using LTO-5 or LTO-6 to backup and archive the data. One specific large-scale customer's configuration is to RAID-5 each of the 16 drive units using the ATTO ExpressSAS R680 RAID HBA and then striping them together under OS X's Disk Utility for a single volume of high capacity and performance. This gives a relatively safe per-chassis configuration while retaining the highest levels of I/O and capacity.

By using the LTO-6 solution for both archival/backup and interchange, you won't need to worry about broadband issues since you could simply courier the dailies on LTO back to Thailand. You would only need an LTO-6 drive in Thailand for restoring the data - and 160±MB/sec from tape is far faster than trying to push that amount of data over the net.

As an aside, we only use HP tape technology and Seagate's Constellation ES.3 enterprise class drives (1.25 Million hour MTBF), and have witnessed less than .05% failure over the past year of sales.
 
I'm not using LTO's I know, tapes should be more safe. But I had such bad experiences with just about every data backup tape format available before LTO so I just skipped it. The need for a special device to access the material is also another reason. Most people have USB ports, not to many carry around LTO players... in a few years time maybe the USB sockets are no longer around on our computers but long before then the last LTO player has problably broken down.... Or is there anyone here that can open all my DTF backup tapes... or DTF II backup tapes? I know I cant since all my decks broke down several years back.

So I have no experience from LTO but still I would not recommend it.

I do not know how many thousands of external drives I gone trough and I actually never had an issue. As long as you do dual copies I would be in shock to hear that somebody actually managed to damage both master and backup drive in a way so that they lost their data.
 
If you need three copies, I would plan three G-Raid or similar drives for each day. ship these drives off set to post and storage each day. Dont ship them together.

And a good fast raid on set for working drive. Large enough to hold atleast a weeks worth of footage.

I normally have my fast raid for working and two drives for each day. One for post and one backup stored seperately for emergency. Have the post facility or someone make LTO archive of each days drive.

That gives you three easy to manage copies that you get off set quickly and a temp working copy on set.


Dusty
 
Björn Benckert said:
So I have no experience from LTO but still I would not recommend it.
Click to expand...

That's a pretty odd perspective - I'm never tasted Chocolate ice cream, but I still wouldn't recommend it?

LTO is FAR, FAR removed from the old DTF format. It's got a lot broader vendor support, proven long term storage and recoverability, it's much faster, has dramatically higher per-tape capacities, and a long term road map that is currently aimed beyond 2030.

I'm actually sorry that you've never tried to use it. Be bold, try the chocolate ice cream. I've had it, and it's delicious! :thumbup:
 
I've spent a LOT of years cobbing together data storage systems, but something to consider Ivan is what's the value of the data... if you're filming a hundred million dollar feature then get the right hardware. I routinely work with data that has 7 figure value and your data is likely worth much more. Personally I'd be talking to someone who specializes in providing reliable storage systems for film (utilizing enterprise level hardware). If you want your data online/nearline then a rack of enterprise storage (130 TB is small enough that it could be a portable rig) and the backup/multisite storage systems to insure data reliability seems like a no brainer (at least from my non-Hollywood perspective).
 
Tim Jones said:
That's a pretty odd perspective - I'm never tasted Chocolate ice cream, but I still wouldn't recommend it?

LTO is FAR, FAR removed from the old DTF format. It's got a lot broader vendor support, proven long term storage and recoverability, it's much faster, has dramatically higher per-tape capacities, and a long term road map that is currently aimed beyond 2030.

I'm actually sorry that you've never tried to use it. Be bold, try the chocolate ice cream. I've had it, and it's delicious! :thumbup:
Click to expand...



It's more like this... i know candy is no good for you. :)

I'm not an expert but I got a little bit of experience within this field and that experience tells me that tapes are bad. I know they state that they last for ever bla, bla bla... But after a few years it's quite often another standard is the standard. So if people say LTO now, I bet it's not the standard in 5 years time. And then I can tell you that I much rather sit with a plug and play drive (that I also can take apart and access in many different ways) without the need of a player and without the need to offload.

A tape on the other hand is just a tape, that needs offloading etc. Just having the direct axes without having to move data is worth so much to me so that alone makes all Tape based systems a no go.

My experience is also tells me that shit usually goes wrong when large chuncks of data needs to be moved. To scroll trough all the takes on a 2 TB drive is no big deal, having to do so with data tapes is really a struggle.

So what is the reason benefits for using LTO? I know they last "forever" and the price / storage space ratio is good but except for those two?

As I never had drives that died over time / in storage. But I had several tapes doing so. And the price difference is pretty slim. So then all the drawbacks from using tapes, I wonder, are they really worht those two?



This is what I learned when I did the conform and grade for the first ever CINEALTA film. And before that a few dussin of 10bit HD Spirit scanned 2k features and alot of Domino 3k work back in the days (read more than 10 years back) Of course now a days all is different, the files are different the cost of storage is different and the workflows are different. But to me people tend to do exactly the same mistake as we used to do back then which I find quit amazing.

So I still say that,

Raids are nice to play with but they are not safe. Safe is small units that are not connected to any kind of power.
Tapes are like sony, sony is evil. :)
 
LTO has been the standard for all long term archive for close to 20 years now. There was a little competition with DLT, but the high end of every industry in the digital world has been LTO since the 90s.

For the last 10 years LTO has been the only solution for professional long term archive of digital material.


Dusty
 
You must log in or register to reply here.
Back
Top