Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

100TB+ of storage... suggestions

Björn Benckert said:
So what is the reason benefits for using LTO? I know they last "forever" and the price / storage space ratio is good but except for those two?
Click to expand...

  • Speed - 140MB/sec with LTO-5, 160MB/sec with LTO-6 (that's 2.5TB of data processed in ~4 hours)
  • Quick Access - Locate and start restoring any file on the tape in 5 minutes or less
  • Broad hardware vendor support - HP, IBM, Quantum, Imation, Sony, Spectralogic, Oracle, and more
  • Standardized mechanism - all drives are compatible
  • 2 generation backwards compatibility - each new version can read and restore from tapes created on drives 2 generations back
  • Long term road map - current design plans through 2030
  • Stable financial base - none of the primary vendors are in financial trouble
  • Broad standardization and adoption - all major players are standardizing on LTO for archival and interchange
  • Underwriter acceptance - Underwriters accept LTO as valid protection of digital assets


Hmm, I'm sure there are one of two more, but that should get things started ...
 
LTO 6 using LTFS is the way to go.

But an LTO is still a "tape" - so if you should always be making a "master" and a "protection" LTO tape - and they should be stores in separate locations.

I still like having all media from a show "live" on Raid 5 or 6 storage from production to delivery - as it's faster and easier to access media than pulling shots form LTO tapes -and it just makes me feel better. Typically, when we do shows that are in a remote location - and need to ship media to a post facility - we try to keep all media live on / or near location and we either ship drives in a peilican case or LTO tapes to post faciltiy that will then verify media on their end before we delete anything on/near set.
 
Mark L. Pederson said:
... or LTO tapes to post faciltiy that will then verify media on their end before we delete anything on/near set.
Click to expand...
If you are using LTFS on those tapes, exactly how does the post facility verify the media? Since LTFS has no verification capabilities short of your providing an external, manual checksumming mechanism (or restoring every file to another volume), I'm very interested in the process that you utilize.
 
That's easy; if you use transfer software (e.g. shotput pro) that generates checksums files and saves them along side the footage, then everything gets copied to the LTO tape.

Whoever gets the tape also gets the checksums with which to verify the footage.
 
Todd T said:
That's easy; if you use transfer software (e.g. shotput pro) that generates checksums files and saves them along side the footage, then everything gets copied to the LTO tape.
Click to expand...

Ah, So it does require an additional bit of non-free software to make that work. That's what I was curious about.
 
Björn Benckert said:
When you are done with editing it's quite simple to suck out the used clips out of the big lot of 1tb drives and make a r3d trim down to one drive and do your grade / online from there. No need to have access to 130TB of data while edting. When you get to those numbers things will for sure start to act up on you. I tried it many times and it's never been a happy ending to it.
Click to expand...
I think this is great advice. Even Peter Jackson's editor for The Hobbit, Jabez Olssen, just edited the entire movie using Avid DNxHD files, and only at the very end did they finally conform the R3D files and start color correcting. And for that, they just grabbed a "pull list" and put just the actual takes used in the film on servers. There's no need to edit in 4K if you can see a reasonable mildly-compressed picture in HD. You figure the editor is sitting about 3 feet from the monitor, and it isn't necessary to judge performance or the timing of a scene in 4K.

Jabez tells his story here:

http://www.postmagazine.com/Publica...012/December-1-2012/Edit-This-The-Hobbit.aspx
 
Tim Jones said:
An entry price of $57K or $6,000 per TB??? I think I need to revisit our price list :out:.
Click to expand...

Not before the Group Buy ends, I hope...

Thank you very much

Fury
 
you dont want to take a big raid with you on set, too heavy, too much risk when power goes down,
and if you add an battery, i wouldt want to move it arround...
There exists now a compact RAID box from Promise with 60 disk inside a 4U chassis.
its SAS, you would need ta add a extra fibrechannel master chassis, or a RAID SAS card.
for fast transfers, add a 10Gbit switch, add LTO, and you get a nice rack.
-- for inspiration : http://www.fcp.co/final-cut-pro/new...age-ready-for-4k-editing-with-final-cut-pro-x
BUT perhaps you could do it this way - i would need more information to propose a optimal setup.

i dont know the project, but with the 100-130TB you're talking,
i guess you will have arround 500-800 hours of material,
lets take 500hrs for the calculations
say 5k, 6:1 compression ration, take 70MB/s, or 250GB/hr for the raw files.

500hrs would then take for 1 copy : 500x250 = 125TB -- 42x 3TB disks

it all starts on set :
you dont want to carry arround big raid, and you need to make multiple copies,
i would recommend using a lacie 2big : it takes 2 drives in JBOD, so you can make 2 copies.
use software with checksum.
You could set the drives in RAID0 for extra speed, but then you need a second 2big,
and be very carefull not to mix drives when swapping. IF you need the extra speed for copying,
which will be probably the case, i would recommend adding a second copy station with a second 2big,
you get extra speed - stereoshoot ?: camera a = computer a, commera b = computer b
and if a computer has trouble, you have redundancy.
if you have time enough, you could even start making the offlines on set after the copy -
use a macbookpro retina 15" with the faster gpu, which hopefully adds extra speed in the near future.
disks would be best 3TB, price/capacity, so for one copy you would need 42 disks in total,
or 82 for 2 copies- you can buy them as spare parts from lacie, or use the disks for the 5big's - see lower.

for the offline :
if avid : dnxhd36, would take at 3,75MB/s, 17GB/hr :500 hrs = 8,5TB
if fcp-fcpx : proxy half size of full hd , would be between 20GB/hr or 13GB/hr, 500 hrs = 10 to 6,5TB.
so the offlines would fit comfortable on a small Promise Pegasus R6 raid, 18TB brut, 12TB nett in RAID6
if you would need higher quality offline, there are bigger raid's available from areca, or thunderbolt to SAS/fibre solutions

you buy 2 of these RAID's, 1 is backup, connect them to a big iMac and make the accessible
over ethernet : on the master machine, you can do the editing, dailies, but on a second
machine you could do the offline calculations, dailies, and perhaps the 3rd copy on 1 or 2 LTO tapes, or a 3rd disk set.
for LTO, i would use a single drive SAS version and make 1 tape per 3TB disk
with LTO6 - you would then have perhaps only 2,5TB on the disk, i did not yet
test of compression still can do something, so take 50 3TB iso 42 disks.
Ethernet connection between those 2 editorial machines can be standard 1Gbit, as you only copy
an offline to the editing machine.
With the vast amount you hours your are talking, you need to check if the offline calculation
will be fast enough on 1 machine - you could do a 1/4 debayer, which is very fast,
and if necessary add an extra machine to double your calculations, and/or add a RED rocket card.
for receving the set disks, i would recommend buying a thunderbolt 5big : you can put 5 trays jbod
in them.
This will also be very usefull once you start onlining: at that moment you will need to have a lot of
the setdisks with the raws online : each 5big takes 5 disk, per thunderbolt you can have 6 of them,
so 1 thunderbolt is 30 disks, so 60 on a imac, much more on the future macpro's...
You would only connect all these disks to do a consolidation in Resolve, and make a trimmed version of the raw files,
for further grading and vfx work.

so your shopping list :
on set : 2x 15" retina with 2 or 3 2big thunderbolt with extra disks, small ups if necessary
if you get to much files in one day, add extra machines.

editorial/offline/dailies : 2 or 3x imac 27" high-end with 2x 18TB pegasus, 6 or more 5Big thunderbolt for extra disks,
and onlining, a network switch, a thunderbolt to sas adapter, sas lto-6 drives with 50 or 100 LTO6 cassettes.
 
Tim Jones said:
Ah, So it does require an additional bit of non-free software to make that work. That's what I was curious about.
Click to expand...

Hard drives (Like LTO tapes) do not provide checksums. You would need additional software regardless of media.
 
Todd T said:
Hard drives (Like LTO tapes) do not provide checksums. You would need additional software regardless of media.
Click to expand...

Hi Todd,


That may be correct for LTFS, but maybe the current LTO6 and BRU system have what we have come to know for decades as built-in, data checksum write/read unlike using LTFS. Surely BRU has a checksumming process that is part of the data stream written to the tape. Otherwise, using the LTO/BRU, Drag, Drop, Click functionality would have too little security and repeatability. BRU's been around for decades and I gotta think that that the checksum/reliability functionality has been cemented in their technologies, going back to the '80s, no?

Thank you very much

Fury
 
Todd,

Fury's right in that. If you use BRU to create your archive tapes instead of LTFS, the checksumming is automatic and no further intervention is required on the backup or the restore end of the pipe. Because BRU is applying a 32bit CRC to every 2K of data in the backup, it provides a much more granular level of verification than file-by-file checksums. Additionally, since it's built in, no additional steps are required - drag, drop, click - BRU automatically provides the verify process and creates archives (on tape, disk, cloud) that can be verified and audited at any time on any platform (far more platforms than LTFS, BTW).

One argument that we hear about LTFS over BRU is that LTFS means no vendor lock-in and no additional software is required. Unfortunately, that is not true, first you must find and install the drivers for your vendor's tape device - you can't use HP's drivers with IBM's drives or IBM's drivers with Quantum drives and that sounds like vendor lock-in to me, then you must find the ancillary components - FuSE and ICU as a minimum and that's extra software, even if you don't include the extra cost of Shotput Pro. With BRU, you install one package regardless of your drive manufacturer and you're done. Plus, BRU doesn't muck around in kernel space adding system level drivers that could cause system issues further down the road.

Remember, it's your data and that probably means your livelihood. If you're writing to tape for safety, make sure your tape solution will let you restore the data when the time comes. With BRU, it's the RESTORE that matters.
 
You must log in or register to reply here.
Back
Top