Archive Wisdom

[Floris Liesker]

oooopsss...Did I miscalculate in my post earlier in this thread. I'll completely rewrite it.

So, at 24 fps, REDCODE RAW produces 28 MB/sec, which should be 98.4375 GB/hour.
So 1 hour is approx. 100 GB, a 500 GB drive will hold about 5 hours...
Still this isn't the same figure that Stuart is giving. What am I doing wrong this time?

With prices Ive found of a around $180 for a 500 GB drive, one hour of storage will cost about $36,00 and twice that for a RAID configuration.
That's OK for a project that is still running, but for permanent storage I would hope for something a little cheaper...

 

[_BK]

Im very curious about this topic as well. I recently got burned by trying to archive data to 3.5" drives... after only 3 years of sitting 2 out of 4 drives seized up on me. Cost me a fortune to recover the data. Right now Im leaning towards LTO 2...

 

[David Nardini]

LTO 2 is still the best value/security balance option out there. Yes a drive and SCSI card will cost you $2K but $40 for 200GB uncompressed tapes that write as fast as a normal Harddrive is hard to beat. Plus I think all LTOs are backwards compatible, so most of the higher end post facilities used to working with Viper/F950 footage will be able to easily handle it.

Hi Evin, thanks for the pointer. Could you let us (me) know the better manufactures for LTO devices (those with a good track record in this technology) ?

Like you all, I'm very interested in this topic; a critical subject I believe.

The RED (and the new digicams) can generate vasts amount of data, and storing this data long term is not a trivial task. Technology moves at such a rate, it's hard to decide the strategy for the medium to long terms !

My experience ...
- disk is nice : but backup strategy can be a real mare (ie : off side copies)
- burning based technologies (DVD, Blue-Ray, etc) : always been concerned about the long terms stability of the medium (have had a number of non-readable disks, even if they verified fine ok at write time)
- tape : not used it personally, but have not heard 100% success stories

David Nardini.

 

[Milan Nikolic]

I am going to use MACOLA 2T (with infiniband interface) HD array for storage and editing. Edited material in thematic chunks of 20 minutes will go to Blue Ray disks multi copies for other place storage.

 

[Sanjin Jukic]

Macola Infiniband interface 4TB and 2TB RAID link
http://www.mac.co.yu/ponuda/macola/macola.html

 

[David Limpus]

macola infiniband drive type, anyone translate for the english only impaired

http://www.infortrend.com/main/2_product/a16f-g2430.asp is looking ok if they do a 16 bay fc-sas will be better

 

[Jochen Schmidt-Hambrock]

2 years ago at the AES conference in SF: A panel on archiving media with a couple of representatives from Exabyte, Sony, etc..

Each one had his 5 minutes where he "proved" that his medium has the best reliability, cost and longevity. (You know those presentations: See a chart - and of course somebody will read it to you)

The last guy (a Frenchman) was unattached to any company. His first words were that until now we all only heard BS.

That got everyones attention.

His point was that only backing up to multiple harddrives makes sense.
And have everything online all the time.

Made sense to me. Over a year I transfered my whole stuff to disk. Threw away tons of Umatics, VHSs, CDs, DVD-ROMs, etc...

Lots of free space now. Never looked back.

And another thing: It took me 4 hours to get the files off my DVD-Roms (you know, the ones that came in a cartridge)
Dig out the Quadra 9500, a monitor with "sync on green", connect the SCSI (where´s a terminator for gods sake!), install some long forgotten OS 8 driver.....)

Jochen

 

[Clark Dunbar]

Off-line storage and archiving.

A major workflow and ASSET quesion. If your shooting a few hours a week, at any 4K to HD modes - your looking at a huge amount of data that needs to be backed up (remember the only original is the file you just transferred to your editing system/sever - so once you've cleared your RED-drive or flash NOTHING remains except the digital copy - no film, or tape to throw in the vault).

So for backup to the project, re-confirguring the final out-put, basic archiving and re-assest utilization (read stock footage) needs to be covered.

You can rack up a huge budget item with offline storage and archiving in your annual overhead on our jump into digital only acquistion. Copies can be made on to harddrives at about $1/gig or you can look into all the tape systems for data - ie: DLT and the like which prices out at about $0.25/gig, and then you have the DVD and BluRay options which are even slower and smaller storage per unit. With the RED-drives being 320gigs, you can start to see that this becomes a major item in your overhead

So, do you look to buying a lot of hardrives and storing them in your media vault (closet or storage room) or do you invest in a tape system - which is slower but more cost efficient? And as we should all consider 2 copies of everything for that on-site copy and an the extra protection of an off-site copy.

Your 'footage' is you future - both as re-purposing the original program to new media but also as stock footage and yet unforeseen new markets.

So - the storage and archiving of all digital acquired media becomes an budgetary item but you also have to consider the time and storage space for the media, just as you would for all the film and/or tape originals that we had before - with the caveat that this must be the first step in your workflow - BACKUP. Both when on the road and the office/studio/edit suite.

So far in reading the forum - I see this as a over-looked and an important part of everyone's business model and an item that needs to be budgeted in to our new workflows.

Long post - but always looking to make sure that my assets aren't lost, so as of this note - we're using a SDLT tape system at 300gigs a tape for everything coming in AND a another copy of the final edit...this adds about $1k a month to our overhead and with 4K (even using RED-Code) we are planning on doubling this.

Even with media and storage prices coming down - this will be an item that needs major consideration. And .. I haven't even gotten into the logging of all this off-line/archived data.

No answers just concerns that we all need to deal with

 

[Mark L. Pederson]

Right now, it appears that LTO3 is best bang for the buck - $51 per tape - 400 GIGs - and yes, BACKUP is KEY to shooting tape/film free -

 

[David Limpus]

http://www.ikegami.com/IKEGAMI_NAB_2007_FINAL.html
Looks like Ikegami doing the holographic archiving HDS-300R external holographic drive. Hope it brings the InPhase Technologies Tapestry HDS-300R prices down.

 

[Bruce Allen]

Hmm, you guys have some pretty weird figures for cost/gb for drives... Seagate 750s are $270 at newegg, making it 36c per gb, not $1. And that's not even the best bang for your buck. I have been looking out for deals and buying drives at roughly the 30c / gb range for at least a year now. So, I personally would vote for hard drives too.

The last piece of data I lost was when my 2 GB drive died. That was a while ago Since then, I simply made sure to have several backup copies of everything spread across drives. I use a little shareware program called SyncBack, which just synchronizes files onto my external drives every night. And then every now and then I take my external drives and put them in a different physical location and connect a different batch up to my PCs for nightly backup.

RAID-5 is not enough - sometimes it even gives you a false sense of security. Just make sure you have your data in three places at the same time - on your online drive, on a spare drive, and preferably on a third drive that is in a different physical location.

Since areal density is very likely to increase at great guns for the next few years, it makes sense to simply buy drives as you go - adopt a JIT assembly line philosophy since the price is dropping every month.

The only time I or my friends have ever lost data is when they have it only stored in one place. The worst idea is to trust only one (usually expensive) backup system "investment". For example, I've seen 2 Xserve RAIDs lose WHOLE TERABYTE RAID-5 SHARES overnight at places I have worked as a VFX artist, and have another horror story from a close friend. One of those systems had a Retrospect tape backup system going, but it took them several days to restore, and they still lost things because a) an old tape had died and nobody had figured out how to get Retrospect to re-backup the data that had been on that tape and b) the crash happened on the weekend, when the tape backup system ran out of tapes. Luckily, however, some people had made a copy of project files to his local computer, and another guy had made a backup copy of some of the stock footage that got lost with the tape, so everyone scraped by.

Tape might be a good option, but I would personally feel a bit happier with multiple copies of my project stored on different drives (getting different manufacturers is good too, as then the chance of both having the same defect is minimal). Every 2 years or so, I'll buy bigger drives and transfer everything to those, making another backup in the process. The nice way is then you are actually verifying your data.

Bruce Allen
www.boacinema.com

 

[Jason Francois]

I'm going to back-up to hand-inscribed tablets of stone....at least until I can figure out a better and cheaper method.

Hey, it worked for Moses and that story has lasted 2000+ years.

 

[Jeff Kilgroe]

Bruce,

Pricing out hard drive solutions isn't as cut and dry as pricing tape, IMO. Unless you're just talking about copying data to a drive and placing it on a shelf (bad idea, IMO). As you said yourself, RAID-5 is not enough for critical storage. Taking that into account, I would be compelled to built RAID-7 arrays or store duplicate RAID-5 volumes. Either way, when you start building RAID-1/5/6/7 volumes, your $/GB changes.

Tape seems a lot more economical in this respect... Even figuring making duplicate copies. The beauty of tape is that as your archival quantities and number of tapes increase, the savings add up more and more vs. hard drives. One of the huge factors in tape archival costs is the tape drive itself. Because just figuring tapes themselves, LTO-3 is less than $0.12/GB when you start buying multi-packs or grabbing overstocks off ebay.

 

[Jeff Kilgroe]

Hey, it worked for Moses and that story has lasted 2000+ years.

Yeah, but how do we know the data didn't get corrupted along the way?

Ooops, I didn't want to turn this into a religious debate... Afterall, Moses was highly regarded by the Egyptians too, or at least by most accounts of him that have been found.

 

[Clark Dunbar]

the really scarcy reality - is that no matter what backup/archive solution you put into place now - in 5-10 years it'll be obsolete and everything on your current media will need to be backup'd onto the "next" generation. Just think floppies, syquest disk, optical disks......

 

[Chris Kenny]

the really scarcy reality - is that no matter what backup/archive solution you put into place now - in 5-10 years it'll be obsolete and everything on your current media will need to be backup'd onto the "next" generation. Just think floppies, syquest disk, optical disks......

Yes, but all the data you generate this year will probably fit on one holographic disc in 10 years. Or into a small corner of your online storage.

And online storage probably is the way to go. You're far better off keeping multiple copies of your data on actively monitored online storage than writing it out to tapes or discs and sticking them in a closet, with nobody paying attention as they slowly degrade over the years, or as technology moves on and it becomes difficult to find a drive that can read the media.

Between the price/GB of hard drives now, the availability of cheap external eSATA cases with port multiplication, and the software RAID features built into modern operating systems (and particularly with ZFS in Leopard), it doesn't have to be very expensive to run such a system. Could be done for around $1/GB of data (taking into account that you have two copies of everything).

 

[JD Holloway]

I have no intention of maintaining ALL of my shot material on tape or drive for all time. I will need to dump data fast to free up drives. Some of my projects shoot 300 min/day sustained for week/s. EDLs will be vital.

I believe tape drive decks can be rented/borrowed in Toronto, so only occasional back-up would be necessary for safety when working on projects. Therefore only the per/tape cost concerns me. Maybe RED user locals can coordinate this service and centralise it.

My concern is with 4k RGB Finished Masters. What is the best way to re-compress them? Any "lossless" methods? A re-mosaic back to REDCODE (jokes)? Of course there is a image loss cost... I'm guessing its close to 2 TB per feature film (assuming 90-100 min) at 350 MB/sec.

Those are big numbers...

 

[Jeff Kilgroe]

the really scarcy reality - is that no matter what backup/archive solution you put into place now - in 5-10 years it'll be obsolete and everything on your current media will need to be backup'd onto the "next" generation. Just think floppies, syquest disk, optical disks......

Of course... But storage continues to get bigger, faster and cheaper. No big deal to those of us who've already done the large scale backup thing on some level. That's why it's silly to sit around and debate which archival format has the best shelf-life or if DVD-R will really last the 100 years, oh wait.. 30 years, they claim. In the next 5 years or so I'll be wanting to re-sort and organize and re-archive my current stuff. If for no other reason becasue I don't want shelves full of boxes full of tapes.

 

[Jeff Kilgroe]

I believe tape drive decks can be rented/borrowed in Toronto, so only occasional back-up would be necessary for safety when working on projects. Therefore only the per/tape cost concerns me. Maybe RED user locals can coordinate this service and centralise it.

Yep. I'm hoping to do something like that... I'm going to be buying into some form of archival system. And I'll definitely offer its use to fellow REDusers if I think it would be practical or beneficial. But I suppose we'll have to see what system I ultimately go with.

My concern is with 4k RGB Finished Masters. What is the best way to re-compress them? Any "lossless" methods? A re-mosaic back to REDCODE (jokes)? Of course there is a image loss cost... I'm guessing its close to 2 TB per feature film (assuming 90-100 min) at 350 MB/sec.

I wouldn't recompress anything. Archive the 4K masters as-is, even if they're 5TB in size. For 5TB of archival you're looking at $650 for LTO-3 tape (13 tapes no redundancy) or 5TB of HDD space (~$1750 for straight drives, no redundancy). Choosing either and making a couple duplicates to ensure it survives for a while is a lot cheaper than creating a film print for archival. Or drastically shrink the storage requirements by remastering to 2K and archiving that, hoping it will be good enough down the road.

 

[Chris Kenny]

My concern is with 4k RGB Finished Masters. What is the best way to re-compress them? Any "lossless" methods? A re-mosaic back to REDCODE (jokes)? Of course there is a image loss cost... I'm guessing its close to 2 TB per feature film (assuming 90-100 min) at 350 MB/sec.

Remember, there's an RGB version of REDCODE as well. You can't shoot it at 4K in-camera, but I'd be surprised if you couldn't encode it at 4K from QuickTime. I'm sure it would have a higher data rate than the RAW, but I'd think it would be pretty manageable.