Welcome to our community

Be a part of something great, join today!

  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Archival: Your Current Methods

Nick Pasquariello

Well-known member
Joined
Dec 23, 2010
Messages
1,010
Reaction score
0
Points
0
Location
Providence, RI
I have a colleague that needs to start archiving projects and footage by the end of this year. If she were larger, I'd say, 100%, LTO, through a Cache-A Unit. That's about a $10k starup cost, and then $60-80 per tape, two tapes per 1.3 TB of info, and store one tape on-site, and one off-site somewhere (either via a service or a relative's house).

But she does mainly graphics work, and not shooting. In 6 years she's amassed 11TB of data. Even though that's an upward trend, I don't see her doing more than 3-5 TB a year. She also rarely pulls from stuff that's multiple years old.

So, the cost of archival, at that rate, if you estimate the life of the Cache-A unit to be, say, 5 years (just speculating, based o standard reliability lives of computers as well as the growth rate of LTO), she'd looking at a cost of around $2.3k per year to archive 2, maybe 3 tapes worth of data.

Of course, a simpler LTO drive might do, but for some reason, I always have trouble finding a drive, with cost, listed anywhere. Downside: would still have to figure out where to hold tapes off-site.

I've also looked into Amazon Glacier. It seems like a good fit, but not a great fit. Their storage cost is cheap, and reliable. However, you get charged PER ITEM to withdraw data. And also for the size of the item. You sort of get credited an allotment of data that you can pull back each moth without incurring a charge . . . it's all needlessly complicated. But you can't request to unarchive, say, a folder with all it's contents. It seems that the best course of action would be to zip up every project you archive, and then withdraw that zip folder if you need any item back. Course, if you need just the project file, you're pulling back a ton of data (and downloading a ton of data, and unzipping a ton of data) that you don't need. So perhaps uploading two zips per project, one of everything, and one with just the project files, would work?

Third option I can think of: Solid State Drives. It's just about economical to purchase SSDs, load them up, and there's your archive. I'm not sure what the life is on SSDs, but it's better than traditional HDs, right? No moving parts. The benefit here is that it gets cheaper over time. Might cost $2k this year to get rid of 1-2 TB, but it'll cost half that next year, and half that the year after.

Still have the problem of figuring out where to store them offsite.



So, for any small shops out there: how are you archiving? Surely you can't store everything in an accessible manner all the time. I know how to handle this for a large shop (managed almost half a Petabyte for Hasbro's internal studio before I resigned 2 months ago), but when it comes down to anything in the midrange, a few Terrabytes, nothing seems both safe and economical. Thoughts?
 
I'm in a not altogether disimilar position and have a hot swappable CalDigit unit (ProAV) which I use to make multiple back ups on to bare sata drives, which I then keep stored in proper soft plastic sleeves. It's not ideal, but it'll do until someone makes the most obvious fortune-waiting-to-be-made in technology, and produces a compact affordable LTO with Thunderbolt and a user friendly GUI. I honestly can't believe no-one has done it yet, they would sell by the truck load. Surely we'll see one at NAB this year.
 
In my opinion, for the relatively small amount of data you're talking about, Cache-A is overkill and expensive. For a Mac User: an HP LTO 4 deck is currently less than $2000, the BRU PE license is something like $600 and a simple mini SAS card is a few hundred bucks. You could probably get the whole thing going for slightly north of $3000. Tapes? Look around and you can get them for $25 each (or even less). 11 TB = 14 tapes = they fit in a shoebox and are more reliable than hard drives.

BRU PE is a very user-friendly interface by the way.
The mini-SAS HBA (get one from ATTO) is useful for more than just the LTO.

I was just talking to a friend -- who just discovered that two of his parked drives have gone bad in the couple years since they've been parked...
 
Hard Drives.
I keep a set in the suite I can power up every few weeks, since I understand its the lack of _operating_ for long periods of time that kills them off. Never had a drive go south yet. Which of course is what everyone sez til the day it happens...
Paul's LTO set up is tempting at the price point. $10K LTOs for small shops seems a bit insane. I think if/when the LTO industry realize the huge market in video production archiving - meaning SMALL companies - then maybe we will see some realistic prices.
Meanwhile....

Stu Aull
Alaska
 
I was just talking to a friend -- who just discovered that two of his parked drives have gone bad in the couple years since they've been parked...

Yeah, I power them on every so often. I've done a full migration to larger disks every couple years to keep my footprint the same. I have everything on about 10, 2TB-3TB disks now, two duplicate sets of course. Definitely looking into a LT0 system though. I too am waiting for a plug and play Thunderbolt or USB3 solution. No need for these things to need SAS cards and interfaces anymore.
 
LTO isn't a bad option if she's looking at 11TB+ annually. The Cache-A system is not the answer, mostly due to the buy-in cost. They're an interesting option, essentially a small form factor PC with installed tape drive in a turn-key system. I would probably recommend an LTO-5 tape drive if she were to pursue tape backup. Although it may not be the best choice for her, depending on how technically savvy she is. There is still the cost of the drive itself, interface card and cable, software to run it and media. The interface card and software chosen differ between Mac and PC platforms. On the Mac, it's actually a simple choice -- ATTO host adapter and Tolis Group's BRU software, done. On Windows, it's not so simple. The ATTO host is still highly recommended, but compatibility with other makers' cards makes for more choices. Although that's somewhat minor and the software consideration becomes more daunting.

Lots of people use hard drives. They're convenient and reasonably inexpensive. But hard drives are not designed to sit on shelves as archival media. It's best to spin them up now and then to keep parts moving and distribute lubricants in their system. Optical discs are another option, but have their flaws too -- first of all, they're terribly expensive in terms of $$ per GB. And they're slow. Shelf life is 10 to 15 years, but it seems anything after about 10 is a crap shoot. Hard drives are similar.

No matter which data archival method you choose, the continuous format updating and consolidation is part of any long-term data storage process. Don't think of this as a weakness, but as a strength. It allows for continuous monitoring and reorganizing of data. The down side is it costs time and money. There is no one perfect fix it and forget it solution here.

SSD's... Also expensive. While they don't suffer from mechanical failures as hard drives often do, they can still fail. You are depending on minuscule electrostatic charges to persist your data over time. They can last for years in the NAND memory cell structure, but they can fail... One of the holy grails of FLASH memory development has been the promise of true write-once FLASH memory that will hold its information permanently, forever. Unfortunately, that stuff has never made it out of the lab into real products. At least not yet.


LTO is probably the most reasonably priced solution with the best overall reliability when it comes to larger data volumes. The tapes are cheaper per GB than hard drives, or any other current media. But to make the cost work, you have to archive enough data to offset the buy-in costs of the tape drive, related hardware and software. I haven't run the numbers for some time, but in the past, with LTO-4, the magic number seemed to be about 25TB of data written to tape was the break-even point. Looks like right now I can buy LTO-5 1.6TB (uncompressed capacity) tapes for about $40 each. With BRU-PE software, I get a real-world 1.5TB of stuff on a tape -- mostly R3D and related media.

So to buy into LTO-5 on a Mac, you're looking at:

HP LTO-5 desktop unit: $2900
ATTO H680 PCIe card: $375
BRU-PE Software: $500
2 meter SAS cable: $125

That's $3900, but let's just call it at $4000 USD to get it the LTO-5 unit set up. It will cost you another $800 to write 30TB of data. That would be double back-ups of 15TB, as is probably the best way to start out.

For $4800, you can buy 30 x 3TB HDDs ($160 each, delivered. And cheaper if you buy the "junk" ones) -- that's 85TB of real-world storage. Or 42TB backed up twice. Hmm... So where is the break-even point between HDDs and LTO today? On the bright side, LTO-5 tapes are $0.025 per GB. 3TB HDD's seem to average about $0.054 per GB. Without showing all the math, today's break-even point seems to be 140TB of archival media, or a buy-in cost of $7450, which buys you either 47 x 3TB HDD's for a total of 140TB or it will buy you the $4K LTO kit plus $3450 worth of tapes, which is 87 x 1.6TB or 140TB...

Actually the break-even point is probably a bit less as I figured $40/tape for LTO5 media. I usually find them for around $33/tape delivered, if I cruise around ebay and other sources. Or buy from a bulk vendor in larger lots.

The way I figured it, that's a lot of data. That's 45TB backed up in triplicate...

So, uh, LTO isn't all that attractive to the little guy whose primary concern is affordability. Lots of people in the industry are using LTO because it does have great reliability and is proving to have superior shelf-life vs. hard drive, so many look at it as being worth the extra expense.

In the OP's consideration for their colleague, hard drive is probably the way to go. I would recommend archiving two at least two backup drives, plus keep a third rotating backup of sorts, maybe write off extra special data to optical disc/ blu-ray or other media when available. Cover the bases, but at 11~12 TB per year, meaning triple backups are going to be less than 40TB per year, LTO is still a steep buy-in. And it's difficult to justify over the reasonable life expectancy of the drive and data format of 3 to 5 years. Because in that time hard drive capacities will double again and so will tape, while prices remain consistent to now, or meaning that the cost per GB will drop in half.


One thing to keep in mind is that data backup and archival is not so much a question of technology, it's more about having a strategy and reliable program you can adhere to. A routine to help ensure your data's longevity.
 
LTO isn't a bad option if she's looking at 11TB+ annually. The Cache-A system is not the answer, mostly due to the buy-in cost. They're an interesting option, essentially a small form factor PC with installed tape drive in a turn-key system. I would probably recommend an LTO-5 tape drive if she were to pursue tape backup. Although it may not be the best choice for her, depending on how technically savvy she is. There is still the cost of the drive itself, interface card and cable, software to run it and media. The interface card and software chosen differ between Mac and PC platforms. On the Mac, it's actually a simple choice -- ATTO host adapter and Tolis Group's BRU software, done. On Windows, it's not so simple. The ATTO host is still highly recommended, but compatibility with other makers' cards makes for more choices. Although that's somewhat minor and the software consideration becomes more daunting.

Lots of people use hard drives. They're convenient and reasonably inexpensive. But hard drives are not designed to sit on shelves as archival media. It's best to spin them up now and then to keep parts moving and distribute lubricants in their system. Optical discs are another option, but have their flaws too -- first of all, they're terribly expensive in terms of $$ per GB. And they're slow. Shelf life is 10 to 15 years, but it seems anything after about 10 is a crap shoot. Hard drives are similar.

No matter which data archival method you choose, the continuous format updating and consolidation is part of any long-term data storage process. Don't think of this as a weakness, but as a strength. It allows for continuous monitoring and reorganizing of data. The down side is it costs time and money. There is no one perfect fix it and forget it solution here.

SSD's... Also expensive. While they don't suffer from mechanical failures as hard drives often do, they can still fail. You are depending on minuscule electrostatic charges to persist your data over time. They can last for years in the NAND memory cell structure, but they can fail... One of the holy grails of FLASH memory development has been the promise of true write-once FLASH memory that will hold its information permanently, forever. Unfortunately, that stuff has never made it out of the lab into real products. At least not yet.


LTO is probably the most reasonably priced solution with the best overall reliability when it comes to larger data volumes. The tapes are cheaper per GB than hard drives, or any other current media. But to make the cost work, you have to archive enough data to offset the buy-in costs of the tape drive, related hardware and software. I haven't run the numbers for some time, but in the past, with LTO-4, the magic number seemed to be about 25TB of data written to tape was the break-even point. Looks like right now I can buy LTO-5 1.6TB (uncompressed capacity) tapes for about $40 each. With BRU-PE software, I get a real-world 1.5TB of stuff on a tape -- mostly R3D and related media.

So to buy into LTO-5 on a Mac, you're looking at:

HP LTO-5 desktop unit: $2900
ATTO H680 PCIe card: $375
BRU-PE Software: $500
2 meter SAS cable: $125

That's $3900, but let's just call it at $4000 USD to get it the LTO-5 unit set up. It will cost you another $800 to write 30TB of data. That would be double back-ups of 15TB, as is probably the best way to start out.

For $4800, you can buy 30 x 3TB HDDs ($160 each, delivered. And cheaper if you buy the "junk" ones) -- that's 85TB of real-world storage. Or 42TB backed up twice. Hmm... So where is the break-even point between HDDs and LTO today? On the bright side, LTO-5 tapes are $0.025 per GB. 3TB HDD's seem to average about $0.054 per GB. Without showing all the math, today's break-even point seems to be 140TB of archival media, or a buy-in cost of $7450, which buys you either 47 x 3TB HDD's for a total of 140TB or it will buy you the $4K LTO kit plus $3450 worth of tapes, which is 87 x 1.6TB or 140TB...

Actually the break-even point is probably a bit less as I figured $40/tape for LTO5 media. I usually find them for around $33/tape delivered, if I cruise around ebay and other sources. Or buy from a bulk vendor in larger lots.

The way I figured it, that's a lot of data. That's 45TB backed up in triplicate...

So, uh, LTO isn't all that attractive to the little guy whose primary concern is affordability. Lots of people in the industry are using LTO because it does have great reliability and is proving to have superior shelf-life vs. hard drive, so many look at it as being worth the extra expense.

In the OP's consideration for their colleague, hard drive is probably the way to go. I would recommend archiving two at least two backup drives, plus keep a third rotating backup of sorts, maybe write off extra special data to optical disc/ blu-ray or other media when available. Cover the bases, but at 11~12 TB per year, meaning triple backups are going to be less than 40TB per year, LTO is still a steep buy-in. And it's difficult to justify over the reasonable life expectancy of the drive and data format of 3 to 5 years. Because in that time hard drive capacities will double again and so will tape, while prices remain consistent to now, or meaning that the cost per GB will drop in half.


One thing to keep in mind is that data backup and archival is not so much a question of technology, it's more about having a strategy and reliable program you can adhere to. A routine to help ensure your data's longevity.

LTO 4 will save her $1500. Then LTO becomes intriguingly cost-competitive for her.

Obviously LTO 5 and especially LTO 6 are attractive. But horses for courses; she's running a different track and the significant cost savings would make a difference. Plus all her friends would chip in with a little extra to have her make some legitimate back ups for them too.
 
LTO-4 is about $1200 cheaper for the drive, on average. But it's also two generations old. Tapes average $20, which actually makes them more expensive per GB than LTO-5... As you say, horses for courses. I have both LTO-4 and LTO-5 drives here. Have not taken the LTO-6 plunge just yet as the media is still pricey. I plan to replace the LTO-4 unit with an LTO-6 at some point this year, but it's not real high on my priority list at the moment.
 
A few points of clarification: she has 11TB combined from the past 6 years. Not 11TB per year. I sometimes forget that I have two "modes" in my mind when I talk about data; the consumer/prosumer mode (where a TB is A LOT OF SPACE omg!) and the Professional mode (where I chortle at managing half a petabyte).

She only deals with Red footage when she hires me to shoot, which looks like about twice a year at this point.

She has an iMac, and two MBPs. No tower. So it sounds like she'd also need either an expansion chasis or a Mac Pro. Or a Windows tower, but she hasn't touched Windows in the 4 years I've known her, at least.

Still, no one's suggesting the Cloud solution. Probably not viable until (read: if) US internet speeds catch up to where they should be.

If it's Hard Drives, there's still the question: where to store the second copy? She has two Synology NAS's set up right now that are mirrored, but they're both in the same office. I guess if anything happened to her office, well, she'd be fucked. Might as well treat the archives the same way. >_>
 
Nick,

Since the majority of the data your colleague wants to archive seldom needs to be accessed - and - since the amount of new data each year may only be between 3-5TB, a suggestion:

Purchase enough LTO-5 cartridges to backup 22TB. Technically, you could squeeze that onto 15 (w/o compression), but I would buy (4) boxes of 5 so you can spread the data out.
That gives you enough tape capacity to make (2) copies of everything.

No need to purchase a drive or related gear. Either rent a LTO system or enlist a DIT that has a system.

After you have the two sets - separate locations. One "offsite" location is a safe deposit box at a bank. Fifty bucks a year should get a box that will hold the number of tapes you need to secure - and you can make it a tax deductible expense.
Treat it as part of the business disaster recovery plan - well above sea level, away from natural gas lines, etc..


I don't know that I would use BRU - just LTFS and keep the directory structure as how the data is currently stored.


So, the question will come up: If your colleague doesn't buy a LTO drive to go with the tapes, how practical is this approach?
First part of the answer: The LTO is a backup archive - that is it's only purpose in this case.

Second part of the solution is to copy the data she expects to be accessing throughout the year on a local archiving solution such as the following (that costs only $ 175).
http://www.videoguys.com/Item/BluSt...+two+100GB+BDXL+discs/C48525F4453555C424.aspx

That saves thousands of dollars of up-front expense.
As Jeff pointed out, the cost of storage will continue to decline - to justify the cost of the hardware, you need to use it enough times before the next model/upgrade is available.
Backing up 3-5TB per year will never justify the cost of a full LTO system.

Hence, why backup today to LTO-5 tapes and not LTO-4? LTO-5 tapes will still be readable when LTO-7 drives are commonplace.
That's part of the beauty of the LTO spec - read back two generations, record prior and current. With tape shelf life rated at 30 years (properly stored), it's a smart path to follow.

Best of all, everything important is in at least two places in a format that is recognized as secure and the lowest cost per TB.
Should the local archive Blu discs ever fail, you have recovery options...
 
Agree totally with Brian here. I thought it was 11TB / yr (and growing).

She should just pay someone a bit to back it up for her on LTO5 or LTO6.

Cheap/easy/done.

And re: LTO4 vs. LTO5 vs LTO6. I have LTO4, skipped LTO5 and will be getting LTO6 along the same reasoning that Jeff mentioned.
 
Yeah, for some reason, the idea of borrowing or renting a drive didn't occur to me until Brian said it. I think because I don't personally know anyone with one (well, there are a few people that I know that may [or even probably] have one, but seeing as how I don't know them well enough to know that, I'm hesitant to call on them to borrow the drive they may or may not have. But a rental fee would probably work out.)

I am well versed in the merits and intricacies of LTO tapes. Right now I would make sure she uses LTO5, and then migrate to LTO7 in about 3-4 years. But, having worked for Hasbro with a Cache-A unit (and a crappy, finicky, unreliable LTO4 drive before that) that was purchased for me to use, I never really went through all the ins and outs of the various methods to get stuff on to LTO.

Which basically all means that I completely understand everything everyone has said, and want to thank each of you for your input.
 
I use Retrospect Server to initially backup project folders and R3Ds from my workstations to a backup server with 16TB (14TB really as it's RAID 5) of Western Digital RE drive based RAID. I can expand that to 32 TB later. The array is always checking itself for issues, scanning the drives.

If I keep the backup sets below 3-4 TB in size, I can quickly back them up from the backup server to external hard drives that I insert into a top loaded Icy Dock MB981U3S-1S Drive Dock USB3 eSata. I'm sure drives will grow beyond 4TB soon.

I rotate these to my safety deposit box.

BTW - I also image all my computers using Acronis, except for the Mac which uses Carbon Copy.
 
Back
Top