Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

looking for ideas for BACK UP WORKFLOW

Miguel "Macgregor" De Olaso

Miguel "Macgregor" De Olaso

Well-known member
Joined
Dec 29, 2006
Messages
796
Reaction score
0
Points
16
Location
Los Angeles
Hard drives die. That's a constant in this universe. :w00t:

Even back up drives die too.

I'm thinking about how to simplify my workflow for backing up all my data. And I´m looking for ideas on how to make it as simply as possible.

I work with video and photos (mostly in windows, although I use mac hardware). I have many projects. And a lot of footage.

These days with fast internet and utilities like dropbox, backing up project files has become really easy and stress free. Simply saving the the project file in its corresponding folder inside the dropbox folder, solves my problems. The project file is uploaded into a remote server, I have access anywhere and I can even go back to a file in case i delete it by mistake.

But online storage does not work with big files as all we know.
Until now my only option was to buy external drives and manually make back ups. And that's fine once I've closed a project. Project is closed --> it goes to external drives.

My problem comes when I am working in maybe 10 projects at the same time. I can´t close them and store them in a hard drive.
So what I´m doing is having a back up drive with the footage, and work with a copy of the footage in an internal/external drive, saving the project files in dropbox.

This is fine with 1 project, but with 10 projects, with VFXs, edits, onlines, offlines, soundtrack, audio mix, texts, pdfs... it becomes a mess.

I guess I could invest in some hardware and get an external raid or something. But have in mind that I fill up drives really quickly, and storing all my projects together makes it more risky in case that external raid gets lost or damaged.
That´s a second concern, physical storage--> too dangerous. If there´s a fire or someone steals the machine, good bye. That´s why I like online store so much.

If there was just something as simple to use as Time machine for windows, it would also help very much.

So... I´m open to ideas and hear what you guys do. :001_smile:
 
You should have at least a RAID1 array for all your hard drive sets in computer. Windows 7 has a powerful Backup and Restore Center, witch which you can backup to a rigid, fireproof external hard drive, such as the ioSafe Solo Pro. Maybe have one backup of these off location. Of course, if you are still paranoid, you must look into cloud backups. Amazon S3 is a decent option, though I have not tried it. Jungle Disk is an excellent frontend for S3 - it acts like a regular hard drive - but if you are going with Jungle Disk, Rackspace is a better hosting option than S3. It is $0.15 per GB/month with free transfers. This article may help you as well. Of course, cloud storage is a relatively new concept which will grow into a primary form of storage with time and faster internet. Google is pushing cloud computing heavily with Chrome OS and Windows 8 is rumoured to have powerful cloud storage features.

The Windows equivalent to Time Machine is "Shadow Copies". Just right click on any folder (on a drive with recovery enabled) and move to the Previous Versions tab. It is a little known fact that Shadow Copies predates Time Machine. It does exactly the same thing as Time Machine although the Time Machine interface is a lot cooler. Windows 8 will have a full fledged "History Vault" GUI for the Previous Versions tab. Backup and Restore Center covers many of the other features of Time Machine, and more. Of course, neither Shadow Copies nor Time Machine are of any use when it comes to a hard drive failure. In that case, you will need to try out dedicated recovery software (Recuva is the simplest) or if the failure is hardware oriented, look for professional recovery solutions.
 
Get a Drobo. If a one of the drives fail, you replace it, and you lose nothing. (RAID harddrive systems).
 
For a finished full feature 90 minutes to be stored you need in 2K you need 1,08Tbyte that’s 19K$ for just 10 years storage. The same in 4K is about four times... that's about 77K$...!!! that's four times the cost of a 4K Filmout... In the details of your contract with the cloud storage provider you should check the liability limits that they assuming and if you are secured for the years to come. That is if the provider is taking responsibility to gracefully transfer your data in a possible sale of his company to a third party.

The Academy of Motion Picture in US has done a very noticeable work that has discover the nasty problem of long term "store and ignore" of digital essence.

http://www.oscars.org/science-technology/council/projects/digitaldilemma/

It seems that nothing can beat the 40 years mark today. The reasons are many and not only related to data retention of the medium used but instead i.e. if there will be a driver for this LTO tape that you used 40 years ago or if the LTO device will work (mechanically) after a storage of 40 years or if the interface will exist then…

Oddly but, It seems that the best medium and the cheapest would be to do a 4K Film Recording which with the today’s film technology we can expect something like 100+ years in store and ignore state.

That’s why we promote the Color grading always to happen with a filmout in mind in order to be possible the long term secure storage of the digital works of art.

PS. Bank records are not expected to be stored for as much time as a work of art, which the value of it can surpass the century easily, when bank data get destroyed in a regular basis for legal purposes. If you want your movie to be able to be used as an intellectual asset from your kids then I thing you can't compare the two...
 
I've seen this discussed many other times here and there are no NEW solutions. Tape backup for archiving is great. I'm on the other side of the fence. I have about 30TB of ONLINE storage i work with almost regularly. I EXCLUSIVELY use hard drives for all of my project backup and online storage. I ALWAYS want access to my files online at ANY MOMENT. i use E-sata 5 bay (Hot Swappable) drives. I just recently changed out all of my drives to 3TB 7200rpm 64mb cache drives that cost $185usd each. That's a lot of bang for the buck. Personally i NEVER raid EACH DRIVE. My concern for drives is more when if i DROP it or if i have some sort of electrical problem and drives get fried. I keep a backup of EACH and every drive in my safe---they don't sit for years at a time--instead, when new and larger drives come out....i rotate them and do a full backup so i have them locally in my safe. LTO for ARCHIVING my data from past 20 years as well so if the house is vaporized and everything in it, i can go to the company that vaults my LTO data and get it all back. Going with 5 bay external hot swappable drives (at the moment) you can run up to 20 drives on RAID 0 for up to 60 TB of storage. With Thunderbolt coming out and as soon as the boards are made, you'll be able to run way more than that....and with E-sata 3 coming....it should make for incredible speed.

I don't think there's a magic bullet out there yet and not for a while. for online storage there's just nothing to beat the cost of hard drives at moment. I know the replies to this will be that they fail etc...but i've been doing online storage now for over 8 years and not ONE failure with regular rotation of drives each year. In fact not even a hint. NOW that said---think about STOCK FOOTAGE sales agencies...where is all of their online storage? Massive array's of raided hard drives.......Been working for them and if there was another solution, i'm sure they would have moved to it or we'd here about it as soon as it's available.
 
There is no magic bullet, that's for sure.

Best is redundant hard drives / RAIDs for online or near-line storage combined with LTO tape for long-term archival. Backup security lies entirely with redundancy in most cases. Hard drives will eventually fail. They don't like to be stored idly on a shelf, either. Tapes can fail too, so can tape drives. Best to have multiple copies of your data and if you have it stored on both HDD and tape in more than one location, you should be good. Just don't be that guy who puts the only copy of your digital masterpiece onto a single HDD...

The problem with online storage is that sooner or later, you might just have too much physical storage to work with as an online resource. Of course, that all depends on how much data you create. 30TB of online storage isn't all that much these days. It's a lot for one person, but for a small production boutique, it's not. For myself, I have about 30TB online myself, 16TB is actually redundant between two 8TB working arrays. That only accounts for two workstations. I have a large and ever-growing collection of LTO-4 (and now LTO-5) tapes.

The one thing that most people don't factor into data archival is continuous migration and updating. Storage formats are continuously evolving, capacities are ever increasing. Whenever I adopt a new archival method, I usually make new backups of all my data on the new type of media. It helps keep my data collection current, provides a new copy of the data that fits into less shelf space, adds redundancy, etc.. I have not done this between LTO-4 and LTO-5 though. Mostly because a lot of my current data overlaps both formats anyway and they're essentially interchangeable. I prefer using the LTO-5 tapes because they hold twice as much in the same physical space. But they're still more than twice the price of LTO-4, which are about the best deal going in terms of $/GB storage -- ~750GB/tape at $20 each.

Anyway, I think the best solution is to have more than one copy of all your data in more than one location. So yeah, if your place burns to the ground or is vandalized, etc.. you don't lose everything. These days, with all the identity theft, fraud, information trafficking, etc.. data tapes and hard drives are primary targets for thieves, in addition to smart phones, notebook computers, etc..
 
I just want to stress the point of Jeff: "Data immigration is a continual work..." don't forget that every couple of years you have to move things to the next technology.

Can someone calculate the cost of that process in a period of 60 years?

Film can be "stored and ignored" for more than 60 years... now do the comparison... and you will understand how big pain is the data immigration...
 
Evangelos Achillopoulos said:
I just want to stress the point of Jeff: "Data immigration is a continual work..." don't forget that every couple of years you have to move things to the next technology.

Can someone calculate the cost of that process in a period of 60 years?

Film can be "stored and ignored" for more than 60 years... now do the comparison... and you will understand how big pain is the data immigration...
Click to expand...

Yes, but film can die as well....when the A/C goes out and it gets 125f.....all things die or can die given the right circumstances. Not that it's significant, but what is the cost of storing film in an insured vault?
 
LTO after 30 years will not even exist a drive to read it... not even drivers to mount it or a port to connect... I don't want to mention the software...

Film in a normal house condition sealed in the can, can be stored for many decades... history has prove that... also new stocks are very stable... than the ones 20 years ago...

The difference from data is that film images can be seen by naked eye... so they don't need special software to decrypt or special hardware to read... film is like marbles of Acropolis for motion picture...

Of coarse all things die if you let them in the sun to burn or in water or in not typical human environment of a western house...

The question is can something like film die in normal conditions just because the company that was doing the drives has bankrupt after 30 years? or because the software engineer that did the compression algorithm can't be found in 40 years?

How many point of possible failure a data archive strategy can have and how much effort needed to cover them? How that results to cost?

A very critical factor is the value of the work of art needed to be kept... if its a movie that has being spend a million to be done then 15K to keep it on 4K film is justified... If its just few reportage that are worthless... then isn't such a big deal... so you can try the archive workflows and play with your luck...
 
You need to make sure that your backup can handle all of the "finger" problems:

  • Fat fingers (typo)
  • Sticky fingers (theft)
  • Butter fingers (coffee spill)
  • Middle fingers (malware)
I think the most common problem is fat fingers. Say your computer goes down due to a bad hard drive, so you get a new one and you're installing the OS and... OOPS, you just installed the OS to the primary backup drive, blowing away your backups. Now you lost your primary *and* your backup. Now what? That's why you need (and have) a secondary backup.

Remember that RAID doesn't protect you against fat fingers, sticky fingers, etc. whereas two separate arrays (preferably one off site) does.

Personally, what I do is have three 6TB raid arrays that I backup to. Two are on-site, but only one is turned on at a time (to help protect against fat fingers, mostly) but every day they rotate. Once a week, I rotate to the offsite location.
 
Evangelos Achillopoulos said:
LTO after 30 years will not even exist a drive to read it... not even drivers to mount it or a port to connect... I don't want to mention the software...

Film in a normal house condition sealed in the can, can be stored for many decades... history has prove that... also new stocks are very stable... than the ones 20 years ago...

The difference from data is that film images can be seen by naked eye... so they don't need special software to decrypt or special hardware to read... film is like marbles of Acropolis for motion picture...

Of coarse all things die if you let them in the sun to burn or in water or in not typical human environment of a western house...

The question is can something like film die in normal conditions just because the company that was doing the drives has bankrupt after 30 years? or because the software engineer that did the compression algorithm can't be found in 40 years?

How many point of possible failure a data archive strategy can have and how much effort needed to cover them? How that results to cost?

A very critical factor is the value of the work of art needed to be kept... if its a movie that has being spend a million to be done then 15K to keep it on 4K film is justified... If its just few reportage that are worthless... then isn't such a big deal... so you can try the archive workflows and play with your luck...
Click to expand...

Point WELL TAKEN Evangelos.
....and what does your well made point have to do with this discussion on a digital medium? Film is great, but a great majority here are not shooting film, and are interested in digital media gathering & storage techniques in use today or in the future. Unless you are suggesting to go to film for archival??? But the point of the thread i believe is backing up a digital workflow; and then long term storage.
I'm very pleased that film is being shot and love film, but it's dead in my world and will NEVER come back....nat. hist. doc realm and wildlife feature doc films....so, i'd think the vast majority of users here are looking for digital solutions. but you have stated well your case for film and the downside of film is vast as well....fail to mention lugging film stock all over the world on a low or small budget shoot; then handling of it when leaving a location and all the in betweens until it gets developed.
 
Interesting how the archivability of film always seems to rear its head in digital archival discussions. While it does derail the conversation, it should be noted that film has plenty of flaws too. Some of the foremost being that it degrades over time, no matter how well kept the film is. There is a finite lifespan to it and any copies made introduce generational loss. Most studios with film archives are regularly making high quality digital masters of their film libraries, often doing it over again as new technology arises. Look at it this way, 6K scans at 16bpc can preserve sub-structure details of the original film stock and can be considered just as good, even superior in some ways, as a master print.

For digital archives, there's always that argument of "in xxx years, no means to read your current media will exist!"... Er, OK. No one knows that for sure, besides we have already covered that with the continuous migration and consolidation of our digital libraries.

Preserving our information is a a continuous process, no matter if it's digital or film or whatever. In my experience, the costs of preserving digital archives are comparable, even negligible in contrast to maintaining an environmentally controlled facility for housing film. And one thing that digital offers, which film can not is that repeated copies are not degraded from the original.

Not everything can be stored on film anyway...
 
Jeff Kilgroe said:
For digital archives, there's always that argument of "in xxx years, no means to read your current media will exist!"...
Click to expand...

Let's make the speculation that in less than 10 years no more feature film will be shot on stock. Osolecence...

Tell wich commercial lab will keep a film scanning device for 60+ years? The knowhow will also dissapear.

Thats why Cinemathèques should also invest in the technology to keep scanning available.

I'm less afraid of archiving a digital master than a neg.

Pat
 
i don't think it's necessarily a bad idea to archive a filmout of a feature film, but that's an entirely different topic of conversation. i do not recommend doing a filmout for simple backup storage... what are you going to do, go get it scanned so that you can cut it up and recreate your original files? uh, yeah... i'd rather just throw it all in the fireplace at the start.

the problem is that there isn't really a great, reliable, cost-effective, long term storage and backup solution. johnny doesn't make a bad point, it seems to work for him, though there are some inherent risks as always. there's also the services online to look at. it all comes down to your needs and how much money you can or are willing to spend. ease of access may be a big factor, portability may be a big factor, cost is always a factor, as is reliability. though terribly imperfect, i recommend tape right now, though i hope we'll see some good solutions with non-moving parts storage at a decent price in the years to come. for stuff that you need access to, i'd additionally recommend redundant raids on top of that.

if anybody knows a better option than that, please share it, but that seems like the consensus.
 
My point goes for feature movies archival ONLY.

For the next 5 to 8 years we will still have the opportunity to filmout... after that I don't know...

But until then to Color Grade for a filmout target and get the film character embedded to your digital film by using a film LUT and have the ability to deliver that one or two film copies for that festival that still accepts film while doing all the major distribution in digital, there are conversion LUT's that delivering great results when used to convert film grade to REC709 or DCP... Off coarse keep also disks if you like to do so...

My bet though is that your grand children they will find the way to scan and revive images from their grand father film and keep a part of our history alive...

I repeat, I'm talking about real movies not everyday thinks... and yes film is a long term backup medium, so its relevant... I would say extra long term....

I am also a digital guy... but some things we must still be able to point them, when they have something to offer to the Art form we all working for.

Please guys read the Academy study about digital archiving and you will realize that there is a real threat our grand children not to be able to see what we shoot today, after 90 years... Its not helpful to put the head in the sand or to isolate voices that pointing out things you can't see...

PS. SSD's are not an option because of the electron immigration phenomenon or gate leakage I thing its called... read the study...
 
You must log in or register to reply here.
Back
Top