Archiving Data, permanently.

[Wayne Morellini]

I was thinking of finding an archival forum to get a pretty good answer on this, but as I am over here already about my media library, and backing up data long-term is similar to backing up footage, I thought I would as about permanent archival solutions for my data. Dies anybody have any advice or know of any forums that do?

The problems I face is that hard disks, flash, data DVD, even tape, deteriorates, or can be adversely affected. The other problems is management and verification. Most systems that verify that data has been written correctly use the hash value of a file or file section and do not take proper account if it is in the buffer/cache rather than the actual written file on disk. This can lead to a number of errors that crepe in as things are rewritten. The other part is management, to keep and only update the changed files and sections of files, or record only changes in new file structure copy, and locating changes or files with same content (searches), most recent timeline etc. I presently get problems distinguishing ages of files. You can get into a lot of trouble just copying complex directory structures that run out of directory path. Does anybody know of any good systems for this?

On which media system to use to backup permanently, I want something impeccably reliable that can outlast me without too much worry (special storage handling)?

Thanks in advance.

 

[PatrickW]

There is no true permanent storage for digital data. You need a good backup strategy, not a magic bullet.

Even if device X offered 100 or 1000 year secured storage, what is to say that it would be in a readable format or even connectable to future computer systems.

 

[Matt Gottshalk]

I use CatDV and A Cache-A LTO5.

Works well for my workflow.

 

[Marc Wielage]

The Academy of Motion Picture Arts & Sciences hasn't figured it out, either. Here's a recent report:

The Digital Dilemma
Strategic Issues in Archiving and Accessing Digital Motion Pictures

It raises many troubling questions. I don't think there's a real, 100% solid solution yet.

 

[Sergei Franklin]

LTO tape should last at least 10 years. If you want to archive for over 100 years you need to make black and white separation films http://www.fujifilm.com/products/motion_picture/lineup/eterna_rds/
http://motion.kodak.com/motion/Products/Lab_And_Post_Production/Archival_Films/2238.htm
If InPhase's holographic disk ever comes back it claimed over 50 years of storage life.

 

[Jeff Kilgroe]

LTO is the best option for now. At this time, digital archives need a babysitter or someone to refresh them every now and then, consolidate and transfer to new media types over time. As pointed out earlier in the thread, backups and archival are a strategy, not necessarily a single technology.

That said, this is changing, but we're still not there. I'm actually involved in some research to this effect, some pretty radical stuff actually.

 

[Bob Gundu]

hmmm... Kryptonite and crystals I bet...

LTO is the best option for now. At this time, digital archives need a babysitter or someone to refresh them every now and then, consolidate and transfer to new media types over time. As pointed out earlier in the thread, backups and archival are a strategy, not necessarily a single technology.

That said, this is changing, but we're still not there. I'm actually involved in some research to this effect, some pretty radical stuff actually.

 

[Eyimofe Prest]

on the note of archiving and preservation, Kodak announces 'asset protection'

 

[Gavin Greenwalt]

I would warn people though that you can still get a good film scan today... but what are the odds you're going to find a decent scanning service at not-outrageous-rates in 10 years when almost nobody is shooting film?

There are a lot of film services still around today so finding someone with a davinci or such is easy. But without commercial customers--your archives are going to be 'human viewable' and exceedingly expensive to 'restore' once that starts drying up.

And for the price of a backup/restore even today you could make a dozen LTO copies. I would rather have multiple backups than one vulnerable film reel.

 

[Marc Wielage]

I would warn people though that you can still get a good film scan today... but what are the odds you're going to find a decent scanning service at not-outrageous-rates in 10 years when almost nobody is shooting film?

There are at least 100,000 hours of significant motion pictures and film TV shows that have not yet been scanned. We have a long, long way to go before all of these have been digitally archived and preserved. One hopes they'll finish before the emulsion crumbles to dust.

And for the price of a backup/restore even today you could make a dozen LTO copies. I would rather have multiple backups than one vulnerable film reel.

I have personally scanned nitrate films (some Laurel & Hardy shorts, circa 1933-34) that was actually in fine condition, more than 50 years after it was shot, and it was fine. Are you absolutely certain that an R3D file today will be recoverable, decodable, and viewable in 2062? I'm not so sure. (And I would say that for all digital file formats.)

I have had at least six major, A-level studio features affected by LTO2 crashes and data losses in the last ten years. Granted, most of them have been fine. But just one of those resulted in more than $500,000' worth of work having to be redone. Don't forget the infamous Toy Story 2 data loss story...

 

[Wayne Morellini]

There is no true permanent storage for digital data. You need a good backup strategy, not a magic bullet.

Even if device X offered 100 or 1000 year secured storage, what is to say that it would be in a readable format or even connectable to future computer systems.

Yes, there was 100 and 1000 years solutions on CD or DVD, what happened to those (also too limited capacity). How does high capacity Bluray (100GB+) go these days? All I need is a hundred years or so of non wishful thinking reliability. The key to a good backup strategy is reliability, start with good make multiple copies and anything that goes wrong and you haven't kept refreshing copies, it is more likely they will be OK. With all the Parkinson and autistic symptoms I go through, refreshing is not a guaranteed option.

What I said about validating bit by bit a copied file is important because common strategies don't, leading to possible corruption as you refresh, which can be an issue the more often you do it.

 

[Wayne Morellini]

I use CatDV and A Cache-A LTO5.

Works well for my workflow.

Thanks Matt, Catdv looks useful for footage.

 

[Wayne Morellini]

LTO is the best option for now. At this time, digital archives need a babysitter or someone to refresh them every now and then, consolidate and transfer to new media types over time. As pointed out earlier in the thread, backups and archival are a strategy, not necessarily a single technology.

That said, this is changing, but we're still not there. I'm actually involved in some research to this effect, some pretty radical stuff actually.

Jeff, you are not the only one. I have not been keeping up, even cursory glance, for many years. I have some ideas too, ironically even converting data to back and white film that can be scanned. The trick is to record the format with the data, which I have come up with a scheme for that can last millions of years (not useful to me here, not developed, funded and not higher density). However, I can say some things that help, more just fundamentals that point in a realistic low density direction. Things that produce a long lasting effect in the stable material. Thinking back to this, digital paper in the 80's was just a punch card like idea that shot holes in the tape at high density. Spatial data recording, more stable than so many high density recording techniques based on effects. Data loss recovery, as used since Viking. Low density. Stability enforcement to stop deterioration. A nightmare, but so enjoyable.

Even if I was to get a perfect crystal lattice and changed it's structure to record data, how long is that going to last? Too much and adjoining data will start to structurally affect each other. Even glass is now shown to be a liquid, leave it long enough and it will flow. I can design a system to compensate for this, though eventually it's got to get bad. All you are left with is designing something in this current level of science and put it in a stasis field, to stop time and therefore corruption, which is impossible in current science (though not in my mind). So brings an idea to mind, use a stable aligned structure lattice to enforce and stabilise the recording media/media lattice structure, similar end effect to stopping time up to the length of the stability of the re-inforcing structure, which brings another couple of solutions to mind to achieve the effect.

The other idea I came up with several years ago for a new stabilised low wear type of tape is actually smart instead. Shame to hear tape is only getting ten years now, they might aswell use the Japanese (nhk, was it?) holographic film they came up with last decade (I actually was considering a cheaper mechanism use holographic like effects). The aim was for a data tape with much much higher data densities and 1000 year data life (possibly in constant use on some media types, but do not know yet). Then again a few more advanced ideas just occurred. Anyway, I remembered I wanted to push the tape idea for quick development sale. Bugger it, I'm having trouble even reading tonight.

Anyway, back to present, there was a 100GB red laser single layer DVD last decade, which technique might be worth looking into, though how to live record at modern densities is an issue. I can't remember, but I think it was shape based.

As for me, except for live backup, a reliable prime meda solution, good verified data backup archive software and multiple copies will do me. Ten years is a bit small. Anymore help would be welcome. I'm just converting my 60k page research reading archive to disk to join the others, by the looks of it, they would last longer on paper , but I can't afford the space.


Thanks guys.

 

[Wayne Morellini]

By the way, what does the thumbs down on my post mean?

 

[Marc Wielage]

The trick is to record the format with the data, which I have come up with a scheme for that can last millions of years (not useful to me here, not developed, funded and not higher density).

I would be extremely surprised if any movie ever made in our time will be looked upon as anything except as a quaint curiosity in 1000 years. How many books from 1000 years ago are still popular today? (OK, except for maybe the Bible.) Who will understand contemporary language this many years from now?

Anyway, back to present, there was a 100GB red laser single layer DVD last decade, which technique might be worth looking into, though how to live record at modern densities is an issue. I can't remember, but I think it was shape based.

I'm not convinced a recordable optical disc will last longer than 10-15 years. Have you read the Academy white papers I linked to in my previous message? It's a pretty grim story, and these guys are very bright.

 

[PatrickFaith]

The Academy of Motion Picture Arts & Sciences hasn't figured it out, either. Here's a recent report:

The Digital Dilemma
Strategic Issues in Archiving and Accessing Digital Motion Pictures

It raises many troubling questions. I don't think there's a real, 100% solid solution yet.

Thanks Marc for highlighting that report, it's is very comprehensive. There was a server that went down last night that plays a lot of my old music (my music is not under my RL name), and I got some messages from fans asking what was up. I'm perty sure my old opera's and stuff are only on a few servers and disk drives since they were never generally released, and if those servers get destroyed for some random reason ... that would be it aside from directors using sections for short movies.

 

[Wayne Morellini]

I would be extremely surprised if any movie ever made in our time will be looked upon as anything except as a quaint curiosity in 1000 years. How many books from 1000 years ago are still popular today? (OK, except for maybe the Bible.) Who will understand contemporary language this many years from now?


I'm not convinced a recordable optical disc will last longer than 10-15 years. Have you read the Academy white papers I linked to in my previous message? It's a pretty grim story, and these guys are very bright.

Hi, Marc, what you quoted is more a conversation with Jeff on technical matters, beyond the Academy (like saying a Jedi equates to be a leading physicists). Actually, most historical documents beyond total trivial stuff are very popular, with their historians and researchers. However, I have much research to keep, and others files it would be good to keep for my lifetime at least. So, the high the number hopefully the statistically less trouble you will have in coming decades. The long length technique I quoted is low density, so don't expect to record feature length movies on it. However, if you have the space, it would be a very good medium for film storage, if you can stabilise the formula, it should last in fires to various heats. Past my practical work, I have considered how to store information for the life of the universe, even how to last beyond such and through to another theoretical big bang with compatible physics, for my fictional writings. Brightness is relative. My tape is meant to greatly increase the reliability and reduce the wear and tear of tape, it would be suitable, and get lengths going into the future as I said. I have actually thought up a further version that increases this reliability as I was writing.

They should be able to design an optical disc that lasts a 1000 years, all they need to do is drop the "consumer" from it and be prepared to make it expensive (very). If you hunt around for a mass market compatible approach you severely limit your options to maximise longevity. For instance, in a simplified example fashion, there has been various liquid diamond coating structures for years, actually I have an eye on a carbon coating for the displays for my devices that has many resistances apart from scratch resistance, maybe still a leader. Anyway, what about forming a close enough to flawless diamond disc with a platter in the middle (though you may not need one if you can alter the structure of the crystalline lattice). You could dope the formula to make it more flexible, it may reduce life, but I'm sure they plan to recopy it within the next 100 years. You then could make permanent alterations to the platter (in reference to a 1000 years). Your biggest problem might be thermal stress of unequal expansion of the encasing later and the platter, but if it is flawless enough the layer should be able to stretch, the platter, depends on what it is.

I think they are better off using the digital paper tape idea I mentioned, replace it with a metal or alloy tape, use a layer to punch hoes in it, then keep it stabilised and sealed (as per the disc). the hoof's should last a very long time. However, as I indicated before, film has a good life and they know how to handle it. You could record digital data in a loss recovery system on film intra frame compression to the required standard in multiple copies across locations. You then get much more life than 15 years and reasonably high reliability as the segregation of binary data is more graceful. Sure it costs money, and uses present facilities until a better solution is found.

My my millions of years (just happens it will last rather than need) is a low density data read only memory for my hardware product designs, it has it's uses.

Design is just a series of directions to choose from.

 

[Wayne Morellini]

As for who will watch the movies in a thousand years, we have approached the pinnacle of 2d screen film making. We watch foreign caption films, even those on stories a thousand years ago, even ones done decades ago. In a thousand years time, if things keep going as is, our current films maybe among the best 2d films ever made, and maybe much better than what they can do then, particularly if tech or law slips back. Maybe people will not want to use 3D holo-decks all the time. I plan on getting the television series "The Sulivans" a ww2-+ family drama for my future kids, to show them what their grandfather and great grandfather's were like. A couple of characters are around my father's age.

 

[Wayne Morellini]

Anyway, what about some more practical information on what is best devices and software? I'll get around to reading the two academy documents, but as I said, it is difficult to read through lengthy things at the moment due to the brain condition, henceforth scanning my 60 thousand page research back reading pile so I can read it more easy (text to speech also) and find what I want.

 

[Brian Merlen]

Personally I like to transcribe my shot sheets on cave walls... As long as those damn monkeys and monoliths stay away I should be all set...