Archiving Data, permanently.

[Gavin Greenwalt]

I have personally scanned nitrate films (some Laurel & Hardy shorts, circa 1933-34) that was actually in fine condition, more than 50 years after it was shot, and it was fine. Are you absolutely certain that an R3D file today will be recoverable, decodable, and viewable in 2062? I'm not so sure. (And I would say that for all digital file formats.)

I have had at least six major, A-level studio features affected by LTO2 crashes and data losses in the last ten years. Granted, most of them have been fine. But just one of those resulted in more than $500,000' worth of work having to be redone. Don't forget the infamous Toy Story 2 data loss story...

Sure the LTO can have a problem but I would point out the Toy Story story was the perfect reason WHY I would go LTO not film. If you have one copy and it gets destroyed (ala Toy Story) what saves the day is having numerous cheap copies. How did Toy Story get saved? It was on someone's laptop. You aren't likely to just find a random extra 3 strip backup somewhere. If you make 300 LTO copies for the price of one optical copy you can hand them out as party favors. That is seriously more resilient than one expensive three strip backup.

And no you shouldn't be using R3D. Transcode to JP2k or equivalent. You won't have any problem what so ever finding LTO decks in 30 years. I could buy Zip Drives by the palette (literally, in unopened boxes) I guarantee one will work in that palette and I guarantee you can get them at a couple bucks a piece. If you're really paranoid you could even fit a small LTO drive (or 20) into the space of 3 film reels along with a cheap netbook. But why bother? It's consumer mass produced hardware. Ask your nerdy uncle if you can borrow his old clanky intel i7 in his attic.

My dad's apartment caught on fire a year or two ago. Lost a bunch of negatives. Digital files? Safe. You can fit a pile of LTO tapes into a safety deposit box. Where are you going to store 3 huge reels? Will it be climate controlled? Can you afford 2 more offsite backups? What's it going to cost to find a good optical scanning service for cheap that isn't owned by MGM and in-house? Your local post house is going to sell their film scanner in a couple years. I've lost 99% of my data to misplacement and accidental deletion and probably 1% to actual hardware failure. The Toy Story incident is the perfect example of redundancy being superior. It's the cockroach principle, you can't kill them all. And if one dies you just copy one and replace it. It's like the moon landing footage. The film wasn't degraded.... it was lost. Luckily they had a cheap copy somewhere else.

 

[Marc Wielage]

Anyway, what about some more practical information on what is best devices and software?

Multiple copies of the final uncompressed DPX files on LTO-5 (LTO-6 is not yet out). Backup hard drives, preferably migrated every 3-4 years to new drives. A high-def 1080 24p copy on HDCam-SR. And a 35mm film negative output. Plus all the audio as digital files and analog 35mm magnetic film if possible.

That's pretty much what the big studios are doing for features: covering their bets by having multiple copies in different formats, hoping one will be playable in 30-40-50 years.

 

[Marc Wielage]

How did Toy Story get saved? It was on someone's laptop.

A few clarifications. It was a desktop.

You aren't likely to just find a random extra 3 strip backup somewhere. If you make 300 LTO copies for the price of one optical copy you can hand them out as party favors. That is seriously more resilient than one expensive three strip backup.

How much actual experience have you had with YCM separation negatives?

And no you shouldn't be using R3D. Transcode to JP2k or equivalent.

Note that R3D files are based on JPEG2000. I do agree that it makes more sense to archive the final uncompressed DPX files. R3D is a moving target, and there's no guarantee that the R3D files being shot today can be debayered the same way in 10 years as they are now. (That's assuming the backup medium still works.)

You won't have any problem what so ever finding LTO decks in 30 years. I could buy Zip Drives by the palette (literally, in unopened boxes) I guarantee one will work in that palette and I guarantee you can get them at a couple bucks a piece.

You assume the disks will still play. In the three or four data disaster cases I personally know of, the LTOs failed and had massive errors, I suspect due to tape shedding. Drives were fine.

My dad's apartment caught on fire a year or two ago. Lost a bunch of negatives. Digital files? Safe. You can fit a pile of LTO tapes into a safety deposit box.

Fires can burn anything. The massive file that burned up one of Universal's warehouses in June of 2008 destroyed films, mag tracks, album master tapes, digital audio tapes, digital videotapes, and digital files. Fire doesn't care what it burns. Hard drives and LTO tapes can burn, too.

What's it going to cost to find a good optical scanning service for cheap that isn't owned by MGM and in-house? Your local post house is going to sell their film scanner in a couple years.

Actually, I think the cost of scanning and scanners has come down, and a lot of companies still have them all over the world. I don't see a huge market for new ones -- I think they've pretty much sold all they need to at this point. And it's not that hard to keep the existing ones running. There's hundreds of them in LA, very few at the studios, far more at major post-production houses.

I've lost 99% of my data to misplacement and accidental deletion and probably 1% to actual hardware failure.

Yep, can't argue with that.

 

[PatrickFaith]

I received an Charlie Chaplin print today, 18 fps 16mm (presound single reel) ... I have to admit collecting old silent movies is perty addictive. Movie played really well (I had to remember my old school techie skills ). Another movie was rewound wrong (i had to remember how to do that half rotate trick). The 2nd short I played I think had only been viewed EVER once ... and I have the only print (master), that has to mean something for the case of film archiving(i recorded it on my RED just in case there was a problem though). It's also just real nice to unroll the reel a bit and just look at the images rather then having to get some digital system. For "puddles of light" i'll probably make a 35mm print just for my private use and archive. Anyway I'm for doing both digital and film archiving, the more formats and the more distributed the copies are, I think is the best archiving strategy.

 

[Gavin Greenwalt]

Note that R3D files are based on JPEG2000.

Until RED encrypted them. Now encrypted they're completely useless. JP2k is an open ISO standard that will be around for decades. I don't think R3D itself is a bad archival format. But RED has made it clear that if they want all of my footage to stop working tomorrow they have the power to make that happen. "Oops REDCine expired."

You assume the disks will still play. In the three or four data disaster cases I personally know of, the LTOs failed and had massive errors, I suspect due to tape shedding. Drives were fine.


Fires can burn anything. The massive file that burned up one of Universal's warehouses in June of 2008 destroyed films, mag tracks, album master tapes, digital audio tapes, digital videotapes, and digital files. Fire doesn't care what it burns. Hard drives and LTO tapes can burn, too.

My point isn't that LTO is some magical format that can survive a fire. My point is that they are *cheap*. How much does a film transfer cost? How much does an LTO5 tape cost? If a fire sweeps through one warehouse... you have another copy in a safety deposit box. The most reliable format in the world is redundancy. If one tape tears... fine, pull out one of the other 30. It probably won't tear in the exact same place. Problem solved.

The world is a dangerous place. The most effective backup in existence is shear quantity. That's why we still have 3,000 year old books. Not because they were written in stone... but because monks copied copied copied copied copied.

 

[Martin Weiss]

Patrick, you might want to get in touch with the Association of Moving Image Archivists, you just might have found a gem.

As to the new Kodak Asset Protection stock, I see two problems for long term archival:

1.) It needs to be processed with the ECP2 (print) process, rather than the process for negatives. As there are hardly any more prints processed these days, it will become harder and harder finding a quality lab that can actually develop the film.

2.) More serious, the market for film scanners is small, and scanners are expensive (the Scanity we bought last year was almost $1m). Most likely there won't be any professional scanners built in 20 years time, and after that it will be only a matter of time before the support for those scanners is no longer financially viable. You might end up with pristine film copies in your archive, but no way to digitize them.

I work at the National Library of Norway, where our goal is to preserve everything for the next 1.000 years. Films are scanned in 4k, and both the DPX files (wrapped in a Mezzanine format), sound files and meta data are saved both on hard disks and tape, in multiple locations deep inside the Norwegian mountains.

 

[Marc Wielage]

Martin makes many good points.

I don't think there will be any new film scanners built in five years. I think they're pretty much done, and there's still quite a few available on the used market. I also think a lot of the parts for scanners will be easy to machine, except for any custom chipsets in the logic boards and video processing boards, plus the CCD pickup itself. If those go, then repair will be an issue.

 

[Wayne Morellini]

Personally I like to transcribe my shot sheets on cave walls... As long as those damn monkeys and monoliths stay away I should be all set...

Multiple copies of the final uncompressed DPX files on LTO-5 (LTO-6 is not yet out). Backup hard drives, preferably migrated every 3-4 years to new drives. A high-def 1080 24p copy on HDCam-SR. And a 35mm film negative output. Plus all the audio as digital files and analog 35mm magnetic film if possible.

That's pretty much what the big studios are doing for features: covering their bets by having multiple copies in different formats, hoping one will be playable in 30-40-50 years.

Yep, spent last night working out how much I needed where, expensive. Will have to keep a schedule book to keep track, was planning on multiple copies in multiple formats. Went shopping tonight.

Yesterday I checked one of my Micro sd cards that I backed my tablet to, looks like one of my rare earth magnets got close to it, the latest backup was totally corrupted, and a couple of files in key areas in the other. Too sick to get around to moving latest version on. I was wondering why windows just did nothing when I tried to copy it, would have been better if it notified me, did a report in a text file for maintenance, and just left the two files where they were and copied everything else. Windows could do a replace file if older and smaller, otherwise ask, option with data integrity verification, during copy and moves, would solve a lot of issues. Maybe they could have options to identify duplicate files and older versions through bit level compare and data integrity verification to determine and also after the move, and replace or keep older versions, gives a incremental backup in the drag/paste functions. Takes so long to improve it.

You might end up with pristine film copies in your archive, but no way to digitize them.

I work at the National Library of Norway, where our goal is to preserve everything for the next 1.000 years. Films are scanned in 4k, and both the DPX files (wrapped in a Mezzanine format), sound files and meta data are saved both on hard disks and tape, in multiple locations deep inside the Norwegian mountains.

Exactly, those mountains are great places. However, the films should still be scanable Martin, simply dye to the progress of technology, people should be able to 'rig' something up. I have a few ideas for a technology that could easily scan it, I am interested in inventing diagnostics scanners for medicine and archeological. I try to keep the discussion here on a few more concrete archival storage techniques, but there is many ways to skin the cat. Simply, put, a researcher, professional or enthusiast will use new tech to decipher the reels if enough call. What is likely, is that so much will be converted to digital and be preferred, and protection will focus on that. But even that, when it gets old and even the electronics on the drives that contain them break (this is a big problem that most people that advocate drives are unaware of, is that the recording process and even the heads are customised to the formulae of the recording media they are used on, even data recovery companies try to use the original heads to recover) once the drive breaks they get issues, which I'm sure you are mire aware of than others. In that case, in the future you could scan the media itself, and decipher the data on a computer after. So, future scientific and professional equipment could be setup to do this. You really should read what I wrote before, holes will outlast most of the strategies and technologies mentioned here, by far. You can't rely in people to do the right thing to avoid disaster, basic media integrity and reliability makes up for a lot:

@ For everybody, put it this way, say you have a overwhelming back up hire across 30, or much more, but the media is only reliable until failure for a year, people are heading for disaster even if they do a thousand copies. However, if you have a media with integrity for a hundred or a thousand years, you needs less and even if you miss integrity tests and recopying in a year or ten, you are likely to be more safe. So extra copies matters, but integrity makes it much more reliable with less work.


Still talking with everybody. What is needed is new standards of archival products. Media suppliers could be making themselves 10 times more profit , in reality, simply by developing common archival standards of format, like for consumer disks formats, for all companies to sell for a bit more (rather than 4 times the amount) when margins are thin (if the manufacture makes 1c clear on the retail dollar they can make 10c, the retailer and distributors etc another 10% each). It does not have to be anything fancy, just proven reliable way to make it in common, so that a common standard label tells the buyer how it should work, easily look able up in the distant future. With optical disks it maybe new, or an adoption of the existing, but for hard disks it will mean that in 20 years, if the electronics fails you can open it up and use a standard recovery head to recover. It means you can sell sealed caddies with head and no electronics, using the most reliable disk media format (the number of recording techniques is numerous and the tricks used to get density are bound to reduce reliability a very lot, and it takes tine for them to optimise those formulas, so that many newer mire denser drives are bound to be much more unreliable, or more unreliable as they figure how to make the format cheaper etc (open the caddy, insert the caddy into a off the shelf recovery caddy, and recover. This is a great commercial strategy to develop, for everybody involved, including museums and libraries. But again, even recording to film digitally will increase reliability as another strategy alongside other copies on tape and drive until a more permanent solution can be found, but also decreases the price of film, by increasing production, and if it is thinned and treated like an oversized cassette). But, the best strategy to also bury in a mountain, is a reliable proven/provable (like holes) low density protected more costly format that will last. These are natural treasures, so even if low density archive technology is not directly suitable for consumer, it is significantly desirable for public and specialist funding, they just need to bite the bullet to set the first standard. With low density binary recording, as I am advocating, it is either there or not, so very easy to read for somebody knowing little, and the data format description can be recorded at regular predetermined intervals. This is the sort of things I think about.

I started looking at designing data storage techniques in the 80's, when there was a optical credit card format developed. I dreamt (at that stage) of making a sd card like scanned optical 'chip' let's call it, to be used instead of floppies. I moved onto new ways to read data and stabilise optical media, even to read data off disk at well over 1 terabyte a second, and techniques so fast you could use optical media as the main processing memory. A 360 degree stadium laser display design proposal I considered stabilising would have required a circumference velocity of over a 1000th the speed of light. I do not say it was impossible, but the likely hood of explosion made it undesirable and designed in a slower way.

rewrite:
The reality is that these more profitable archival standards I am proposing, could account for up to half unit sales, as many consumers want to store stuff long term as well and might think that what they are buying will be fine for the job, but if they knew would prefer something that actually works for actual archiving.

Tonight I was out looking and talking. Most external hard drives don't tell you about data life or reliability, as with flash (working backups). I had tremendous problems finding archival quality disks, even at the leading photographic store. I found only two that even mentioned archival, one Sony products, but no information to qualify it except one product with a unlabeled plot graph, and Verbatim BD-R disk that mentioned the harden coating and the metal recording layer (no real details). How can anybody know and buy with certainty. If Red wanted to get into physical media for consumer and professional the things I mentioned here are a good to look at.

 

[Wayne Morellini]

Marc, sorry, I mistakenly left out the reply to you in the last post . I have put it in now.

 

[Wayne Morellini]

Hmm, Martin, are there any good guides out there on the best back up formats products and strategies. Probably a good strategy to find the best disks and products?

Thanks, and thanks guys.

 

[Wayne Morellini]

http://www.tomshardware.com/news/Hitachi-quartz-glass-data-dots-Takao-Watanabe-Superman,17856.html

 

[Marc Wielage]

Hmm, Martin, are there any good guides out there on the best back up formats products and strategies. Probably a good strategy to find the best disks and products?

Some of that is covered in the Motion Picture Academy's article I mentioned earlier. I generally see Quantum and HP LT05 drives out there, often tied to HP or Dell servers, and Maxell and Imation tapes.

Every studio I've dealt with seems to have a slightly different backup strategy. But I think the idea of backing the final data up to two sets of hard drives, two sets of LT05 tapes, HDCam-SR HD videotapes, and 35mm negative, is pretty sound. You also need all the final audio mix stems as WAV files on drives, optical discs (I have seen Blu-ray recordables used for audio data), and analog 35mm mag tracks. The only sure fire data methodology would involve migrating the data every few years to newer drives and/or tapes, and doing a check-sum to make sure the files are OK.

It's a good question as to what SMPTE and the MPAA are proposing, but the Association of Moving Image Archivists is having a convention in LA on October 26th, and I know this is an ongoing hot-button topic with them. They are extremely concerned about the long-term storage issues of digital files, both sound and picture.

 

[Wil Wong]

what about all the raw footage

what about all the raw footage

Say, do all your suggestions also apply when it comes to the raw footage as well? In other words, Do folks try to hold onto the outtakes and cutting room floor raw footage or does that stuff in the digital age get erased? Storage for all the takes must be horrendously large. Thoughts?

 

[Thomas Koba]

.....just.....relax....breathe in......and let it go.....everything fades eventually.....:closedeyes:

 

[Marc Wielage]

Say, do all your suggestions also apply when it comes to the raw footage as well? In other words, Do folks try to hold onto the outtakes and cutting room floor raw footage or does that stuff in the digital age get erased? Storage for all the takes must be horrendously large. Thoughts?

That's a very good question, Wil. There have been many great 1960s and 1970s movies -- I'm thinking in particular of The Godfather -- where the studio was wise enough to hang on to many of the outtakes, workprints, test footage, and screentests used in the making of the film. Years later, they were able to salvage that material and use it as extra bits on the DVD and Blu-ray releases.

I'm doubtful that most digital filmmakers today will want to hang on to every single scrap of material shot for their projects, especially if it's 4K (Red or otherwise). It's just too damned much data to have to worry about storing. It's hard enough making multiple copies of a finished 2-hour 4K feature; if you have an hour of footage for every minute of finished material, it could be an enormous amount of data.

The sad part of that is, I tend to doubt that we'll get outtakes, alternate endings, and other footage from digital movies made today and re-released on home video 20 or 30 years from now. Most likely, the only thing that will survive is the finished movie, as-is. No doubt, there will always be cutting-edge A-feature guys like Soderbergh, Jackson, and Jim Cameron who will be the exception.

One can make a good argument that the same is true of modern still photographers, where future scholars won't be able to go back and look at all the discarded photographs of the next Ansel Adams. The same thing has already happened for rock artists, where there's no tape vault of dozens of hours of discarded songs and outtakes, since they just record everything on Pro Tools. Chances are iffy that all that stuff will survive over a long, long period of time.

 

[Wayne Morellini]

Good stuff gets dropped from even the top films because of studio pressure, and just length pressure. I generally watch the outtakes on my discs of good movies (try that with iTunes) and any special features that I can learn from. Most of the time the out takes are good, and up to half the time taking them out actually makes the movie worse. I think there is gold in them, and they know what to keep. I think people should make the best few of stabs at it to keep. I hate to say this, we really need big meta data buttons one labeled something like 'utter crap', another 'Keep' and two rows labeled good, excellent, fantastic, and the next row, some good, some excellent and some fantastic, and simply hit the ones that most apply as we watch it the first time, and when we finish, go through everything starting from fantastic downwards until we have enough for each scene to keep, and hitting the buttons for the bits of the scene we like. Hopefully we will have enough foitage to keep by the time we get through fantastic labeled footage. The remaining 90%+ can fade away.

Marc, still bogged down here trying to scan in my research reading and being sick off and on, so forgot about that article.

 

[Wayne Morellini]

Forgot to mention, a few things. Sometime ago I read they estimate they could store an unlimited amount of information on the wave function of an electron. Researchers in Australia are now recording on electrons, but am unsure of the method. Realistically the electron would then become the cat in the box, and the more data the more havoc reading it would make. So, a simple methods short term might be to leave bands empty that could be used to read the data with minimal loss (or clone the electron wave function for each read). I should be able to non destructively read and write (past the new info). with the method I am thinking related to the idea I mentioned before for archiving longer than the life of the universe. Very late and my Brain is leaving, but the cat in the box thing, where observing the cat set's it's state. Scientists are not engineers, they have very simple views. When I first read about that it occurred to me it could be cracked, there were ways to observe without effecting the state too much , or at all, long-term, same with when I heard about various forms of quantum cryptology. Some scientists might tend to see things in black and white, according to what they know and their prejudices, why cynics that call themselves skeptics, think they understand science, it is about the absolutes of fuzzy denials, rather than genuine scientific exploration of truths to establish truth, even newer truth. They really mentally make me roll my eyes cynics. Now, rather than black and white, a good engineer simply figures out a way to repaint the room. Been doing a lot of this.

Anyway, sometime ago I came up with a concept to revolutionise intra and inter chip communication, by using photons to communicate. A number of times faster than electrons and also revolutionary useful in simplifying board design, which I was aiming for low volume boards and hobbyist. Another company has come up with the concept now. Can't tell much about the optics of my design, as far as I know, they have not got that yet. However, I have narrowed down the last block in the methodology path of my optical computing technology. I am now looking at response times well in excess of 10ghz native for memory, it will be a challenge to keep up with it until future generations. Just started to glimpse into archival forms of this memory then but a loud recording started on my TV recording has driven it away.