AIT drives, for long-term storage ?

[Nils Ruinet]

Hi everyone,

I just read an article about Red in the April issue of the American Cinematographer.

There is a quote from Stuart English regarding ways to store all the recorded media :

"AIT-5 uses 8mm tape media in a compact drive. For about $60, you get a 400 GB tape with an approximately real-time transfer rate for 24P 4K RedCode RAW footage. We're on the lookout for a model with an eSATA or Ethernet interface to allow us to more efficiently move all that data."

This sounds interesting, 400GB would be enough to copy a full 320GB RedDrive to one tape, and Sony anounces about 24MB/s native transfer rates. There are compact 3.5-inch drives. And it looks affordable.

Here is one for example :
http://b2b.sony.com/Solutions/product/AIT-e1040s
But as always, it's SCSI, which is not very handy to use in the field on a laptop...

Is there a chance that Red will make a custom portable archiving system, maybe using these tapes, that would be easily usable in the field, with just a laptop (or even better, a standalone system, so you would just plug your RedDrive and it would start recording to the tape) ? And maybe a software that would help manage backups, and log everything you recorded onto each tape, making it easier to later find a shot you're looking for...
Did anyone find an AIT drive that has eSATA or Ethernet connections ?

I think storing and backuping all the footage we'll shoot with this camera will be one of the biggest issues in the upcoming years. Having an easy and reliable solution would be great. Tape still seems to be the cheapest and most reliable solution for now, isn't it ?

 

[Jeff Kilgroe]

AIT has a somewhat muddy reputation... I'd look to LTO and DLT-S systems first. But in the end, they're all about the same in terms of what they can (or are supposed to) do.

DLT-S5 and LTO-4 are both 800GB per tape. LTO-3 is 400GB per tape like AIT and it's a bit cheaper, more widely used and regarded to be more reliable.

I'm not expecting RED to provide archival solutions... Plenty of options already out there. I'm also guessing that as write-once FLASH hits the market at commodity prices and just having cheaper solid state solutions, we'll end up shooting to media that can be stored on shelves anyway, as we do now with DV, HDCAM, or DVCPRO tape.

 

[David Battistella]

Jeff Makes a great point.

I think if the price of CF (compact flash) solid state media continues to drop, this might be the best form to shoot on and preserve as your archive media. You can transfer it as many times as you need and keep the CF card as the archive.

This might be a more efficient solution and possible scenario in the future:

Buy 100 CF 32gb cards as a record medium. This will give you about 20 minutes per card

Shoot your film.

Transfer the cards once or twice and back up that media.

Keep the card as an archive.

To me, the smartest in the field back up solution would be a the option to do high speed transfers from your CF card to the RED drive. This way you can fill up your RED drive as an "in the field backup" and keep replacing the CF's

Three RED drives would give you 1TB of, "in the field" storage and you would also use them for long set-ups (interviews etc)

I have not heard whether this functionality is going to be built in to the RED or whether the camera's computer software would allow this type of transfer. You can do this between takes or in transit, etc

It would save having to have a laptop on every shoot which to me is the BIG downside of most "tapeless" workflows. Remember, with tape there was an actual "laptopless" workflow and it is actually very efficient.

Most especially for small production.

David

 

[Jeff Kilgroe]

Yeah, I'm thinking FLASH or other forms of solid state are the future. Obviously, none of it is here just yet, still a few years off before we can start thinking of FLASH as commodity, shelf-able media I'm guessing.

For now, I think the best bet is current tape systems, hard drive backups or some combination of the two. And these should be sufficient until something better comes along. IMO, I don't think holographic storage is going to be the answer. I'm anxious for it to arrive due to its capacity promises and a few other things. But it will have no better shelf life or durability than current optical media. In fact, current holographic media is already showing to be deficient in these areas, even compared to BluRay and HD-DVD. I think it's the next logical progression for optical disc media, but not a great candidate for long-term archival or durable storage.

 

[David Battistella]

Jeff,

I am not sure it is as far off as we all think. I think it will get larger and cheaper and faster FAST. I think eventually it will be as fast as RAM. They are already using this work flow in Spain, so we know it is a very reliable way to record.

With 16 and 32 gig cards I think it will be the "tape/film" medium of the future. An realisitically I have not worked on very many producitons that duplicated every tape or every roll of film shot. So this would fill a giant void in the tapeless world.

David





David

 

[PenGun]

I dunno the XDCAM disc system is supposed to have a 50 year lifetime.

http://www.expandore.com/product/sony/proav/model/XDCAM_HD/PDW-U1.htm

One of these puppies will achieve all your data to XDCAM disc. This is not a great capture solution but a very good achieve solution.

 

[JohnF]

Well I don't know about the XDCAM disks as they are far more sensitive to damage than previous optical media plus even some of the best glass-mastered optical media starts to show signs of significant degration after 10 years or so.
So I don't know how Sony can claim that 50 year life claim.

As for CF cards and the like what are their limitations? I have seen recommendations to keep them away from x-rays(of course) and magnetic fields(why exactly? maybe current induced problems?). At the moment they have no record as a long term medium. What issues will they throw up?

Tape stock will take you to the 20-25 year mark. Which is probably the best outside of film.

By the way people keep mentioning the compressed capacities of LTO format when compared with the uncompressed capacity of the AIT format. Thus making the LTO format sound like it can store more than AIT. Picture/film storage will not compress well so it's better to quote uncompressed as the more realistic number.

Both formats are moving forward in their capacities, some intresting developments are due over the next year or so... I think someone mentioned SATA?

I've spoken with a number of IT guys regarding tape based storage and AIT and LTO both got the thumbs up from their users.

They did mention that when uploading material from tape to server and any errors occur, most of the time just re-trying worked without any further problems.

Regarding LTO/AIT storage of footage, unlike broadcast tape formats or XDCAM, one will have to upload the entire tape rather than random access parts as required. So you need one shot? That'll be two hours wait to upload!

At least that was what I was told... anyone know better?

JohnF

 

[MarcusX]

By the way people keep mentioning the compressed capacities of LTO format when compared with the uncompressed capacity of the AIT format. Thus making the LTO format sound like it can store more than AIT. Picture/film storage will not compress well so it's better to quote uncompressed as the more realistic number.

Well, AIT5 is 400 GB uncompressed, LTO3 is 400 GB und LTO4 800 GB uncompressed.
While AIT Tapes are smaller, LTO systems are a lot faster than AIT.

Regarding LTO/AIT storage of footage, unlike broadcast tape formats or XDCAM, one will have to upload the entire tape rather than random access parts as required. So you need one shot? That'll be two hours wait to upload!

At least that was what I was told... anyone know better?

I use LTO(2) every day and you can read and write small files as you like. It just takes some seconds for the drive to seek the right position.

 

[Nils Ruinet]

AIT has a somewhat muddy reputation... I'd look to LTO and DLT-S systems first. But in the end, they're all about the same in terms of what they can (or are supposed to) do.

DLT-S5 and LTO-4 are both 800GB per tape. LTO-3 is 400GB per tape like AIT and it's a bit cheaper, more widely used and regarded to be more reliable.

I'm not expecting RED to provide archival solutions... Plenty of options already out there. I'm also guessing that as write-once FLASH hits the market at commodity prices and just having cheaper solid state solutions, we'll end up shooting to media that can be stored on shelves anyway, as we do now with DV, HDCAM, or DVCPRO tape.

I mentionned AIT because I read about in in that article, but I never used it, and in fact LTO-3 seems to be a good choice.

As for using CF media to shoot and store media, indeed I think it's the future, but it will become interesting in a few years, when prices drop and capacity gets up (I'd say at least 32GB).
And as JohnF said, it would be intersting to know if there are any limitations with CF media, and how good they are for long-term storage.

I'm thinking about using this camera for different kinds of work, including documentary films. In this particular case, you can end up with a few hours of footage each day, so here CF won't be a solution before a few years... Some weeks ago I ended up with nearly 4 TB of P2 DVCPRO-HD footage, stored on 2 raid-5 drives (the second one was for backup). I would have loved to be able to backup this easily onto LTO tape each night after shooting. I really don't trust harddrives.

So ok, let's say we backup on LTO-3 tapes, which I believe can be a good solution fo now. There is still an issue : managing all your media. How will you be able to easily find a shot you stored a few years ago ? LTO / AIT don't work like an HDCAM tape, you can't just put it in a deck and hit the play button to see what footage you've got there, and find a specific shot. As far as I know, you won't be able to see a shot until you've copied it to your hard drive. Not very efficient...

That's why a specific software that would help to manage all your footage would be great. It should memorize all the metadata of each clip, and on which tape it has been backuped. And, why not, a low-resolution proxy, or a few frame grabs that would help find the footage you're looking for.
Maybe this could even be integrated to RedCine, as we will probably have to process all our RAW footage there anyway ?

Same question, what happens if you need to do an EDL conform with footage that is no longer on your harddrive, but has been stored on some LTO tapes (or on dozens of CF cards) ? With a classic tape-based workflow (I mean with Beta / HDCAM), your editing software would just prompt for each tape by its name or number, go to the TC-in and digitize the needed part. But what happens here ? Do you need to manually find each shot and copy it back to your hard drive ? Or will you have to copy all your footage back to your hard drive, even if you only need 10% of it ? :wacko:
What would be the solution to that ?

Also, I read that Quantum created an MXF-aware 300 GB LTO drive :
http://www.broadcastnewsroom.com/articles/viewarticle.jsp?id=42710
Maybe something like this (but please cheaper than these $8000) for Red would be cool.

Thanks for your answers !
Nils.

 

[JohnF]

I use LTO(2) every day and you can read and write small files as you like. It just takes some seconds for the drive to seek the right position.

Cool, well that cleared that up!

The guys I spoke to use tape drives to mirror entire networks for large organisations therefore probably never bother with that kind of random access.

How fast are LTO drives then? Would you consider the quoted 40MB/s a reliable figure to go by or is that a "peak" number?

LTO 3 and 4 are quoted as having even faster rates, some are even suggesting that they are faster than hard drives!

But one thing is going for AIT-5(~25MB/s)drives and that is size. They can fit in a 3.5 inch bay! (small tapes too!!!)

JohnF

 

[number6]

I deleted the email from Tiger Direct, because I don't need another hdd at present. But I'm pretty sure they were selling Seagate 10 series (perpendicular recording) 500 GB hdd's for about a hundred dollars and with no rebate requirement. This to me is the best solution for the present.

If I was going to receive my camera anytime soon, I would have loaded up on them and just swapped them out of an eSATA external storage device as the drive filled up (also works as USB) and shelved the data on the drive. Seems cheapest and easiest to work with via computer, to me.

 

[MarcusX]

How fast are LTO drives then? Would you consider the quoted 40MB/s a reliable figure to go by or is that a "peak" number?

We have a daily backup running from harddrive based virtual tapes to an Ultrium2 tape robot and it writes 1985 MB/min, which is 33 MB/sec. I have an LTO3 drive, too, but it's currently not installed.

LTO 3 and 4 are quoted as having even faster rates, some are even suggesting that they are faster than hard drives!

I'm pretty sure LTO4 does more than 100 MB/sec. More important is the minimum streaming speed, the drives can't write below a certain threshold.
If the system can't deliver a constant datastream, the drive stops und starts, which ruins the throughput.

 

[Thomas Mathai]

Whatever you use, make sure to make TWO backups of the same data.

High density magnetic tape media CAN have a short shelf life.

Also make sure your archive is universally retrievable. Some use Retrospect or Windows Backup.

A tar archive is still an industry standard, since it can be extracted on a standard Unix, Linux system. I know there were tar apps for Windows and Mac, not sure what the current ones are.

It seems tar on OS X can't work with devices, but that just means someone has to write the drivers.

 

[JohnF]

Whatever you use, make sure to make TWO backups of the same data.

High density magnetic tape media CAN have a short shelf life.

Yep

Note that LTO-4 will be using a thinner tape stock. This may or may not impede its reliability...

What interests me is will smaller production companies or sole operators need a complex archiving system and possibly archiving staff? Or are there simple programs for both backing up and retrieving data that require little effort.

Lets put a case study here.

One is shooting a small corporate or low budget music video. Maybe four hours of data may be needed to be backed up to tape.
Is this a simple case where the day's shooting data is up-loaded from the REDDRIVES to a computer (basic non-raid drives) and then with a simple program launch quickly then set the data transfer going.

Would a set of raid drives be better than a single SATA drive as far as data transfer rates go? because as MarcusX mentioned:

I'm pretty sure LTO4 does more than 100 MB/sec. More important is the minimum streaming speed, the drives can't write below a certain threshold.
If the system can't deliver a constant datastream, the drive stops und starts, which ruins the throughput.

How badly can throughput be effected and can this effect data integrity?

JohnF