Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Red and Blue-Ray

spoiled users

spoiled users

Rob Lohman said:
How do you figure? The IT world has been managing GB's & TB's of data on a daily basis through various systems like tape backups. Complete with automatic tape libraries & robots.

This stuff exists and is reliable. Of course you can backup to either an offline or online hard drive system, but that doesn't have nearly the same track record.

It might be cheaper to store it on drives, but I don't see why tapes "just don't cut it".

How did you think the IT world do their backups?
Click to expand...

i think we're spoiled. we've been so accustomed to fast random access at such cheap prices that many of us overlook some very real solutions like tape backup. my first experiences with tape backup was about 10 years ago when it was much cheaper than the hard drive equivalent storage. it was a hassle but i saved money using tape. now, especially to users like me, it's not so clear. why should i invest in $5000 tape machine at $75 a tape. why not just keep buying and building sata raids? they will only get cheaper, bigger and you still have that convenient and fast random access. i'm not up with the latest tape backup innovations but they don't seem to keep pace with more consumer-like solutions like hard drive. i think i'll just keep buying drives, occasionally archive to even bigger drives and hope the holy grail of hologram storage will finally surface one day. until then, at least i know i'm spoiled.
 
vanguy said:
You can get a 2 TB drive from LaCie for $769 US. I'd guess you'd want about 10 TB to store a feature (100 hours or a 50:1 shooting ratio, to be generous), so double it for safety, and your storage costs are $7690. Shelf space is about the size of a medium beer cooler.
Click to expand...

Not to diminish the discussion of storage because it is a very important issue, but a 50:1 shooting ratio? OMG! This is what (affordable) digital acquisition has done to us. Back when I was in film school (and is was not THAT long ago) a 15:1 shooting ratio was considered excessive. That's because they taught us to use things like Storyboards, Rehearsals, and pre-viz. Granted we didn't have Hollywood budgets, but just because we are replacing the film equipment doesn't mean we have to replace all the film techniques. Many of them are still valid and beneficial.
 
I picked 50:1 to be extra generous. Doc work can easily go that high. But I'm allowing for the Rodriguez style of "let the camera roll; tape is cheap" dramatic filmmaking, which is sure to happen when you're using disk media.

Speaking as an editor, I'd sure like it if my clients (most of whom use digital acquisition of one kind or another) adhered to those planning techniques. I usually get boxes and boxes of tape, all of which has to be viewed and sorted. The project I'm working on now has over 200 hours of source media on the drives. Fortunately, most of it is logged.

That said, Red allows you to do things you would've looked twice at before, like shooting the full scene from all the angles, and not aborting to save stock. Or trying a new camera technique knowing you can delete the take if it's bogus. And many image control options.

But it would be nice not to have to sort through all that media.
 
Log-jams and bottlenecks

Log-jams and bottlenecks

Large amounts of source material is where camera liketh XDCAM-HD shine because of the disk based file system. You can immediately see icons of your clips and delete unwanted ones. There is even software for your laptop to allow clip sorting, logging, and meta tagging on location.

If RED can eventually have this capability it will make life so much easier and and also cut into Sony's advantages with XDCAM.

Sorting though source material is drudgery. Now that video is file-based, technology can help ease the pain so you can put your time and money on the creative.

Keith
 
wigby said:
... why should i invest in $5000 tape machine at $75 a tape. why not just keep buying and building sata raids?
Click to expand...

It depends on how you rate reliability. Tape is the safest archival medium we have. It is also fast and compact, and for really big storage volumes it is quite cost effective.

Example: The LTO-4 (which costs rather less than 5000 USD) will store 800GB uncompressed per cartridge, and transfers 120MB/s. A 20-pack will cost around 2700USD, for a storage cost of 17 cents(US) per gigabyte.

The 16 TB storage (drive+1 pack tapes) is a bit over 7000 dollars. Compare this to the cost of 80 400GB SATA-drives you need for RAID-1 with lower reliability, and the cost of convenience is not negligible after all.

You can reduce the number of drives by using RAID-5, but with lower reliability and the risk that you will be locked in to a specific controller or software.

I use SATA disks for my home videos (and other personal stuff), but then I copy important stuff to other media. Even so, I am tempted to get a used tape drive even for home use. For work there is no alternative to tape backup.

http://www.tandbergdata.com/index.p...&task=view&id=173&sid=1&Itemid=245&pdbl=1&a=8
 
vanguy said:
I picked 50:1 to be extra generous. Doc work can easily go that high. But I'm allowing for the Rodriguez style of "let the camera roll; tape is cheap" dramatic filmmaking, which is sure to happen when you're using disk media.
Click to expand...

Well, I can tell you that I most certainly won't be thinking "disk is cheap" when shooting with a RED. :)
 
I thought I'd break it down (because that's the kind of mind I have):

Tape: total backup time 30 hours for about 10 GB. You must be there to change tapes every hour or so. Cost $4K (one-time) for the drive. $1600 for media. Remount: 30 hours, attended. So you gotta add the cost of a tape monkey for every show. Or buy a robot ($$$$$).

Disk: Total backup time 37 hours. Can be set up to run completely unattended. Cost $3245 for firewire 800 drives, or $2400 for raw drives plus maybe $600 (one-time) for enclosures. Remount: instant for firewire, maybe a half hour for raw drives.

Disk may require double backup or RAID 5 parity storage which increase the cost by 100% or 50% respectively. But the price of HDD storage is dropping according to Moore's law, and will drop below tape in a year or so.

Both methods seem to have a shelf life of 3-5 years according to people's experience (including mine; I've had more DLTs go squirrely than HDDs). So both would require rearchiving every 3 years or so.

Seems like about six of one and half a dozen of the other, both cost-wise and reliability-wise, but HDDs are a lot easier.
 
Rob Lohman said:
How do you figure? The IT world has been managing GB's & TB's of data on a daily basis through various systems like tape backups. Complete with automatic tape libraries & robots.

This stuff exists and is reliable. Of course you can backup to either an offline or online hard drive system, but that doesn't have nearly the same track record.

It might be cheaper to store it on drives, but I don't see why tapes "just don't cut it".

How did you think the IT world do their backups?
Click to expand...

I am completely aware of how the IT world do their backups. I am also aware of how much it costs, the time required, the personnel required, the storage conditions required and the complete pain in the arse that it is to restore something from say a year ago. Tapes and tape systems are not getting cheaper, discs are ridiculously cheap. I used to use tapes extensively and in my experience it is necessary to upgrade your tape system on about the same time scale as you would your disc back-ups making the supposed reliability advantage moot.

I think making tape archives make sense for an organisation where the information is rarely restored. These days I seem to be creating mountains of video and graphics data and I need to have instant access to that data for at least a couple of years, or more. Now in three years time do I archive that on tape or simply quadruple my online storage at minimal cost?

I think disc/file systems will be getting a lot smarter (ZFS?) allowing for relatively risk-free 'on-line all-the-time'. Tape's days are numbered.
 
I think we have three tiers of data access:

1) projects currently in the pipeline, probably want those on fast raid's or nas/san solutions with a high-speed network

2) completed or on hold projects you might want to keep online for a while. These could be moved to a larger but slower access system, probably still disk based though

3) you should also have a backup system in place for long-term storage and backup of the previous two tiers

When talking tapes I was a) responding to your comment that they don't cut it and b) I was thinking about tier 3.

I never said they were in-expensive or did not require maintenance (which any long-term archival solution needs!!!). The only thing I said is that it works and has been proven to be reliable (through various means).

Depending on what amount of data you need to store and how reliable you want that to be you'll pay a certain amount for a tape system. This can be as low as couple of hundred bucks for a DAT based system in a DELL PowerEdge server or go all the way up to hundred of thousands of dollars for a tape robot.

My main point is that you should not discard what has been proven to work just like that. Managing stacks of hard disks has its own problems.

However, personally I would probably be inclined to store it on both drives & tape to reduce the risk of either one failing. Of course the tape will be restored (like a verified copy to a hard drive) to verify it has been properly written.

In both cases you will need to check / maintain your archive to make sure everything still works and is sufficiently backed up.

You can create some pretty neat backup systems that automatically copy data from one tier to another (access getting slower and slower) to facilitate multiple backups.
 
The one thing about disk-based shooting is that it starts to follow an IT model rather than a media model. Which is new for us media-only types, and sometimes just plain too much work. I can't really afford all the infrastructure for a three tier storage system and the time to manage it properly. I'm a one-man show.

I need something that approximates videotape or film; shoot, bring it online for editing, put it in the vault. Forget it.

My experience has been corporate/educational/documentary with some drama. The drama usually gets archived forever, almost never being brought back online after completion.

But the other stuff gets reused for years, sometimes decades. And you need to bring it online relatively quickly. I've had some experience with DLT backup in the past, and it was always faster to redigitize from the camera tapes. And if you weren't sure which videotape something was on, DLT would have taken silly amounts of time compared to scanning videotapes.

So I kind of wince at an IT Solution which involves thousands of dollars of investment in the kind of technology that normally goes obsolete in a very few years, takes many hours of management on every show, and is rather difficult to bring back online. And if I put "data management" as a line item on an invoice, my clients won't complain, they'll go elsewhere.

It has to be stupid easy. Putting a show on firewire drives fits that model. I'm not dissing data tape as a reliable, cheap long-term storage medium. Go ahead and use it. But it won't work for me.
 
Disk is still a linear media when you think about it. It just has a much lower seektime to tape. You want real random access, flash is it. The great thing about tape is that unlike disk, the data is scattered accross hundreds if not thousands of feet of media. Your eggs are not all in one basket. If You archive everything on one platter, and that platter goes, then your screwed (If storing on drives, you would have 2 mirror drives at the least anyway). On the other hand, if part of your tape physically gets damaged, you can cut and splice it like you would film and attempt data retrieval. It would be good if REDCODE could be stored as seperate frames so that any corruption in backup wont effect the entire file..
 
I think someone's gonna have to come up with a way to cheaply, easily, and safely store the petabytes of data we're all gonna have in our vaults. Holograms? 1.8" 1 TB mini-drives? Ginormous flash RAM? Self-backing, error checking storage arrays?
 
My archival storage plan will center around the modern, ubiquitous 3.5" SATA hard drive, sitting on a shelf, wrapped only in one of those soft silicone protective bumpers and labelled with the relevant info. They will only ever be used twice - once to record the camera's footage to them and verify that it is correct, and a second time to transfer that footage to the editing system drive (again verifying its integrity as you go) to do with what you want (and make further backups if you are that paranoid).

I come from the IT side of the fence and having seen plenty of drive failures in my time, it basically boils down to this - unless it has a head crash while operating, the data on the physical platters will be fine even if the electronics board dies or some other mechanical part of the drive dies. This means worst case scenario (remember, we're talking IF you need to restore from the original archive for some reason) you take your drive to a recovery centre, pay your money and they recover the info to a new drive.

For my own footage (my movies and projects) I will have two archival copies of all the original data in separate locations. For my clients, I will retain one copy of the data and I will give them the second copy as their master. How they back up from there is their concern, but they can always come back to me for the other copy if needed. Everything will be on 3.5" drives, with the client paying for the backup copy I retain (which at roughly JPY 8,000 per 320GB SATA drive is hardly a huge cost in the scheme of things).

Neat, simple and for all intents and purposes providing a secure backup system.
 
Guys, holographic storage system is HERE!
http://technology.guardian.co.uk/weekly/story/0,,2080840,00.html

The world's first commercial holographic storage system is launched this autumn, with the product able to store the equivalent of 64 DVD movies on a disc about the size of a CD.

Holographic storage offers extremely fast data transfer rates - currently up to 160Mbit/sec, though there are plans to increase this.

The first holographic products are certainly not mass-market - a 600GB disc will cost around $180 (£90), and the drive costs about $18,000. Potential users include banks, libraries, government agencies and corporations. (Filmmakers..anyone?)

The source material (such as a photographic image or video footage) is encoded with error correction and channel data, but instead of the encoding being carried out on a bit-by-bit basis (as with say, CD or DVD), around 1m bits are generated at one time, which are recorded as a single page. The process uses light from a single laser split into two - a signal beam, which carries the data, and a reference beam. Where the two beams intersect, the interference pattern (hologram) is recorded within a photosensitive storage medium.
 
Er... InPhase is already selling their holographic drive with 300GB media. 600GB is due by the end of the year and 1.2TB media sometime middle of next. I stopped by their HQ (about a 25 minute drive for me) and saw the demo. I have to buy through one of their dealers, but I contacted two of them to get quotes. Both quoted full MSRP and wouldn't budge from it (those slimeballs... maybe I should become a dealer). But MSRP is $16,999 plus any cables and interface controllers that may be needed. That price gets the drive unit and one piece of 300GB media. InPhase is also selling the drive under other brand labels like AGFA.

160Mbps is not extremely fast... That's actually extremely slow. Only about double the speed of BluRay, which is not fast. Many tape systems are much faster than this and hard drive storage races past this quite easily. 160Mbits is only 20MB/s.

The InPhase Tapestry system as well as the Maxell holographic product, the media is very durable. However, it doesn't have a very long shelf life at this point -- roughly 15 years estimated as a maximum. The holographic film currently being used by these manufacturers degrades over time as badly as many of the original organic dye based CD-R media. InPhase has been very open about this and they are working on correcting this as a top priority.
 
It will get better, obviously... Just think back to burning CD-R media with the first generation burners... 80 to 90 minutes a disc. But everyone thought it was the coolest thing ever -- we were writing to a CD! Wooohooo! I paid almost $1800 for my first CD writer and blank discs were $14 each. So, BluRay isn't much different. Actually cheaper to start out with. Prices will drop and performance will improve. I just hope Sony doesn't use their format licensing to control BD media prices with the lame excuse of controlling piracy. Which they have already started to do in some sense. Not cool...
 
You must log in or register to reply here.
Back
Top