Jim, how about data backup?

[Peter Majtan]

A) RAID 0 you have 30% more speed, not double.
RAID 1 you may have the same speed if there is a well designed controller.
Is in RAID 5 where you decrease the performance, because the calculation of parity bits. RAID 1 hasn't parity bits, is just a mirror.
and of course, it should be HW controlled because is an external drive connected to an eSATA bus.

Dude, dual drive RAID-0 gives You pretty close to double (maybe between 190~195%) rate of a single drive, not 130%...

RAID-1 can be HW, but it can also be SW (especially on mac). And I have clearly said that You would get almost the same speed (~100%) of a single drive.

cash test: DVD_taiyo, DV_Exellence, HDD_Seagate, BR_MCC
DVD - 0.10 $/GB
MiniDV - 0.25 $/GB
HDD - 0.13 $/GB
BR - 0.60 $/GB

Why people always think $$$$/GB as the only factor when it comes to ARCHIVE/BACKUP? By far the most important factor is longevity of media. How about:

DVD - 2 to 3 years at best
MiniDV - 12 years + (good tapes do 25+)
HDD - 5 years max, but drop it once and say goodbye to your data
BR - early to say, but would not expect more then 2 years

Also - with all optical media - scratch it and is almost the same as dropping HDD...

Tape - You can't scratch it by handling it, You can drop it from a roof of a building if You want and still be able to read the data...

THIS THREAD IS ABOUT MEDIUM to LONG-TERM ARCHIVE SOLUTION, not online, or near-line (short-term) ones...

 

[kmikami]

Yet they had (and have) the time to develop RED RAY...

Which is a 4k delivery and playback system that's totally unique, specific to the cinema market, and necessary for Red workflow. Nobody else was going to make it for them. Data backup is another issue.

 

[kmikami]

Why people always think $$$$/GB as the only factor when it comes to ARCHIVE/BACKUP? By far the most important factor is longevity of media.

You're the one asking for it cheaply! You said you want an indie solution that's under $500. So burn to multiple sets of DVDs or backup to multiple sets of hard drives and store them in separate locations. Verify the data once a year and replace any bad media. If you don't have a lot of money to put into it you're going to have to put a lot of sweat into it.

 

[Peter Majtan]

RED RAY looks likely to serve a different purpose. I don't think it's a backup soluton, and the investment in it's proprietary nature therefore is quite a different issue.

I have discounted RED REY as a backup solution right at the beginning of this thread. I was referring to You saying that "Worse than that, the solution would be entirely proprietary - there are no existing DV-data drives."
Meaning that RED RAY drive is also proprietary (as would be MiniDV-DATA drive), but unlike RED REY, Mini DV media is not... Therefore one could get the media anywhere (not just from RED)...

 

[MistahTibbs]

DVD - 2 to 3 years at best

Hey Peter,

That can't be correct.

Granted it's the corporation's perspective but Taiyo Yuden DVD-R media have an archival shelf life (when kept in appropriate conditions and that's pretty easy to do) of up to 50 years.

The downside is only 4.7 gigs per disc which means a TON of swapping while archiving but, still, for now, until larger media (blu-ray, etc) come out at viable prices for said archiving the DVD is a reasonable avenue to pursue for those on a budget. Hell, TY DVD-Rs are about .45 cents each.

It'll be tougher for those who shoot weddings and recitals d/t the amount of footage shot but for those doing shorts and films that shouldn't be an issue.

sincerely,


MT

 

[Allbird]

Dude, dual drive RAID-0 gives You pretty close to double (maybe between 190~195%) rate of a single drive, not 130%...

RAID-1 can be HW, but it can also be SW (especially on mac). And I have clearly said that You would get almost the same speed (~100%) of a single drive.

Why people always think $$$$/GB as the only factor when it comes to ARCHIVE/BACKUP? By far the most important factor is longevity of media. How about:

DVD - 2 to 3 years at best
MiniDV - 12 years + (good tapes do 25+)
HDD - 5 years max, but drop it once and say goodbye to your data
BR - early to say, but would not expect more then 2 years

miniDV tapes has 12 years of longegity, but you could have a drop out since the first second.

DVD has 3 years longevity over the shelf of my living room. in its best conditions for best brands (taiyo, not princo, or pringo [fu*ked], like we say in spanish), it should be 20 years.

I dont know how much lives a HDD. but for my, now, is, of course, the best solution.

This HDD has its own raid controller, & transfer across FW800.
This shelf show 7TB striped in RAID1. it means 3.5TB of data fully duplicated.
it means 28 hours of redcode36 raw.
an indi movie should not be more than 20 hour long of footage.

Then, the purchase of this shelf is now about $1300.

For me is not so much if you think that we are releasing our own 2 hours long pro movie.
If we want a cheaper solution, we have to wait 3 years until it cost $400, and remain working with a tape-based camera like sony or canon.

never before like now we were able to be part of the optical/digital cinema because the extreme cost of it.
If you want to be part of this revolution era of "ourselves" filmmakers you have to pay it.

 

[Dylan Reeve]

I've had a single SCSI hard drive (4GB I believe it was) running 24/7 for nearly 10 years before failure. Hard drives can and do have a good life-span, but that becomes something of a crap-shoot if they are left sitting unused an extensive period.

There's no reason that a RAID setup like the one above couldn't be kept in operation for many years without failure. Even if a drive fails, it can be replaced without data loss. This process could conceivable continue for as long as you're able to provide power and have a method of connectivity. Your biggest risk (and one of the drawbacks of HW vs. SW RAID) is controller failure, where the inability to replace it with an identical or compatible hardware controller can render the array unreadable.

Personally I think high-capacity data tapes (even with the prohibitive initial cost) is the best option. As demand for media data backup increases, it is increasingly likely that people will offer out-sourced services to do it. Something like 'you send me a SATA hard drive, I return it and a few LTO-4 tapes.'

And while LTO is very expensive (but also offers a fantastic capacity) there are other less expensive option. AIT drives are quite a lot cheaper (although offer lower capacity, still greater than Blu-Ray or DVD however). Also DLT and DAT/DDS tape drives are a possability.

 

[Peter Majtan]

How about remote locations?

Just imagine indie filmmaker shooting a doco for 3 months in Madagascar (like I did). Even if You could carry RAID storage with You, it will die on the first day from the vibration after driving on bumpy roads for the first hour. My back still hurts just remembering that... Even if it somehow managed to survive - how do You power it? A small portable digital tape recorder could easily run on batteries that You could recharge on solar-power, or even run it of solar directly. We had few Motorola's portable solar panels that charged our Iridium phones, radios and all camera gear...

Any optical media in this situation will also have trouble not getting scratched...

Digital tape is the best option by far and Sycophant is right there. However they all require special tapes that might be hard to come by in a remote regions. Of course some could argue You can stock up before Your trip, but You always end up shooting more then You thought thanks to Murphy...

I still doubt Allbird that You can buy 7TB with RAID controller for $ 1300,- USD, but HDD's will get there (and beyond) soon enough. Issues there are high power consumption and associated maintenance.

The Taiyo Yuden DVD-R are impossible to come by in Mexico, especially in our region. Heck - we even can't get hold of the Verbatim's Gold Archival DVD's here without ordering large quantities from USA...

And lastly - I am not trying to get things "cheap" for me. After all we have in our companies several 4Gb fiber RAID enclosures in 19" racks exceeding dozens TBs...

But I have not forgotten about the indies on extreme tight budgets (that I was once myself) and Scarlet is clearly targeted to them...

Few of my guys loves to do indie projects in their spare time (which I don't give them much :innocent: ) and I fully support them. They are all over their heads right now having to wait so long for Scarlet to come out...

 

[Allbird]

hi peter.

u are right with remote locations. i still remember a work i made in Menorca that became a simple holidays when i came & discovered that my hdd becomes to brick! i think it wasn't vibration, i thought about hot.

i was thinking about normal conditions for a short-movies or pictures in standard locations. sure that LTO is the best solution, but you need a deck.

yes, the WD HDD cost is here, in europe, €433. 2TB.
last month was €480. do u understand? if we give again the calculator:

1TB (2TB mirrored) = 8 hours of REDCODE36

100GB = 48min

1GB ≈ 30seg

or

€433 / 1024GB = 0.4 €/GB ≈ 1€/1 minute of redcode36

here are the calculations for other formats:

DVD ≈ $0.20/min
miniDV ≈ $1.5/min
BR ≈ $1.4/min

in the HDD solution, u has more speed than optical formats.
in the HDD solution, u has direct access from any computer w/o decks.
in the HDD solution, u has a fully duplicated backup, where the others don't.

yes peter, for u the price is not so important, but for me is.
and i am agree to develop another solution for remote locations.

thanks peter for this post, this question always make me eating my head!

_____
EDIT
------

Originally Posted by Sycophant
There's no reason that a RAID setup like the one above couldn't be kept in operation for many years without failure. Even if a drive fails, it can be replaced without data loss. This process could conceivable continue for as long as you're able to provide power and have a method of connectivity. Your biggest risk (and one of the drawbacks of HW vs. SW RAID) is controller failure, where the inability to replace it with an identical or compatible hardware controller can render the array unreadable.

RAID1 controller only works on write.
in our situation, once done the back up, we don't need the controller any more.
if u want, u has two HDD separately with the same data.

 

[Dylan Reeve]

RAID1 controller only works on write.
in our situation, once done the back up, we don't need the controller any more.
if u want, u has two HDD separately with the same data.

I don't believe that is correct. The drives are not stand-alone drives. They are part of a RAID set. They can't simply be removed and used independently as a standard drive. While it may be easier to access data from a RAID1 set with a different RAID controller than with other RAID setups, it still requires the RAID controller.

 

[Peter Majtan]

hi peter.thanks peter for this post, this question always make me eating my head!

Thanks Allbird, I think this is the question that is by far the most underestimated in our industry.

When I was developing the restoration system in Japan I had the pleasure to work with the actual original negative shot by Akira Kurosawa. Even after so many decades and despite the film being greatly damaged - I was able to see the "data" (image) and have enough information to restore it successfully.

I just can imagine any of the current formats going through what this film was and being able to recover anything...

I have developed an optical system to record digital data using film recorder onto archival Kodak stock that could easily survive hundreds (yes - there is "s" at the end of the word!) of years and protect the data. This is by far the best option right now. However is it way to expensive and only suitable for material of high cultural or other value...

What is missing is truly portable and affordable solution that could archive to readily available and also affordable media with reasonable shelf life (10~25 years), that is not prone to easy damage...

DVD's fill that quite well except for the part of resisting damage...
Ditto for HDD's...
LTO, DLT's and other tapes are not too affordable and certainly not readily available...

SSM (solid state media) could be the answer when they drop in price and rise in capacity. But no one knows yet what happens to data on SSM after it has been in the archive "box" for 10 years...