Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Cheap eSATA RAID solution

Rick Darge

Rick Darge

Banned
Joined
Dec 29, 2006
Messages
3,775
Reaction score
0
Points
0
Age
44
So this is what I'm thinking about doing...

Buying the 4bay Firmtek + 4host eSATA PCI-X card $650

http://firmtek.com/seritek/seritek-2eEN4/

Going to newegg and picking up 4 500GB SATA drives at $140 a pop, $560 for 4

http://www.newegg.com/Product/Product.asp?Item=N82E16822144016


Then I would stripe these 4 together as a Raid 5 - Roughly 1.5 Terabytes of space..

1.5TB = 15 HOURS of RAW 4K if I'm not mistaken

Nice thing about this setup is after I fill up 1.5 TB, I can shelve all 4 drives and drop another $560 for 4 more drives + the firmtek cages at $22 each, making the price roughly $650 for an extra easily swappable 1.5 TB of space.


Crap, this is going to get expensive fast shooting 4k

1.5tb=15hours= $43 per hour = $0.72 per minute +

Still cheaper than film :alien:
 
rgdfilmsRED said:
So this is what I'm thinking about doing...

Buying the 4bay Firmtek + 4host eSATA PCI-X card $650
Click to expand...

OK, sounds good. Why PCI-X? Considering REDCINE won't run on non-Intel Mac systems and will require a more current PC... Most everything has gone PCI-Express these days.

The 500GB Maxtors are good drives... So are the Seagate and Hitachi units. All perform about the same. Can't buy from a better place than NewEgg either... I love those guys.

RAID5 would be the way to go for reliability. I would consider a straight RAID-0 array for performance though if you plan to do any editing from this array. However, you mention shelving the drives for storage. IMO this is a bad idea. ...I know we've been over this in other threads, but hard drives (even if they are components of a RAID5 configuration) are not designed to sit idle on a shelf for any extended period of time.

Crap, this is going to get expensive fast shooting 4k
Click to expand...

This is the other reason swapping out those drives for archival isn't a great idea. LTO-3, DLTS4, and other large format tape solutions are far more reliable for archival and are intended for that very purpose. They are also cheaper than hard drives by a good margin. The tape unit itself is expensive, but once it's cost is recovered, it's far more economical. If I had to buy an archival solution today, I would go for the Exabyte or Dell (ADIC) autoloader units. I can buy brand new Exabyte 12-tape LTO-3 autoloader systems off ebay for less than $3K. Figuring $3500 for the tape system with SCSI interface and cables and $0.15/GB tapes... If you compare that to hard drives ($0.33/GB plus swappable enclosure, interface, gables - $650). The tape system makes a lot of sense.

Compare 20TB of backup capacity -- why I chose 20TB will be immediately clear.

LTO-3: $3500 + $3000 (bulk buy of 50 tapes) = $6500

HDD: 40 * 500G drives = $6500

So the break-even point between the two solutions is about 20TB at about $6500. Lets look at the cost for the next 10TB. Oh, and we'll assume that the price of 500GB HDDs has dropped another 25% in the mean time.

LTO-3: $1500

HDD: $2435

With the amounts of data we'll be working with in 4K, 20TB isn't going to be archiving that much considering original "negatives" or RAW files, plus intermediate data, projects, finals, etc.. 1TB of drive storage is roughly 10.5 hours of REDCODE RAW 4K @ 27MB/s. The first 20TB (210 hours) of archived footage equates to about $31 per hour. From that point forward, LTO3 or DLTS4 would equate to about $7.15/hour.

We also have new tape formats coming up like DLTS5 and LTO-4 which will double capacities and keep tape prices roughly the same. So expect tape offerings to go from $0.15/GB to aout $0.09/GB within the next few months.
 
well you can also consider that the LTO-3 tapes occupy less physical space and have less weight when you have to ship them.
 
http://firmtek.com/seritek/seritek-1en2/

I got 2 of these FirmTek SeriTek/1EN2 Dual-Bay Hot-Swap External SATA Enclosures a while ago. My PCI-X card is a Sonnet Tempo 4+4. I use Seagate Barracuda 7200.8 400 GB drives. I would not buy this same setup knowing that Redcine won't run on my G5. I will wait until the new dual quad-core intel Mac Pro's are available and get a PCI Express card with RAID array.

I brought this up on the old dvxuser RED forum:

If the RED-RAID (20-40 drive mini-fridge) had drives that were user-replaceable (doesn't need to be hot-swappable), you could use it as your desktop RAID. When you need to use the RAID for 4k 60 fps or 2k 120 fps, you could just swap out the drives for another set. Just make sure you put the drives back the same order.

I hope the Holographic storage comes down in price quickly. It is supposed to have a long shelf-life, not to mention random-access, high throughput, and high-capacity.
 
1.) You'll actually get less than that - 1387GB formatted capacity

2.) Nowhere NEAR fast enough for uncompressed RAW

3.) It makes no sense for you to work in uncompressed RAW if you're on that tight a budget - you're shooting/transferring compressed, so roughly 100GB/hr, and THAT would be a datarate you could do.

4.) Are you Mac or PC? If you're Mac, you have no RAID 5 options with that card. There's no software based RAID 5 for Macs.

5.) Yeah - gotta be on a PCIe based Intel Mac system for Redcine

6.) and if you had 4K footage, how would you monitor and output it? Shoot 4K compressed RAW, then downconvert to HD for lovely results that you could potentially output and watch in a meaningful way. But your 4 drive RAID still too slow for that, probably even at RAID 0.

Not meaning to sound discouraging, just realistic.

-mike
 
  • Thread starter
  • Banned
  • #6
I should have been more clear. I'm not looking to edit in 4k by any means, even when I get my 8-core 8GB Intel-Mac at the end of the summer. I'm just looking for a practical solution of where to store all this data coming in. That's why I was thinking temporary storage would be on a RAID 5 1.38 TB - When that filled up after projects were finished, I would archive to tape or something else. The more I think about it though, the more this whole post idea scares me. we're talking HUGE amounts of data

What's your setup like or going to be like after you get the RED Mike?
 
I'm looking at this:

http://www.macgurus.com/productpages/firewire/Burly1FirewireHS.php

or similar for backup. We'll be buying a few of these and lots of $20 sleds. HDs are about $100 for 250gig and $200 for 500gig.

I was ready to jump on the LTO path. But after talking to a few people who have been using these and researching the software required to run the hardware I realized that even though the capacity has grown, the limitations of a linear storage medium have not. These solutions are not much better than they were 10 years ago when we used DLT to backup our AVID.

A hard drive costs a little more per gigabyte, but when a client calls and asks[demands] that they want to revise that project you did last year and they need you to send a low res quicktime for them to review now. Do you really want to load up an LTO tape[or what if you striped a project across 4 tapes in an autoloader?], run Retrospect and spend an hour or more resurecting an entire project?
Wouldn't it be faster to pull a hard drive off a shelf, plug it into a hot swap drive bay and access that file immediately?
There are also a few good disc catalogging programs that are shareware or freeware that do a good job of keeping track of which hard drive I need to pull down off that shelf. We've been using one of these with DVDRs for years.
 
Sean Michael 808 said:
I was ready to jump on the LTO path. But after talking to a few people who have been using these and researching the software required to run the hardware I realized that even though the capacity has grown, the limitations of a linear storage medium have not. These solutions are not much better than they were 10 years ago when we used DLT to backup our AVID.
Click to expand...

Well, there's a few things about tape that are being ignored here. Autoloaders don't run striped volumes. They're spanned from one tape to the next. Firewire isn't all that fast and individual hard drives aren't going to be much (if any) faster than reading tape. If you have proper backup procedures in place, you can pull the proper tape (or tapes) from your archives and load them just as fast as you could from individual hard drives. Sure, tapes are linear, but a proper tape system with good backup software will tell you which tape you need and it will fast forward right to the data it needs.

I know that a lot of people have a dislike of tape systems... I know several IT guys that detest tape just because it's tape. Their opinions have nothing to do with how reliable or practical of a solution it is and they continuously go on and on about how other storage media is far superior. I've got a buddy that preaches holographic storage as if it's going to save the world. Fine, great, sounds good... Where can I buy it? But there's a real flaw with that logic. Other than hard drives, no other media provides equivalent capacity and nothing touches tape in terms of TCO and ROI numbers when you start considering large volumes of data. Personally, I would feel very comfortable if I have to pull data from an LTO-3 tape 5 years from now. I would not feel so good about pulling data from a 500GB hard drive that has been boxed up for that long. And in a year or two after I have been using my RED One and I've got over 100TB of archived data. I know that I'll be happy I don't have $30K invested in hard drives, but rather $12K in a format that is designed from the ground up to sit on shelves with fewer moving parts and less chance of magnetic dispersion and corruption of the data itself, which tends to happen over extended periods of time on hard drive platters. Not to mention the motors, internal lubricants, head mechanisms, etc.. that have just been sitting there for 5 years or more... Before the next great backup format comes along (holographic?) and I can revise and condense all the tapes to something more efficeint and hopefully something more reliable.

I do consider tape to be more reliable than HDD... I've been using both for years and have managed large SANs and helped run datacenters. While I love the idea of hammering that final nail into the proverbial coffin for all tape formats, that day just isn't here yet. When it comes to long-term archivial of tens and hundreds of terrabytes, even petabytes and zetabytes of data, the world still operates on tape. Large companies have various levels of backup solutions and ways to get at their data. But some of the biggest out there, like the library of congress for example, still uses tape as their final archival step and will do so for the forseeable future.

For people planning to shoot a lot of video with RED (200+ hours per year), they would be best served to have a SAN in place for short term backup, project workspace, etc.. Archival to a common tape format, perhaps consider secondary backups of the really important stuff to additional tapes and/or smaller projects to optical media as well... If the price of optical media ever comes down. Right now a 50GB BD+RE disc is the same price as a 400GB LTO-3 tape.
 
Thanks Jeff, do you have an LTO system in house now?
I'm still on the fence about which solution we're going with. We'll probably be living with it for a long time. Any user feedback is much appreciated.
 
No LTO system here now, but I have used them and some of the other sytsems like DLT-S3. Currently everything here is on RAID-5 storage accessible via SAN. My SAN consists of an XServe RAID and two other PC nodes each with a RAID-5 array. I'm using a VXA-2 tape system for archival (160GB uncompressed per tape) for archival of my animation and video - mostly DVCPROHD via HVX200. A single slot VXA-2 drive isn't going to cut it for RED and its data requirements and I plan to buy something new when my RED ships. I'm planning on either LTO-3 or DLT-S4, but hopefully LTO-4 or DLT-S5 will be available later this year in time for my RED. As for workstations, I have a couple G5 quads, a C2D Macbook Pro and a few PCs. My latest PC is an 8-core 2.33GHz system I just built about 6 weeks ago. I plan to buy a new 8-core Mac Pro and retire one of the G5 quads to render farm duty sometime later this year. Most of my workstations are configured with a system drive and two additional drives in a RAID-0 stripe for workspace, etc.. My primary G5 quad just has two 500GB drives in a RAID-0 stripe as its primary and only HDD volume. I keep a few external fw/usb2 drives handy as well.
 
Good posts, Jeff. I've wondered about the economics of tape vs. hard drive. For personal backup and long-term storage, I'm going with cheap hard drives all the way. But your math on the tapes (assuming it's right) is pretty informative.

You bring up some things you worry about with hard drives, but without studying the matter in detail over time, I have alternate worries about tape.

1. The degradation of the binder that holds the oxide to the tape, compounded by tape's flexible nature and its travel over curved surfaces. Audio tapes made during a certain period later started to shriek when played back because of (the absorption of or loss of) moisture in the binder. It was probably absorption, because the only way to recover the recording was to bake the tapes at low temperature for hours or days, and then you had one opportunity to play the thing out and copy it. We have better compounds today, but the concept inspires concern.

2. Exposure to magnetic fields. Hard drives (even today) are housed in stout metal enclosures. Maybe these tape cartridges are shielded, but if not, I'd worry about magnetism more than I would with a hard drive.

3. Physical damage, like stretching.

In a tangent point, someone mentioned products like this:
http://www.macgurus.com/productpages/firewire/Burly1FirewireHS.php

This product suffers from a glaring design omission that is rampant and aggravating in this product arena: no eSATA connector! I'm using SATA drives for a reason! Given that my computer has SATA, the drives are SATA, and this standard is likely to grow in popularity (despite the baffling failure to include POWER in the connection standard)... why would you omit a SATA connector from the enclosure?

I've been shopping around for a long-term strategy for hard-drive-based archiving, and vendors' failure to recognize the importance of having all three connectors (eSATA, Firewire, and USB) on any SATA-drive enclosure kills lots of otherwise promising products.
 
that looks pretty nifty chas - haven't used it, I use the firmteks currently and love them
 
Stokestack said:
I've been shopping around for a long-term strategy for hard-drive-based archiving, and vendors' failure to recognize the importance of having all three connectors (eSATA, Firewire, and USB) on any SATA-drive enclosure kills lots of otherwise promising products.
Click to expand...

Why you will not consider Blue-ray Disc as a good long term archive.
At this moment double sided BD has 100GBytes of capacity. Next year it will be 400GB
I agree, it requires segmenting, but you have to load it in to the computer for processing any way so I bet you can load 800GB data from 8 BD faster then from the tape?
 
The dual layer BD is 50GB the two sided dual layer is 100GB
Don’t mix dual layer with two sides, normally you have label on the second side but for archive purposes you record on both sides, put 4 or 8 of them in the box and you label the box. Don’t forget to put the discs numbers on the inner, unrecorded part of BDs since you do not have room for label anymore on two sided. Also make sure that you include label in the title on the disc data.

http://en.wikipedia.org/wiki/Blu-ray_Disc

http://www.supermediastore.com/blue-ray-blu-ray-dvd-media-center.html
http://www.tdk.com/procommon/press/article.asp?site=con&recid=140
 
Andrew said:
Why you will not consider Blue-ray Disc as a good long term archive.
At this moment double sided BD has 100GBytes of capacity. Next year it will be 400GB
I agree, it requires segmenting, but you have to load it in to the computer for processing any way so I bet you can load 800GB data from 8 BD faster then from the tape?
Click to expand...

The problem with BluRay as a long term archival solution (right now) is that cost per GB is very high. Additionally, the shelf life or reliability of BD+RE media has not been proven. Even up to a few years ago, we were hearing all these claims about optical media having a shelf-life of 100+ years. However, many are now discovering that most common CD-R media that made such claims 12+ years ago are now failing. Most manufacturers have changed their tune about shelf-life and only claim 25-30 years with current CD/DVD media. Sony claims 25 year shelf-life with their current BD media.

Realistically, even a long-term archival solution only needs to last 8 to 10 years. This allows adequate time for data storage solutions to increase several times over and archived data collections can be re-organized and consolidated onto more current media.

Anyway, back on the subject of BluRay... It may make sense as an archival solution in a couple years, right now it doesn't. And I'm not sure what you're saying about 400GB. Not going to happen. The BluRay spec can only support up to 12 layers per disc. That's just how many they can physically pack in there and even the Sony reps I've talked to at various presentations expressed doubts (based on what their tech guys have told them) that BD will actually reach 12 layers and still be readable from one side of the disc. Dual-sided, 6 layers per side should be do-able. But either way, BD media tops out at 300GB, which translates into 291GB of real-world storage.

Holographic discs will probably replace BD media in a few years anyway. Current low-density proof of concept devices pack 800GB on a standard CD/DVD size disc without using multiple layers. But we're still at least a year away from being able to buy a holo-writer.

But to revisit the cost per GB for BD media... Currently there's 100GB discs (dual layers on each side of the disc). But I can buy a 400GB LTO3 tape for the same price. It's actually cheaper to just buy the 50GB single sided BD discs as they're under $30 now in bulk.
 
You must log in or register to reply here.
Back
Top