Raid 1 for on-set data

[vladk]

Hi everyone, this is going to be my first post here - I figured I'd ask the community. I've been DITing red shoots for a little while now the classical way (fw800/esata hd's, copying, mdsumchecks, etc).

I am considering changing the strategy by getting a 2-bay quad-interface raid enclosure for DIT work. The plan is to get the production to buy 2 internal drives, and put them into the enclosure. Set it up for Raid 1 (mirror) and not have to deal with copying twice, since the setup should do it by itself.

I see the following advantages:
1) saves some time on set for the copying process
2) saves the production money on hard drives by going internal

Disadvantages:
1) production will need another enclosure when they want to access the files if they don't have a desktop.

I wanted to ask if anyone's done this and if there are any obvious downsides/dangers to this kind of approach.

 

[James Brundige]

I use fw800 dual enclosures, with raid 1 mirror. Works great.

I shoot docs, and archive this way every night (or every time there is electricity).

After shooting, I copy these files to my edit system, which I also back up. Then there are 4 copies of each shot. Should be enough. Looking forward to red ray or new high density disk to eventually replace the dual drives.

 

[Mark L. Pederson]

Hi everyone, this is going to be my first post here - I figured I'd ask the community. I've been DITing red shoots for a little while now the classical way (fw800/esata hd's, copying, mdsumchecks, etc).

I am considering changing the strategy by getting a 2-bay quad-interface raid enclosure for DIT work. The plan is to get the production to buy 2 internal drives, and put them into the enclosure. Set it up for Raid 1 (mirror) and not have to deal with copying twice, since the setup should do it by itself.

I see the following advantages:
1) saves some time on set for the copying process
2) saves the production money on hard drives by going internal

Disadvantages:
1) production will need another enclosure when they want to access the files if they don't have a desktop.

I wanted to ask if anyone's done this and if there are any obvious downsides/dangers to this kind of approach.

Welcome vladk!

We do tons of jobs this way.

We use TrayFree RTX Series enclosures from Wiebetech (RTX100H-Q) -

http://www.wiebetech.com/products/RTX100.php

They also sell hard cases to pop your SATA drives into -
http://www.wiebetech.com/products/cases.php

We actually go against the "norm" by doing TWO SEPARATE file copies as opposed to Raid 1 - but if you are checksuming you should be fine with Raid 1 - we had SEVERAL (leading brand) HARDWARE RAID 1 enclosures fail us - so we don't play that anymore -

If you are MAC and you are going to do a software raid one - YOU WANT THIS -

http://www.softraid.com/

 

[vladk]

Welcome vladk!

We do tons of jobs this way.

We use TrayFree RTX Series enclosures from Wiebetech (RTX100H-Q) -

http://www.wiebetech.com/products/RTX100.php

They also sell hard cases to pop your SATA drives into -
http://www.wiebetech.com/products/cases.php

We actually go against the "norm" by doing TWO SEPARATE file copies as opposed to Raid 1 - but if you are checksuming you should be fine with Raid 1 - we had SEVERAL (leading brand) HARDWARE RAID 1 enclosures fail us - so we don't play that anymore -

If you are MAC and you are going to do a software raid one - YOU WANT THIS -

http://www.softraid.com/

It's unfortunate to hear about hardware raid boxes failing. I was really hoping that would be a solid solution, but I might just have to get one and stress-test it to make sure. Could you maybe plop down some names of brands that failed? And failed how? Was it along the lines of the controller breaking and it stops mounting, but the hard drives are okay? Or did they corrupt hd's, etc?

 

[Mark L. Pederson]

It's unfortunate to hear about hardware raid boxes failing. I was really hoping that would be a solid solution, but I might just have to get one and stress-test it to make sure. Could you maybe plop down some names of brands that failed? And failed how? Was it along the lines of the controller breaking and it stops mounting, but the hard drives are okay? Or did they corrupt hd's, etc?

I don't want to name the brands that failed - because they were extremely cool and gave us prompt refund and they felt very, very bad. Hard to know if this these were very "isolated" incidents.

We had things like - raid unit suddenly, for no reason, start re-building itself.

On another unit, from another brand, we did in fact get corrupt files.

Now, I know TONS of people that use these things successfully everyday. But, I try very hard to not make the SAME mistakes over again. And you know ... once bitten ... twice shy.

If I WAS going to use one - it would be G-Techs G-SAFE - they use quality components - and we've used all their stuff for six years - http://www.g-technology.com/products/G-SAFE.cfm

The great thing about internal SATA drives is you can swap sizes, etc. -

On a run and gun stuff we use the Sonnet F2 which is awesome - http://www.sonnettech.com/product/fusionf2.html

 

[Conrad Hunziker]

I use fw800 dual enclosures, with raid 1 mirror. Works great.
{snip}
After shooting, I copy these files to my edit system, which I also back up. Then there are 4 copies of each shot. Should be enough.

I dont mean to point fingers here, but I do want to put down a misnomer that some people are making with RAID 1. People are counting both drives as individual copies in a RAID 1 system. Thats not true.

Since they are both on the same filesystem (i.e. mountpoint), they should be considered as one copy, with redundancy. If a hard drive fails, you still have a vaild media - however if the file table fails or you accidentally erase an enclosing folder, both "copies" are gone.

I am well aware that there are ways to get the information back if the table of contents are lost, or the trash doesnt mean the data is lost. But if you have to get to that point where you need to do serious data recovery, then your backups are not good enough.

To be clear, Im not pointing fingers that First Light should make more backups or somehow their system is flawed, but I just want to point out that each filesystem copy should be counted as a copy, not each hard drive copy.

 

[Conrad Hunziker]

I wanted to ask if anyone's done this and if there are any obvious downsides/dangers to this kind of approach.

If you go this route, you really should have the production BUY the enclosure and drives to go with it. The biggest disadvantage is that not all RAID 1 systems are setup the same. You cant just take a pair of drives in RAID 1 (assuming a 2 drive setup) and take it from enclosure to enclosure and expect it to always work the same. Different manufactures write their framework in different ways on the head of each drive. So a RAID 1 setup on one system may not be recognized in another.

In addition, if you are going from a hardware (external) to a software (internal) setup, there is the possibility that during the configure you wipe the contents of the drives.

In short - you want the least possibility of human error - so just have the production buy the enclosures also.

 

[Mark L. Pederson]

I dont mean to point fingers here, but I do want to put down a misnomer that some people are making with RAID 1. People are counting both drives as individual copies in a RAID 1 system. Thats not true.

Since they are both on the same filesystem (i.e. mountpoint), they should be considered as one copy, with redundancy. If a hard drive fails, you still have a vaild media - however if the file table fails or you accidentally erase an enclosing folder, both "copies" are gone.

I am well aware that there are ways to get the information back if the table of contents are lost, or the trash doesnt mean the data is lost. But if you have to get to that point where you need to do serious data recovery, then your backups are not good enough.

To be clear, Im not pointing fingers that First Light should make more backups or somehow their system is flawed, but I just want to point out that each filesystem copy should be counted as a copy, not each hard drive copy.

This is really critical to understand and fundamentally why we choose to make two separate copies, on two separates drives, not in a raid 1 config.

 

[David Didato]

The biggest disadvantage is that not all RAID 1 systems are setup the same. You cant just take a pair of drives in RAID 1 (assuming a 2 drive setup) and take it from enclosure to enclosure and expect it to always work the same.

We use the same Wiebetech RTX enclosures as mentioned above, which are great. And the caveat is, if you set up RAID 1 with firewire, and then try to mount the same drives using USB or SATA it will fail, and potentially crash your system.

D

 

[Mark L. Pederson]

We use the same Wiebetech RTX enclosures as mentioned above, which are great. And the caveat is, if you set up RAID 1 with firewire, and then try to mount the same drives using USB or SATA it will fail, and potentially crash your system.

D

or ... you can stop all this Raid 1 obsession and make two copies on two separate volumes.

 

[Conrad Hunziker]

Forgot to mention, all the systems Ive setup, either consulting for other red shows, or shows Ive setup, have had the following:

- One on-set RAID. Sometimes RAID 1, but usually RAID 5. This is a temporary master copy of everything.

- One set of shuttle drives. These are usually bare drives in an enclosure, that is sent to post. At least 3 of these floating around between set and post.

- Second set of shuttle drives. Same concept as above, except for these go to archival or to a director or whatever. The point being that this is in a duplicate copy of everything in a geographically different location. (Not all productions have the budget to do this step)

And all copies are always checksummed. So there is always at least 2 verified copies, on 2 separate filesystems, and in 2 geographically different locations. The On-set RAID really helps for continuity and checking previous scenes, as that usually will hold several days back in time, if not weeks (depending on the raid size and amount of data shot).

Its also important to note that I never consider, and I try to impress upon all the shows that I work on, that the on-set raid is the master. It is NOT the master. The master copy of all the footage should be on some environmentally controlled, stable, RAID system in a place far far away from the set. We all know how "stable" the set is.

 

[Michael Thornton]

I use fw800 dual enclosures, with raid 1 mirror. Works great.

"To Err is Human."

Working long hours will make this a costly mistake.

you can stop all this Raid 1 obsession and make two copies on two separate volumes.

I agree with Offhollywood totally.

Making the copy twice is actually more secured than once.

Human will make mistake if there is no check and balance.

I will fire any media manager who bring a single raid 1 set on set.

If he bring two, that is better. Still do the copy twice.

Tek

 

[Mark L. Pederson]

The master copy of all the footage should be on some environmentally controlled, stable, RAID system in a place far far away from the set. We all know how "stable" the set is.

Wise words.

 

[Jeremy Newmark]

In my opinion two copies is definitely the way to go, not RAID 1. We've been using two single bay eSATA enclosures, on location, from MacGurus

http://www.macgurus.com/productpages/sata/HotswapEnclosures.php

On set we go from the Red media to two separate eSATA drives using a dual port express34 card on a MacbookPro. We have extra trays so you can easily swap out drives when they are full. We use R3D manager to make the two separate copies with checksums. The drives are then split up until they can be backed up to LTO at our studio. With the price of internal hard drives these days, you can build a rather large mobile system for a very reasonable price.

 

[Trevor Meier]

- One on-set RAID. Sometimes RAID 1, but usually RAID 5. This is a temporary master copy of everything.

- One set of shuttle drives. These are usually bare drives in an enclosure, that is sent to post. At least 3 of these floating around between set and post.

- Second set of shuttle drives. Same concept as above, except for these go to archival or to a director or whatever. The point being that this is in a duplicate copy of everything in a geographically different location. (Not all productions have the budget to do this step)

And all copies are always checksummed.

We do a similar setup, using Photocon's software actually (R3D Manager). Only difference is several shows use a Quantum LTO-3A drive to do an on-set backup. So we end up with an on-set RAID, LTO-3 copy sent to the Production office, and a transport SATA drive sent to Post which copies onto a RAID-5 as the master. Three copies, three formats, three locations. And the Red mags aren't reused until the checksum copy is confirmed back at post.

If you're doing a bonded show... it's the minimum they'll be happy with.

The LTO-3A copy could happen at post... speaking of which - Photocon, how's the FTP support coming in R3D Manager??

 

[tj williams]

HI All Mark Pedersen, great insights Thanks..

We have been using outside Dits on shoots so far. Now single person going out of country recording to two sep. E sata drives. What is the current best method to check sum? is there a markedly superior utility available for this on the PC?

 

[Conrad Hunziker]

The LTO-3A copy could happen at post... speaking of which - Photocon, how's the FTP support coming in R3D Manager??

Dreadfully slow, sorry to say. We are still working on the source queue and then next is the copy to/from ram.

 

[Conrad Hunziker]

HI All Mark Pedersen, great insights Thanks..

We have been using outside Dits on shoots so far. Now single person going out of country recording to two sep. E sata drives. What is the current best method to check sum? is there a markedly superior utility available for this on the PC?

We should have a PC version of the R3D Data Manager soon. I have a alpha copy in my hands now.

 

[jon erwin]

We do one raid 1 drive and then mirror it at the end of every day for off site backup. Works really well! Lacie make a really cheap, really good 1TB FW800 that allows swapping drives.

 

[Chris Gearhart]

For smaller jobs, we use an external RAID 1 configuration alone. For larger clients, two copies onto two external RAID 1's. Fairly soon we are going to make every job conform to the latter data flow. It's just too scary making one copy on ANY shoot.

This way the client can also take one of the RAIDs with them, if their deliverable needs warrant it.