Drive duplication in field...

[Matt Redmond]

I have been doing my head in... :waaa:

Looking for a good solution for taking data from the Red Drive (MASTER) out to two duplicated drives - One being the WORKING COPY, the other the BACK-UP. The client would then take either one or both copies depending on their situation.

The two duplicated drives could theoretically be in a hardware RAID 1. Two SATA drives in removable caddies for easy insertion and removal.

The issue comes with the finding the best way to get the data from the Red Drive to the RAID 1 (or other).

?Connect the Red Drive via 1394b to a laptop, then copy out an eSata adaptor to the RAID?

Did it copy across right? This footage IS your project...would it not be a good idea to verify your copy? Is it worth the wait to confirm the copy?

Another thing too - if you're going via laptop in field then this is drag and drop stuff and human error territory. Something like p2genie would be marvellous where you connect your drive, software detects it, press "copy," it copies, it says it has copied, it verifies, then DELETES the footage off the Red Drive should you request it. I want REDGenie damn it.

The best solution for us would be a separate hardware duplicator. There are devices which only copy over the used sectors of the drive - they even verify and can spit out to multiple drives at once. But finding one where you can just plug in your firewire 800 drive is a bit more difficult. Also, does the fact that the Red Drive sits in RAID 0 interfere with that process? I'm still waiting to hear back on the SOLO III to see if its firewire port can support something like the Red Drive (forgot to ask about the Raid) - the documentation seems a bit shady on the matter of connecting firewire drives directly to the unit, as opposed to using it just dock to an external computer...

Then you also have to take care of the Compact Flash option...fun fun fun.

If I've missed something, then fill me in. All in the aid of discovering how to make life for our customers as simple as clicking the GO DO button.

 

[Mark L. Pederson]

I can't imagine RED isn't going to do a software utility and/or a hardware clone and verify RAID 1 drive unit.

I suspect there are LOTS of things RED is going to do fanastically AFTER they ship the camera.

There are some solutions available already that will do 90% of what you want and Offhollywood will be testing the best options the moment we get our cameras - but you are not missing anything _MR_ - you are right on the money.

From RED or another 3rd party - you'll have you want for sure.

 

[Gavin Greenwalt]

You could also look for something like this:
http://www.logicube.com/products/hd_duplication/omniclone2u.asp

Which connects through a USB port and just use a straight up drive sector cloner.

 

[Mark L. Pederson]

You could also look for something like this:
http://www.logicube.com/products/hd_duplication/omniclone2u.asp

Which connects through a USB port and just use a straight up drive sector cloner.

LOL! Pretty cool ... if you are in the CIA.

That's hitting a small nail with a big hammer.

 

[Gavin Greenwalt]

Well then how about this:

http://tmcscsi.com/diskjockey.shtml

handheld. Portable stand alone unit. Sector level duplication. If the RedDrive has a sata connection it will take care of all the duplication without any need for a laptop.

 

[Scott Webster]

Well then how about this:

http://tmcscsi.com/diskjockey.shtml

handheld. Portable stand alone unit. Sector level duplication. If the RedDrive has a sata connection it will take care of all the duplication without any need for a laptop.

The problem there is the Red Drive has a custom Lemo connection which also provides power to the drive. It does not have an e-sata connection port for transfer. This must be done via bus powered USB or FW.

See Jeffs post here

 

[Gavin Greenwalt]

Well that little gizmo has both firewire and USB ports on it. Problem solved!

 

[luminoir]

There's a similar thread to this going on in the CML forums
so I'll reproduce my post there, with amendments made for relevance.

What about an incremental RAID1, battery operated solution that is tethered to the Red Drive or any of the high speed data interfaces on the camera.

Whenever it's plugged in, it's backing up the contents of the drive whenever the drive is not being accessed by the camera during acquisition.

This is akin to background rendering in NLEs or batteries charging, just making optimal use of spare cycles/downtime.

If you need to run untethered, the backup truncates the incomplete file,
deletes it and resumes from that point when the connection is restored.

For visual verification, it comes with an on-board monitor and video out port with a GUI to parse footage already backed up.

This can also serve as a secondary monitor/ review tool. Additionally, it does not destructively delete files off the Red Drive.
That should be a conscious decision made by the operator, not arbitrarily executed by any hardware/software.

Hence, the review of footage takes place on the field backup unit/monitor and when the Red Drive due to be cleared for space, you can check which of the files existing on the drive are already backed up. (possibly a flag to indicate this?) and delete those because there are 2 copies of the copied files on the RAID1 array of the field recorder.

Because it's incrementally backing up throughout the day during downtime (in between takes, makeup, waiting for sun, adjusting lights, blocking, dry runs) that also means less time waiting after the last shot for the shoot to be backed up.

Dustin Lau
Director/Editor
Singapore

 

[Gavin Greenwalt]

I just want to try and parrot what you said to see if I fully understand your idea.

So the RED Camera is tethered to a RAID while filming. Whenever the camera cuts it begins offloading footage to the RAID from its local storage system (Flash/RAM/HDD). If the operator hits record it stops streaming data to the storage system and begins recording again. As soon as it stops recording it resumes streaming where it left off.

If the camera needs to go untethered it stops streaming until it is reconnected.

If additional space is needed the operator can ask the program to confirm it's downloaded all of the shots and that they are indeed intact. Once confirmed all securely backed up files can be deleted off the Camera storage.

Is that correct?

It's an interesting idea, however I believe you would see some pretty significant lag in the transition from the drives being unmounted from the Camera. Mounted on the PC. Begin transfer. Stop transfer. Unmount. Remount to camera. Begin recording. I suppose you could add a GigE connection and share the drive over IP.

 

[Jeff Kilgroe]

There's a similar thread to this going on in the CML forums
so I'll reproduce my post there, with amendments made for relevance.

What about an incremental RAID1, battery operated solution that is tethered to the Red Drive or any of the high speed data interfaces on the camera.

This seems to me like it's being made far more complicated (from a workflow point of view) than it needs to be.

~34MB/s is needed for 4K REDCODE @ 30fps. ...That's roughly 272Mbps. I can see an application there for dual-band 802.11n and no need to be tethered and it can just record the stream live instead of recording to the RED DRIVE or simultaneously recording to RED DRIVE or CF. Obviously, someone will need to create the necessary hardware and support software, but that goes for the other RAID solution too...

Simple real-time backup or duplicate copy production would be the easiest to achieve. How about simply connecting two RED DRIVES to the camera and using a mirroring adapter on the SATA cable (obviously a custom one would have to be mad) to write to two RED DRIVES at the same time. Perhaps the ability for RED DRIVES to be daisy-chained could become a possibility, or the one connected to the camera via SATA can act as master or host to a slave drive connected via FW800?

 

[Gavin Greenwalt]

Or just buy a half dozen 16GB CF cards and swap them out every 10 minutes. I don't usually go through more than an hour's worth of footage per camera every day.

CF Cards go into hands of assistant who takes them to card reader. Dump in between takes.

I'm assuming you're on set because I can't imagine a nature documentary crew backing up on location.

 

[luminoir]

It's an interesting idea, however I believe you would see some pretty significant lag in the transition from the drives being unmounted from the Camera. Mounted on the PC. Begin transfer. Stop transfer. Unmount. Remount to camera. Begin recording. I suppose you could add a GigE connection and share the drive over IP.

Yep, you've pretty much got what I meant but I'm not sure about this paragraph. Which drives are you referring to that are being unmounted from the camera? the Red Drive or the Backup Drive?

I imagine you would have to shoot about 750GB before that would be necessary. Ideally, this solution should last the duration of the shoot and the transfer of the data should occur between shoot days.

Otherwise, it'd be slightly overkill to introduce triple redundancy with a 3 drive RAID1 and take one of the 3 to hotswap out of the Backup drive into a seperate casing to mirror into the laptop, while the remaining 2 backup the Red Drive. When the 3rd drive is swapped back in, the RAID 1 rebuilds it to restore the triple redundancy. Seems a bit over complicated at this point though.

 

[luminoir]

This seems to me like it's being made far more complicated (from a workflow point of view) than it needs to be.

~34MB/s is needed for 4K REDCODE @ 30fps. ...That's roughly 272Mbps. I can see an application there for dual-band 802.11n and no need to be tethered and it can just record the stream live instead of recording to the RED DRIVE or simultaneously recording to RED DRIVE or CF. Obviously, someone will need to create the necessary hardware and support software, but that goes for the other RAID solution too...

I see what you mean about going wireless N on the connection and it certainly is worth considering.
Main concern will be how this impacts on the development on the camera since N is still not released.

I think the gist of my idea, technological gotchas notwithstanding is incremental backup.
It's more reassuring to know you're backing up your takes
as you go along rather than praying the Red Drive
that has the only copy of your shots for the day
doesn't die prior to ingest/backup.

Simple real-time backup or duplicate copy production would be the easiest to achieve. How about simply connecting two RED DRIVES to the camera and using a mirroring adapter on the SATA cable (obviously a custom one would have to be mad) to write to two RED DRIVES at the same time. Perhaps the ability for RED DRIVES to be daisy-chained could become a possibility, or the one connected to the camera via SATA can act as master or host to a slave drive connected via FW800?

I reckon this would actually slow throughput down if I'm not mistaken.
The data would be considered to be written only when
the slower of 2 harddisks has confirmed receipt of the data
similar to how a RAID1 is slower that a single harddisk.

The reason I suggested a solution that starts copying only when
the Red Drive is not acquiring footage
is to avoid impacting on the acquisition throughput
by adding additional write access requirements.

 

[PenGun]

It's not directly related but I will point out the new Samsung 64G solid state drives are just coming out. Stuff 2 in a Red Drive and you got 128G Red Ram for about $550 per drive. Write at 90MB/s in RAID0. I'd guess $600 for the Red Drive sans drives so $1700 or so for dead quiet solid state swappable Red Drive units. Hold an hour or so plenty of time to dump the other one.

I imagine hacking up an interface to the LEMO connector will be pretty simple and SATA dumps from the Samsungs could exceed 100MB/s if you could deal with it.

 

[Jeff Kilgroe]

Main concern will be how this impacts on the development on the camera since N is still not released.

WiFi "N" was finalized over two months ago. :biggrin:

At ~270Mbps for REDCODE RAW 4K, a single band of wireless N could handle it, theoretically. However, I would look for a bonded, dual-channel implementation for guaranteed throughput -- or at least as close to guaranteed as can be. I see no reason why such a device couldn't be produced to attach to the camera where the FLASH modules do. The receiving computer would need a dual-channel, bonded N receiver card, driver software and additional application software to make it work though. However, that isn't too far off of what's needed for a tethered solution either. We'll see what RED does after the camera has shipped and has most or all of the intended features enabled. This may cross the line into third-party add-ons.

I reckon this would actually slow throughput down if I'm not mistaken.
The data would be considered to be written only when
the slower of 2 harddisks has confirmed receipt of the data
similar to how a RAID1 is slower that a single harddisk.

Keeping two attached drives in sync isn't that big of deal. yes, the slower drive will put a drag on the faster one, but I don't see this being an issue... If both are RED DRIVE units, certified to work with RED.

The reason I suggested a solution that starts copying only when
the Red Drive is not acquiring footage
is to avoid impacting on the acquisition throughput
by adding additional write access requirements.

I think such an implementation would work for some, but not for others... Just like most other recording media, I guess, but a bit more restrictive. For narrative work, the camera will see down-time between set-ups and there should be plenty of time for it to offload. OTOH, instead of attaching the extra drive to the camera and making the camera do the work... Why not simply swap the RED DRIVE or other record media over to a computer that automatically duplicates and dumps the data when connected. Just cycle the media... This is what most of us coming from a P2 workflow are already used to and it's no big deal. With RED, there's a lot more flexibility here too.

 

[Scott Webster]

It's not directly related but I will point out the new Samsung 64G solid state drives are just coming out. Stuff 2 in a Red Drive and you got 128G Red Ram for about $550 per drive. Write at 90MB/s in RAID0. I'd guess $600 for the Red Drive sans drives so $1700 or so for dead quiet solid state swappable Red Drive units. Hold an hour or so plenty of time to dump the other one.

I imagine hacking up an interface to the LEMO connector will be pretty simple and SATA dumps from the Samsungs could exceed 100MB/s if you could deal with it.

Given your prices, why is the Red Store RED RAM unit still at $4500? I'm hoping for a price drop before shipment as this would be our preferred drive option until CF increases in GB.

 

[Gavin Greenwalt]

Yep, you've pretty much got what I meant but I'm not sure about this paragraph. Which drives are you referring to that are being unmounted from the camera? the Red Drive or the Backup Drive?

Mounting is what the Operating System will do when it recognizes a new drive.

So if you connect a Hard Drive to your PC it has to mount it. Recognize the hardware and create an abstraction layer of sorts to utilize it. All that takes about 2-3 seconds each way. So I'm saying is unless you have some redundant ethernet connection which can act as a SAN host the RED camera would have to unmount the hard drive. Serve it to the the PC which would then have to mount it. And vice versa the other direction. All told you're looking at about 6 seconds delay between stopping transfer and starting record.

 

[PenGun]

Given your prices, why is the Red Store RED RAM unit still at $4500? I'm hoping for a price drop before shipment as this would be our preferred drive option until CF increases in GB.

No idea at all. I am building a laptop for capture and I'll have 3 of those Samsungs in RAID0.

http://www.avadirect.com/product_details_configurator.asp?PRID=3620

Nice eh'.

 

[luminoir]

WiFi "N" was finalized over two months ago. :biggrin:

Perhaps you know something I don't,
but this timeline from the IEEE 802.11 WG
seems to indicate the earliest approval to be sometime in 2009
and the timeline for the release of any protocol seems to be a little later.
http://grouper.ieee.org/groups/802/11/Reports/802.11_Timelines.htm

We'll see what RED does after the camera has shipped and has most or all of the intended features enabled. This may cross the line into third-party add-ons.

Agreed there. I think RED's strongest point is its open ended architecture
that welcomes 3rd party developers and partners
to enhance its value as a complete system.

Why not simply swap the RED DRIVE or other record media over to a computer that automatically duplicates and dumps the data when connected. Just cycle the media...This is what most of us coming from a P2 workflow are already used to and it's no big deal. With RED, there's a lot more flexibility here too.

That's true, however, a number of people seem to want an alternative workflow
and probably have their own unique reasons to do so.
Swapping a drive out means for a period of time,
the camera has no recording medium present
and would be unable to record if required.
That may be undesirable in certain circumstances.

I'm not sure if we're off topic as this thread was intended to address this question.

Looking for a good solution for taking data from the Red Drive (MASTER) out to two duplicated drives - One being the WORKING COPY, the other the BACK-UP.

Certainly, there are alternative workflows and with Moore's law,
in 18 months, we'd be up to our ears in solid state media that would approach a far more practical price point.

These would present us with a wealth of options
that would allow us to recommend those above
_MR_'s suggestion of field duplication
but until then I'm trying to see what's the best we can do given his parameters
of the files on one RED Drive backed up to 2 physical disks.

 

[luminoir]

Mounting is what the Operating System will do when it recognizes a new drive.

So if you connect a Hard Drive to your PC it has to mount it. Recognize the hardware and create an abstraction layer of sorts to utilize it. All that takes about 2-3 seconds each way. So I'm saying is unless you have some redundant ethernet connection which can act as a SAN host the RED camera would have to unmount the hard drive. Serve it to the the PC which would then have to mount it. And vice versa the other direction. All told you're looking at about 6 seconds delay between stopping transfer and starting record.

Ah, I understand what you mean now.

I don't think mounting time would be an issue,
because the idea is an incremental backup with the priority
set on being unobtrusive to acquisition so the wait time is probably acceptable.

I think your point about unmounting is very valid
and probably can be addressed with a buffer that allows for
forcible disconnection without dismounting that protects against data courruption from said tranferrus interruptus