Need backup/storage solution for 25 day shoot Red One MX

[dan freshman]

Hello,

I am new to the forums and pretty new to red low budget DITing...

On a project where we will be shooting 4k Redcode 28, 24fps. About 5 hrs a day with no offloading of media for the duration of the trip to post or through the mail(will be in the middle of nowhere). It looks like around 70GB/hour or 350/day (700/day when backed up only once)....As far as looking at drives, what would people recommend? I will be offloading from a macbookpro with USB 3 and a USB 3 hub. With this math, it's looking like around a terabyte a day. Does this sound right? Is there a recommended method for dealing with a setup like this??

Thanks for any and all help and advice,
dan

 

[dan freshman]

I am planning on doing 12 x G-Tech 2TB external drives. On our budget, these seem to get decent reviews...someone tell me im an idiot.

 

[Michael Romano]

I would get 2x Pegasus R6 Thunderbolt RAIDs. They're about $3k for an 18TB unit. I would avoid the USB3 hub, many of them seem to give you a hit in the speed department. The R6 is fast, has RAID5 redundancy and is relatively inexpensive compared to what a unit like that used to cost. I would also seriously look into LTO6 archival BEFORE the job is complete. We have the systems and knowhow to provide you with long term, reliable tape copies of your film. Email me at michael@strawberrycolor.com.

 

[ericyoung]

What kind of a shoot is it? Documentary, event, wildlife, narrative drama, reality show? If you aren't going to be offloading until the end of the shoot, are you sure you're getting enough Red media to cover 25 days? That's a huge number of Red Drives and/or Red SSD mags! Those alone will mean that the budget can't be that low.

G-Tech drives are fine - but note that 12 drives in boxes will take up a lot of physical room if that's a concern. On the other hand having fewer larger drives means more eggs in fewer baskets.
Also note that it's good practice not to fill hard drives up near 100% - probably no more than 80% is better. A full hard drive drops in performance significantly and can behave a bit unpredictably.

You definitely should have dual backup drives. Any backup media (new or used) can fail at any time. Dual backups is a minimum requirement - it's not a good place to cut costs.

Get Katadata app to give good guidelines on storage requirements.

As Michael suggests, LTO archival is also a good idea - if you don't want to buy an LTO solution, then hiring one may be a cost effective alternative. Thunderbolt raids will be faster which is a huge timesaver with large data quantities.

Also seriously consider using an MD5 Checksum software to do your backups.

 

[dan freshman]

Thanks for the reply...It's a 25 day shoot in Nigeria with a low low budget...it is a fight to get those drives even. We only have 1 red drive I believe and some red cf cards for backup. Shooting on the red one kinda limits us to firewire 800 for initial transfer then once on external drives, we will be able to play back rmd files through usb 3.0 in final cut.

I think a terabyte a day is a decent estimate and that's with a backup...shooting 4k redcode 28 yields about 70GB/hour so 350GB per day X 2 for the original and backup. I've heard that checksum apps like r3d manager slow down transfer times drastically, is this not true?

thanks again for an input.

 

[joe hedge]

Another good option is bare drives with docking stations

A LaCie TB to eSata hub, or Express 34 esata card if you have an Express34 slot, and eSata docks

Get Pelican cases for the drives, cut slots in the foam, works great for traveling as well

By the way, if you're shooting outside the US, Customs and FedEx/UPS will bone you on mailing the drives back here

Better to just hang onto them and carry them on the plane in the Pelican case

 

[ericyoung]

Checksums do slow down backups considerably, which is not good if you are limited to Firewire800, although the latest high spec MacBook Pro i7 CPUs will speed the checksums up.

The alternative is to check BOTH your copies back in realtime in theory as that is the only way to spot corrupt frames. Of course, you can just check one copy in realtime, and just spot check the other, but that's in the knowledge that there is a risk your second copy isn't perfect. If you or your production can live with that then fine.

While bare drives in docking stations can work, I personally would be uncomfortable with them in the fields as by their nature the drives are not physically secured, so it's much easier for them to be jogged while being accessed or for a rushed backup resulting in putting in a drive or pulling a drive without properly unmounting and depowering first. This will result in a corrupted drive. Also, you need to observe anti-static precautions.

If you're putting bare drives into a stock pick & pluck foam Peli case, make sure to use the anti-static bags as that foam isn't anti-static.

 

[joe hedge]

Checksums do slow down backups considerably, which is not good if you are limited to Firewire800, although the latest high spec MacBook Pro i7 CPUs will speed the checksums up.

The alternative is to check BOTH your copies back in realtime in theory as that is the only way to spot corrupt frames.

Checksums don't check for corrupt frames, they just check if the file sizes match. Matching file sizes doesn't mean nothing's corrupt, it just means you might have matching corrupt files. Best way to check for corrupt frames is to transcode.

 

[ericyoung]

Checksums don't check for corrupt frames, they just check if the file sizes match. Matching file sizes doesn't mean nothing's corrupt, it just means you might have matching corrupt files. Best way to check for corrupt frames is to transcode.

Checksums actually do check for frames/files corrupted by the copying process - the most common weak point. And they do more than check if file sizes match - as Finder/Windows Explorer file sizes don't look at the whole file, just the file headers.
But matching checksums do give a good assurance that all the copies' binary bits match the camera source files.

Of course, you are right they could be perfect copies of footage corrupted by the camera on the original Redmags. But if checksums match, we only need to check one copy manually in realtime for that, knowing the other copy is the same.
Transcoding takes additional time before you can view it, and a delay before you can report a problem with the original footage off the camera, and maybe allow a possible reshoot of those shots before the crew moves on too far.

Still every DIT has different workflows. This is just mine, although sometimes time and circumstances may prevent use of slow MD5 checksums.

 

[joe hedge]

Transcoding takes additional time before you can view it, and a delay before you can report a problem with the original footage off the camera, and maybe allow a possible reshoot of those shots before the crew moves on too far.

RC-X generates error messages as the transcode occurs, and Resolve actually plays back the raw file as it's being transcoded. By watching either one, you know if something's wrong before the transcode is finished.

Not sure what you mean by "a delay before you can report a problem". By far the most important part of the job is QC'ing the camera files.

 

[Todd T]

Not sure what you mean by "a delay before you can report a problem". By far the most important part of the job is QC'ing the camera files.

It means you can't verify the transcoded output until the process is complete, which could take hours, as opposed to just copying the R3D files with checksums to verify the integrity of the data.

On another point; As for the Firewire interface, if you use a program like Shotput Pro, Double Data, or CopyVerified, there is minimal or no speed penalty copying to multiple drives; these programs read once from the source drive and make simultaneous writes to the destinations.

That way you won't spend 40 minutes copying data from the Red drive to backup disk #1 and another 40 minutes copying from Red Drive to backup disk #2.

 

[Todd T]

I am planning on doing 12 x G-Tech 2TB external drives. On our budget, these seem to get decent reviews...someone tell me im an idiot.

You are not an idiot. Given the constraints of the your budget and location, a bunch of individual drive is a viable solution. You just need to assess your risks.

Be sure you have enough spares with you or access to spares in a timely manner. Since you won't be (or should not be) editing on the drives you can fill them up 95%, just be sure to run thorough disk tests on all the drives before you go.

Dragging Raid units to remote locations is just as risky as any other solution, For minor raid failures, you will loose time rebuilding raid sets from minor failures, even more time to mirroring the data from one raid to the next because of catastrophic raid failure. Either way you need a supply of spare drives.

 

[joe hedge]

It means you can't verify the transcoded output until the process is complete, which could take hours, as opposed to just copying the R3D files with checksums to verify the integrity of the data.

What? You can't check an individual transcode till the whole batch is complete? Why not?

And again, no checksum verifies the integrity of the data - it just checks if the file sizes match. It won't catch a file the camera corrupted. It'll copy bad camera data w/o telling you it's bad.


And if the transcoding a single reel "takes hours" you shouldn't be doing it on-set in the first place.

 

[sander kamp]

Go with small 2TB USB 3 drives. You're probably not going to have stable power and you're likely on the road a lot so go with bus powered drives. With 1TB per day for double backup you only need 12 of them (with a few spare) so that's totally doable. Label them and distribute the drives over the luggage in the crew. Count on having luggage lost or stolen during the trip.

Get a newer MacBook Pro with good battery life, connect the two drives to the two USB ports and get a thunderbolt to Firewire adaptor to connect the Red Drive or card reader. Put the whole thing with maybe a backup battery in a pelican case. Imagine the whole thing in a rainstorm in the back of an open pickup truck going high speed down a dirt road or something. A backpack you can keep on all the time is also a good idea, as long as it is water and dust proof.

I tried al the copy software and found that only ShotPut Pro is fast enough to be usable, it actually copies to two drives in the same time as you would copy to one so with it checksum it is still as fast as making two separate copies. Double Data was very very very slow when I tried it.

Don't worry too much about corrupt frames and such, you can't really go transcoding and watching back 5 hours of footage per day while on the road and doing backups at the same time. Do random checks, make sure the footage is mostly OK and the camera is behaving well. What are you going to do if you find corrupt frames? It doesn't sound as if you can reshoot scenes or anything. Same for LTO drives or Raid systems - too cumbersome and time consuming, they need power and setup time and space and all. Keep things simple and small and on battery power. And separate those drives!

 

[Todd T]

What? You can't check an individual transcode till the whole batch is complete? Why not?

And again, no checksum verifies the integrity of the data - it just checks if the file sizes match. It won't catch a file the camera corrupted. It'll copy bad camera data w/o telling you it's bad.


And if the transcoding a single reel "takes hours" you shouldn't be doing it on-set in the first place.

To quote your earlier post

Best way to check for corrupt frames is to transcode.

So yes, you can't be sure that everything is valid until all of it is transcoded, but that will always take a significant amount of time. How much time would you estimate it would take to transcode the Dan's projected 350GB/day of footage? And how would you propose to verify the data on the second backup disk?

As far as checksums go, in addition to what Eric said earlier:
The MD5 message-digest algorithm is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value. MD5 has been utilized in a wide variety of security applications, and is also commonly used to check data integrity.

Dan is working with very limited resources, so he will have to take some risks and make some compromises. Programs that copy and generate checksums will provide some confidence in verifying the integrity of his footage. Picking random clips to scrub through RCX will add more confidence.

 

[joe hedge]

How much time would you estimate it would take to transcode the Dan's projected 350GB/day of footage? And how would you propose to verify the data on the second backup disk?


As far as checksums go, in addition to what Eric said earlier:
The MD5 message-digest algorithm is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value. MD5 has been utilized in a wide variety of security applications, and is also commonly used to check data integrity.

Dan is working with very limited resources, so he will have to take some risks and make some compromises. Programs that copy and generate checksums will provide some confidence in verifying the integrity of his footage. Picking random clips to scrub through RCX will add more confidence.

350 GB a day is about 90 minutes of 5KHD at 8:1 compression. I'm currently getting slightly better than 2:1 transcode ratio with no Rocket with that, on an 8-core/GTX 680 set-up. Meaning I can pretty easily stay caught up with the transcodes with decent (220 MB/s) i/o bandwidth. And that's with 3 copies of the raws, not 2.

And again, checksums only check for matching file sizes. If the original camera file on the Red SSD is corrupt, a checksum won't catch it. It'll just verify that you made a matching copy of a corrupt file.


Still wondering what on earth you meant by "you can't verify the transcoded output until the process is complete" - you certainly can playback individual transcodes before the entire batch is finished.

Oh, and filling drives to 95% of capacity isn't something I'd ever do. 80% is pushing it.


How experienced are you at this, really? Perhaps not enough to be posting advice about it? "He will have to take some risks" - that sounds like something a director would say to an actor, not something anybody would want to have going on with raw camera files.

 

[Todd T]

For the purposes of this discussion I absolutely no experience with cameras, digital file formats, computers, or disk systems.

I have three simple questions:

Why 80%? Why not not 75%, or 85%

What procedure do you propose to verify that both backup copies are intact?

How does your solution and capabilities meet Dan's requirements and constraints?

 

[Eric Bowen]

Drive Manufacturers are shipping drives with an acceptable amount of bad blocks nor does it seem they are marking them out a head of time. You want to leave enough known good space that a drive can move data in bad block areas to good. Mechanical drives start dropping performance over 50 to 60%. The performance drops like a rock once over 75 to 80%. If you wish to avoid bad blocks in critical data areas with mechanical drives, write zeroes to the drives before you take them out for media data work. Takes a day or 2 of prep time but it's worth the time to get the bad blocks marked out ahead of time.

 

[Todd T]

Thanks Eric.

Those type of performance figures were generate by measuring random access read and writes to multiple and typically small files, and meant to represent file activity and performance on a normal system drive.

How much is the drive performance affected when writing large, sequential, contiguous files to a freshly formatted disk that is only being used for archiving?

 

[Eric Bowen]

The Testing we do is always on empty drives using HDD benchmark utilities that test data reads at different locations of the drive. I assume they use an algorithm to determine that figure based on the response time for the data return. I have seen similar results with Sandra, HDTune, and some older testing benchmarks I don't use much anymore regarding the performance drop.