Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Need backup/storage solution for 25 day shoot Red One MX

My point is, the 80% rule is dogma that doesn't apply to archiving scenarios.
If you are just copying large files, performance is not an issue.
Saying you won't use 400GB of space on a 2TB drive, or 800GB on a 4TB drive "just in case there are bad sectors" does not make sense any more.
 
joe hedge said:
RC-X generates error messages as the transcode occurs, and Resolve actually plays back the raw file as it's being transcoded. By watching either one, you know if something's wrong before the transcode is finished.

Not sure what you mean by "a delay before you can report a problem". By far the most important part of the job is QC'ing the camera files.
Click to expand...

Well, it does depend on what your workflow is after receiving a camera mag.

When I receive a Redmag, my first action is to make at least two, if not three, backups. If I DON"T use checksums, the only way I know for sure all copies are good copies is to watch all 2 or 3 copies in realtime. Using MD5 gives a high degree of certainty that all the copies are identical. I therefore only need to watch one copy for errors which will pick up bad camera files which I can report as soon as I spot them. This will also spot dodgy focus, booms in shot, and other non data problems. Yes, using checksums ARE slower than Finder copies, but preferable to watching it more than once, for me at least!

If your transcode is twice as fast as realtime, then you save half the time there, but have to repeat the transcode, or watch in realtime, the one or two other copies to be sure they are also good copies, which negates any timesavings there. If you are using Resolve to transcode and using the preview feature, then your transcode will be slower than realtime - again negating any timesaving over watching one copy in realtime. And you still don't know if the other copy(s) are good.

Basically transcodes are not a complete alternative to checksums or watching footage in realtime. They are different processes with some possible overlap. However, sometimes circumstances (such as the OP may encounter on his Nigerian project) MAY dictate that it's impossible to do multiple backups with checksums, AND realtime watchback of a copy AND transcodes. He should then decide, in consultation with his DOP, and production, where they are willing to compromise and willing to accept a higher risk.
 
Use bare drives. Way less expensive than G RAIDS per TB, dual docks are cheap, eSATA is fast, and you can back up to two simultaneously. You don't have to leave them banging around in a bag - put them in plastic cases made for 3.5 inch drives - check Amazon. Just make sure you have extra cables and an app like R3D Data Manager that does checksums. Easy.
 
joe hedge said:
350 GB a day is about 90 minutes of 5KHD at 8:1 compression. I'm currently getting slightly better than 2:1 transcode ratio with no Rocket with that, on an 8-core/GTX 680 set-up. Meaning I can pretty easily stay caught up with the transcodes with decent (220 MB/s) i/o bandwidth. And that's with 3 copies of the raws, not 2.

And again, checksums only check for matching file sizes. If the original camera file on the Red SSD is corrupt, a checksum won't catch it. It'll just verify that you made a matching copy of a corrupt file.


Still wondering what on earth you meant by "you can't verify the transcoded output until the process is complete" - you certainly can playback individual transcodes before the entire batch is finished.

Oh, and filling drives to 95% of capacity isn't something I'd ever do. 80% is pushing it.


How experienced are you at this, really? Perhaps not enough to be posting advice about it? "He will have to take some risks" - that sounds like something a director would say to an actor, not something anybody would want to have going on with raw camera files.
Click to expand...

350GB/day on RED ONE MX of 4K at Redcode 28 - as the OP specified - is approximately 200 mins of footage. He also mentions it might be 5 hours which is more like 500GB a day. Even with 200mins of footage, that on some occasions may arrive at the end of a shoot day in one go in this project. It's going to be a challenge to backup, visually check and transcode on a MacBook Pro (even with a Red Rocket in an external chassis) and still get a decent night's sleep. Simple logistics dictates that some compromise and acceptance of higher risk may have to be taken in these difficult shooting environments and schedules. We are not talking about a blockbuster Hollywood feature film here.

MD5 Checksums DO spot file data corruption in the transfer process from Redmag to backup drives. If it simply compared file sizes it would not need the significant time and CPU cycles hit - and would offer zero advantages over a Finder copy!
I have used CRC Checksums as a faster, but less certain, alternative. That too does more than compare files sizes.

I don't think it's possible to playback individual transcodes smoothly in realtime while other queued transcodes are accessing the same drive? Some visual problems are only really visible while viewing in realtime.
 
Todd T said:
My point is, the 80% rule is dogma that doesn't apply to archiving scenarios.
If you are just copying large files, performance is not an issue.
Saying you won't use 400GB of space on a 2TB drive, or 800GB on a 4TB drive "just in case there are bad sectors" does not make sense any more.
Click to expand...

You're right that drives can be filled 100% for leisurely pure archiving. However, when speed of backup writing in the wild is a factor like it appears here, then disk performance IS an issue. Besides, when planning for how many hard drives to buy for a remote location where there won't be many computer hardware stores available, it's good to err towards too much storage rather than too little. A 20% safety margin is a reasonable safety margin, although on a recent shoot, the amount of footage a day exceeded the planned amount of data by considerably more than 20%. Luckily we were in a major western city with hard drives easily available. Hopefully Dan's film crew and director will be more disciplined about not overshooting.
 
I agree that a drive should not be filled 100%, and I also agree that having a 20% cushion over the estimated size of the footage capacity is prudent as well.

I don't agree with saying that individual drives should never be filled beyond 80% capacity; it does not make sense with respect to the capacities of modern drives.
 
You must log in or register to reply here.
Back
Top