How to copy and back up?

[Nick Shaw]

So you'll need to have 2 copies of the footage and 3-way verify the last version before you manually(!) delete the first copy. I dont like it, but thats what Im thinking would be necessary to completely mitigate all the risk when pulling the drive before the initial checksums are complete.

I understand what you're saying about "mitigating all the risk". However I am talking about having the option to take some unmitigated risk if necessary. It would not be the norm, but I want the option to return the drive to the camera for formatting, knowing that at least one copy has been made (but not verified) rather than hold up shooting.

When working on a bonded picture, I know there may be conditions imposed, and processes that HAVE to be followed. But on other kinds of shoot, there are always compromises to be made, and associated risks to be taken. I just want options.

My ideal would be for the software to copy everything from the Red media first, then checksum and compare one .RDC folder at a time so that in an emergency abort situation you knew which clips were verified. Ideally the ability to flag circle takes to be verified (and maybe even copied) first would be useful.

 

[Jonas Rejman]

Photocon, you told me to use other hardware. I can't really, because RED is build that way. I have to use firewire cables, I have to use CF readers, I have to trust the firewire ports of the readers and those of the notebooks.

I may be totally wrong here, but I think, that the biggest problem on the data's path to the post's RAID's is between the data written onto the CF inside the camera, and till the point, where it is first copied on the first offload drive.

This is the most dangerous part, this is where most of the mistakes and bitfilps might happen. The set conditions are rough: cold, weather, dust, people are tired, etc. We bought R3D data manager and while studying how it works, we found that if something is supposed to go wrong, it might just during the process when the data-manager is reading the data for the first time, creating the checksums.
But it might have copied the data by that time already. Because it just takes too long to deal with the data, before it actually makes the first copy.

That means, that for example, when the data-manager reads its first checksums from the media, and at this very moment, the CF card blows, or the CF readers decides to flip the data on the card, without the data-manager I would have copied the data by then, and have at least some parts of the data on another drive.

I am not saying my approach is as pragmatic and save as yours, I am saying, that practically, it might be faster, to make the first copy without any checks. VISUALLY check the data on the copy. Make that first copy as fast as possible, not reading the original media three times, THEN do the copy.

IF the data is ok, then make checksums, verify the checksums, and go for the next copies. If the data is NOT OK, if an error occurred during this first, most dangerous steps, then you have to make a thorough investigation anyway, to find out what went wrong.

There were users, turning off the checksum function of the R3D data manager because your approach just takes way too long. while being pragmatic, it just does not seem to me practical on the set. With the fastest CF reader at 50MB/s it still takes ages to read 16BG three times.

With my approach I can read from SSD's to create the checksums AND I have visually checked the data, while the data has been copied as fast and as soon as possible. Of course, I have the responsibility stating, that the first copy is a "new" MASTER, but I tend to think, that that forces the DIT to actually check what is inside the takes, rather to rely on a copy procedure.

Say the camera CF writer is bad and it writes something bad onto the CF-card in the camera. Say, there is one quarter of the picture in a moiree pattern, images we have sometimes seen. With the R3D data manager, the DIT might be in false security, that the image is ok ALWAYS, because you do the thorough check-summing.

With my approach, he HAS TO check the data on the first copied drive visually. Open them, check them, state that they are ok. There is no way for him to get out if this responsibility. In my eyes, this is where the money lies on the DIT.


We had a lot of clients using the R3D data manager while shooting on reddrives. They tend to shoot half a day, and then offload, trusting blindly a harddrive in raid 0. It took then about 5 hours to make a checksum checked copy of the data on another drive with R3D data manager. But they have never checked the data, because they were busy with the copies. When I did check the data and found, that one take was corrupted, because of a dying reddrive, they could not believe how this might have happened - they used the r3d data manager after all.
Of course they used older firewire HW, of course they should have offloaded more often. But unless they loose data, they won't do that. And if they do, they will blame the camera and/or the R3D data manager for that, who else?

One of the reasons I love the 16GB cards, is that they FORCE you to offload, but you still have almost as much material as on a 600m film roll.

Maybe some sort of a protocol should be established, and I wonder if this might be a compromise of speed and security:

0. OFFLOAD as often as possible
1. Make first copy of media to master storage as fast as possible, without checksums
2. FORCE DIT to visually check all data, while
3.1 Read media again, create checksum, store off original media
3.2 Simultaneously read master storage again, create checksum
4.1 Read media again, verify checksum
4.2 Simultaneously read master storage again, create checksum
5. IF all checksum match AND IF data visually ok, original media can be released


Does that make sense, and is practical?

Would it suffice, to NOT create checksums from the originals, but visually check first copy and create and verify checksum, to speed up the process?

Phocon, what is your take on bitfilps of SSD's, worse or better than HDDs?

How many shooters are really making thorough check-summing, and do not just rely on finder (that should have some sort of checks as well, no?), to speed things up?

 

[Ted Chu]

Can't agree more, the DIT needs to do a backup and check the footage quickly while you still have the ability to reshoot the scene if the file is corrupted. Then you can do your checksum backups. I have also been checking the "good" takes on the camera playback. In my experience, it seems like it won't playback if the file is corrupted.

 

[Raffi Kryszek]

I know the ability to offload footage over eSATA has been on the wish list of RED users since the camera’s inception. The R2E LEMO to eSATA cable lifts the burden off of a computer’s FireWire 800, FireWire 400 or USB buss. By simply powering the RED-Drive or RED-RAM using the original power adapter, plugging the LEMO end of the R2E cable into the drive and the other end into a standard eSATA port, the user gains instant access to RED footage with eSATA performance.

www.CoolCameraGear.com

http://twitter.com/coolcameragear

 

[gbalaji]

I've couple of things about R3D Data Manager. I use for for my 3 feature films and couple of ADs and it works great.

What needs to be done when the results shows errors, Do I need to start copy again. I don't find any answer for this?

When copying from Red Media, erase, leave option needs to be removed since it may cause panic for couple of users when the copy is done and they have not selected leave option the media will be deleted which no one wants at the set.

I'm using RAID-5 for couple of users and when I start copy to RAID from my source disk some days transferring takes lot of time in comparison with External Harddrives used as another backup. Any reason for this?

The above issues are pointed out to have great product and I believe the product is evolving over a period of time.