How to copy and back up?

[Thomas Mathai]

Has anyone tried using rsync in OS X Terminal?

 

[Chris Parker]

so what do you use now mark?

 

[Jonas Rejman]

We started to develop our own application called Cinezoid, because of R3D Managers limitations.

As Mark, I was horrified, when I found out, that it writes into the magazines. Also, its not able to copy to a backup storage from a main storage, but reads the camera media again, which takes awful long and blocks the CF card.

We took another approach in our program. It basically starts with the checksum on the first copied storage. I do not see any benefit in reading checksums from the CF cards, only to find out, that they do not add up on the first media. You have a huge problem in that case anyway - it can be the cables, readers, firewire port, camera CF reader... - just too much hurdles to take, where a checksum will not help you anyway.

As you simply HAVE to visually check the data on your first copy one way or another, Cinezoid generates the checksums on the first copy, re-reads the data and makes a backup copy, with another checksum. This all happens in the background, and gets interrupted, when you insert another card to offload.

With 2 Intel X-25M SSD's it takes about 15 minutes for 16GB to copy, read to generate checksums, read to verify checksums, copy to backup, read to verify backup checksums. With normal drives, it takes about 30 minutes.

Cinezoid has an itunes-like interface, its written in java 1.6 under netbeans and its a 64bit application, so it will work well with snow leopard and it will be open-source. Currently its an alpha version.

I have started a wiki about it here, on the Kenai Forge.

We are looking for hardcore beta-testers, to iron out some bugs, people that would not mind to put the program through its paces and report issues at kenai Forge.

PM or email me at jonas@rejman.com, but only if you are serious about alpha-testing. Its nothing pretty... :-D

 

[Conrad Hunziker]

I have not personally used R3D Data Manager in a while - so, someone correct me if this my comments here are outdated and my issue/concern has already been addressed.

Yes, you are a bit late on the information. The latest release already has code to mitigate this issue, due out shortly.

I never understood why R3D Data Manager doesn't allow you to SET A DESTINATION FOLDER for the checksums - so you can write them on a volume which is not your camera media. Did this feature already get implemented?

The next release will allow you to write checksums to a separate folder.

However, there are a couple of more issues to look at:

- Not everyone follows the same workflow. Many users offload their drives in increments, keeping data on the Red Media until it is truly full. For those users, it will be possible to have RDC folders that already had checksums created but will need to be re-created due to other factors. So that will cost them additional time in creating those checksums.

- It is entirely possible to have duplicate RDC folders from DIFFERENT cameras. By writing checksums information to the original RDC folders you can mitigate which folder is which, and use the correct checksum. However, you will not be able to do that when writing checksms to an external folder, as you will not be able to determine which folders are using the correct checksums. This was a huge issue for camera builds before build 15, but is still an issue to this day even after Red changed the folder naming format from containing the year to random letters. I personally have seen on 2 occasions RDC folders that were exactly the same name, the only difference being the camera letter (ie, camera A, B, C...)

 

[Mark L. Pederson]

Yes, you are a bit late on the information. The latest release already has code to mitigate this issue, due out shortly.



The next release will allow you to write checksums to a separate folder.

Good news Photocon - I expect we'll start using again then! I do like the app - except for this issue.

However, there are a couple of more issues to look at:

- Not everyone follows the same workflow. Many users offload their drives in increments, keeping data on the Red Media until it is truly full. For those users, it will be possible to have RDC folders that already had checksums created but will need to be re-created due to other factors. So that will cost them additional time in creating those checksums.

Understood - and I respect people work in different ways - IMHO it is NOT SMART to offload in increments for several reasons such as an increased chance of human error - more opportunity to corrupt a directory, etc. - it feels a lot like the old days of "remove 35mm mag from camera, breaking the neg in the mag, putting mag back on the camera" - sure it works - but you are increasing you margin for error. IMO this entire "data management game" is all about reducing the opportunity for human error and redcing the chance for a file table/directory corruption.

Again, horses for courses - but I think folks who want to offload in increments can pay the price of slower checksums.

- It is entirely possible to have duplicate RDC folders from DIFFERENT cameras.

I personally have seen on 2 occasions RDC folders that were exactly the same name, the only difference being the camera letter (ie, camera A, B, C...)

Yes - this CAN happen - EXTREMELY RARE - and if your tech does not have different camera ID letters on each camera - he/she deserves it. I DO NOT buy this as a good arguement for writing checksums to my digital negative.

Again, if you give me the option to write those checksums to another folder - outside the RED MEDIA - I'm very happy and will have my team start using the app again.

 

[JanneJansson]

Has anyone tried using rsync in OS X Terminal?

Yepp I'm using that method. Works very good, very fast and realiable.

 

[Conrad Hunziker]

Also, its not able to copy to a backup storage from a main storage, but reads the camera media again, which takes awful long and blocks the CF card.

This is a popular misconception with R3D Data Manager. It can copy from any media - the Red Media or non-Red Media. The original copy or copies of copies. There is no reason why you cannot copy from any mountable filesystem to any mountable filesystem.

We took another approach in our program. It basically starts with the checksum on the first copied storage.

I strongly urge you to re-consider your approach, and hope you do not deceive the public into thinking this flawed logic is correct.

At no time should anyone ever consider the checksum of a COPY to be the standard to which all other copies are to be made. I cannot emphasize this point strongly enough. If you consider the checksum of a COPY to be the standard, then all you are doing is ensuring that all of your COPIES will forever have any defects that are in that copy but NOT in the original. This includes slipped bits, backward bits, misreads and miswrites.

Again, this is such an important point, I need to be very very clear. Your checksums must always be done from the original source, not ever from a copy. The copy's checksums must always be verified to the original, not a copy.

Every modern hard drive, regardless of manufacturer, has a feature called ECC - error correcting code. They do this because the data read off the hard drive will have errors in it. Drives are packed so tightly these days that it is literally impossible to get a 100% reliable read at any sort of sustained speed. The drive manufacturer knows this and has code to help mitigate errors on the fly. The read errors from the hard drive can be in the 10s of thousands every minute (during a sustained read), and the code will hopefully catch each one of those errors. But how do you know it catches every one of them, or that the data it replaced was valid?

So by the time you are creating checksums from those copies, you may have 2 levels of read errors already incorporated into those checksums - one from the Red Media, and the other from the first copy drive. Now you are proposing to use those checksums as the source for further copying?

I respectfully submit that you seriously re-think your entire approach. And I hope that the people who use your software do not rely on its checksum features as currently implemented as a cornerstone for any loss mitigation process.

By the way, it is not possible to see the ECC errors that your drive is correcting for by looking at the SMART information for your drive. SMART drives do collect this info, but do not publish it. You'll need a third party program to divulge this info from the drive. And yes, you will find that your drive, under normal sustained read operations, will have thousands of errors per minute - even 10s of thousands are "normal" - which the drive hopes to correct before passing it on to your computer. This is the standard operating procedure for every modern hard drive made this century.

I do not see any benefit in reading checksums from the CF cards, only to find out, that they do not add up on the first media. You have a huge problem in that case anyway - it can be the cables, readers, firewire port, camera CF reader... - just too much hurdles to take, where a checksum will not help you anyway.

Honestly, if you cannot count on your Red Media via the firewire port, cables and CF reader to make accurate checksums, why should you count on those items to deliver that media to your copy destination without error? If those things are not generating reliable data that you can count on, you should be using different hardware, plain and simple. And in reality, if any of that hardware goes bad (as can be the case), you should know as soon as possible - not after several copies have been made and assumed to be good, and even worse, after the original data has been overwritten.

One more point about checksums: It really needs to be a three-read process to create an accurate checksum and copy. You need to read the data from the source once to create the checksum and once to copy. Then you read the data once again on the destination to verify the checksum. Only by doing that can you verify that the original checksum from the source is valid, and not full of random read errors itself. Every checksum generation process is subject to random read errors. The only way to combat that is to read the data multiple times. If you only read once, then you only know the checksum of that particular read session, not that its a valid read.

As you simply HAVE to visually check the data on your first copy one way or another, Cinezoid generates the checksums on the first copy, re-reads the data and makes a backup copy, with another checksum. This all happens in the background, and gets interrupted, when you insert another card to offload.

I will not disagree that viewing the material after a copy should be part of one's normal data management duties. However, I will submit that your order of operations will require that you erase a Red Media of its data before you view the first copy. Because of this, there may be perfectly checksumed and copied data that doesnt add up to anything visually. You really need to ensure that data is always 3-way verified: all files are copied, all files match, and all data is valid. Your proposed workflow on working sets will require that only one will be accomplished before the Red Media is erased. This, in my view, should not be the standard for any production.

 

[Conrad Hunziker]

Good news Photocon - I expect we'll start using again then! I do like the app - except for this issue.

Watch for the upcoming 5.0 release. BTW, it will only work in Mac 10.5

Understood - and I respect people work in different ways - IMHO it is NOT SMART to offload in increments for several reasons such as an increased chance of human error...

I wholeheartedly agree - but people like their workflows, and R3D Data Manager tries to accommodate all types. I personally feel that this is one of the ramifications of the way the release of the camera occurred. Thats not Reds fault - but there was so much mis-information out there that the opportunity to instill correct workflows was filled instead with bad information and people's fears. To this day I still correct people with information from camera build 8.

 

[Conrad Hunziker]

Has anyone tried using rsync in OS X Terminal?

I have a couple of comments about using rsync.

First, it doesnt provide a way to re-verify your data at a later date. It only provides a way of making sure the destination matches the source. Once you erase the source, there is no chance of re-verification.

Second, it doesnt provide a way of validating the source. It just assumes that the read data is correct.

Third, there is a whole world of opportunity for human errors in the process. If you are computer savy (not just point and click, but understand what the commands ls, cd, su, awk, sed, and vi do) and can handle the stress, rsync may work for you. But there will be this one time after 16 hours on set you accidentally copy to the wrong area, potentially overwriting data.

I use rsync on a daily basis for backup of the 3 dozen or so servers I maintain (+/- 4.6Tb daily, non-red related). It works well. But for on-set data management, Id prefer a one-click to take care of it all, so I can return to set duties. Choose what works best for you.

 

[Mark L. Pederson]

Watch for the upcoming 5.0 release. BTW, it will only work in Mac 10.5

Please make sure I have the option to COPY the media BEFORE performing any checksum operations. Get the copy done stat. Then check the data. This is VERY, VERY important. Especially when folks are using DRIVES.

 

[Conrad Hunziker]

Please make sure I have the option to COPY the media BEFORE performing any checksum operations. Get the copy done stat. Then check the data. This is VERY, VERY important. Especially when folks are using DRIVES.

Mark, perhaps you didnt read my diatribe above. Creating the initial checksum will always be first, not copying. You can, right now with the current software release, remove the Red Media while the software calculates the destination checksums. There is no issue doing that. However, you must create the initial checksums from the source - that is not ever going to change, for the reasons I outlined above.

You can use R3D Data Manager to just copy, and get the benefits over finder of ensuring that every file is copied, buy disabling the checksums altogether.

And one more point - if you are overwriting (reformatting or recording) the original Red Media before you do all 3 stages of verification (verify files, verify contents, verify valid data), then you are taking a risk. As you pointed out before data management is really risk mitigation. So if you are not verifying that all files copied over AND verifying the data in the files is correctly copied AND verified the data is valid image data, then you are taking a risk. R3D Data Manager - and really, any tool out there - can only do 2 of those. We are working with Red to provide a way to do #3 autonomously.

 

[Nick Shaw]

Please make sure I have the option to COPY the media BEFORE performing any checksum operations. Get the copy done stat. Then check the data. This is VERY, VERY important. Especially when folks are using DRIVES.

I'll second that request. It is the reason I do not currently use R3D data manager. I understand the importance of check-sums, but agree with Mark that the number one priority should be making a copy of a mag as soon as it comes of the camera.

EDIT (after reading Conrad's reply above):

Is it not possible to prioritise copying media, THEN make checksums FROM THE ORIGINAL, then copy those to the destination, then compare checksums. Surely copying first does not remove the option to make checksums from the original, it just delays it.

 

[Conrad Hunziker]

Yes - this CAN happen - EXTREMELY RARE - and if your tech does not have different camera ID letters on each camera - he/she deserves it. I DO NOT buy this as a good arguement for writing checksums to my digital negative.

One small point with this statement. Yes, the tech does deserve what they get for not having different camera letters and the cameras generate random letters that happen to be exactly the same for the RDC folder name. But as the author for software that has transferred literally pentabytes of data to date, I cannot, honestly or morally, release software that I know does not account for every issue that I know of. Since I have seen this occur to me personally on sets twice, I know it happens on other sets in a predictable fashion. Therefore software used day in/day out must account for it.

We have a solution that will be the best of both worlds, I think.

 

[Conrad Hunziker]

I'll second that request. It is the reason I do not currently use R3D data manager. I understand the importance of check-sums, but agree with Mark that the number one priority should be making a copy of a mag as soon as it comes of the camera.

Guys- Ill beat against this stone for as long as it takes. You cannot EVER create the checksums from anything else but the source. It must only be done from the source. You cannot release a Red Media from the computer until after the checksums are made. There is no getting around that.

If your workflow requires that, then you seriously need to re-consider your workflow. I will not write software that is setup to fail. By releasing a Red Media before the checksum is made, that is failure - end of story.

 

[Conrad Hunziker]

EDIT (after reading Conrad's reply above):

Is it not possible to prioritise copying media, THEN make checksums FROM THE ORIGINAL, then copy those to the destination, then compare checksums. Surely copying first does not remove the option to make checksums from the original, it just delays it.

Ill reply to this edit-

Thats not a bad idea. I need to do some simulations first before I say yes or no to it. In the 3 minutes I thought about it, it could work, but let me think for a bit to figure out all of the ramifications. There may be a small issue with the reads, but let me think it through.....

EDIT: (though about it some more)
There will be a performance hit. Right now R3D Data Manager creates the checksums while you enter data or add more sources. For example, while you enter data about the footage (script scenes) or setup destinations or add another source, it is already calculating the initial checksums in the background. So it would be waiting for you to start the copy (after doing all the above) before it actually calculates any checksums. The difference can be 5 to 10 minutes, depending on how fast you are.

 

[Nick Shaw]

Following from my comments above, while I agree that in an ideal world this should never happen because you have enough Red Drives on a shoot, I envisage a situation when a drive may need to be returned to the camera before the full download and compare checksums is complete. If the download is done first, then you may only be aborting the checksum operation, which while far from ideal does mean a mag can be freed up to return to the camera.

One example where this sort of thing might happen is a shoot I worked on recently where we had two Red Drives with the camera and a backup body which also (by chance) had one Red Drive with it. It was then decided (as often happens) that the backup camera should be used as a B camera, so we found ourselves cycling three Red Drives between two cameras. We wanted to swap mags at points convenient to the shooting, not dictated by the download process, so if I had used R3D data manager I would have worried that when I got a Red Drive with over 50GB on it, it might have been needed back at the camera, and the checksum would be complete but the copy would not. Making a checksum of 50GB of data over FW800 takes a long time!

 

[Mark L. Pederson]

THEN make checksums FROM THE ORIGINAL, then copy those to the destination, then compare checksums. Surely copying first does not remove the option to make checksums from the original, it just delays it.

This is how it should be done.

 

[Conrad Hunziker]

One example where this sort of thing might happen is a shoot I worked on recently where we had two Red Drives with the camera and a backup body which also (by chance) had one Red Drive with it. It was then decided (as often happens) that the backup camera should be used as a B camera, so we found ourselves cycling three Red Drives between two cameras. We wanted to swap mags at points convenient to the shooting, not dictated by the download process, so if I had used R3D data manager I would have worried that when I got a Red Drive with over 50GB on it, it might have been needed back at the camera, and the checksum would be complete but the copy would not. Making a checksum of 50GB of data over FW800 takes a long time!

Well, I respect your situation. That particular situation and other similar ones have happened on sets Ive worked. But I respectfully submit that it is the workflow, not the tool, that is the issue here.

Regardless of if the copying is done first or second during download, returning the drive back to the camera with footage on it that has not been 3-way verified is just asking for trouble. On one shoot I was on in particular (for clarification, I was the DP, not data manager) that exact situation happened where the data manager decided to return the drive to the camera before the data had been verified. And the AC promptly formatted it - and we lost the footage. The copied data had not yet been 3-way verified yet, but the data manager thought it would be fine. Turns out he was wrong, the the footage was never recovered, and we re-shot several days later.

So, in your situation above, I agree that the best time to reload is always when there is a convenient time to reload. But that carries the caveat that there is the ABILITY to reload. In your situation above, I would not have reloaded until the data was 3-way verified. Doing so increases the risk, and data management is really risk mitigation.

Im still thinking about the technical issues with doing the copy first, but I submit that has nothing to do with the issues above.

The next version of R3D Data Manager will come with a ETA, so you can better guess when the download will be done. In addition, if your download station has internet access and you have an email account setup in Mail, it will send a SMS text message or email to you to let you know its finished and the finished state (good/warn/bad).

By the way, I still hire that data manager, because he will never make that mistake again.

As an aside, the amount of time it takes to create a checksum on modern hardware should be the exact same amount of time it takes to copy the data via the same medium. The processor is usually waiting on data to come in, not the other way around.

 

[Nick Shaw]

Well, I respect your situation. That particular situation and other similar ones have happened on sets Ive worked. But I respectfully submit that it is the workflow, not the tool, that is the issue here.

I agree that it's the workflow not the tool which is "wrong" in the situation I describe. However I would like the software I use to give me the option to take a risk if the requirements of the shoot dictate, and all concerned understand the nature of the risk they are taking.

As an analogy, I prefer traction control on a car to have a switch to turn it off. That way I can choose to take a risk in return for a performance benefit, rather than have a computer tell me "You can't do that. It's not the approved approach"!

 

[Conrad Hunziker]

Let me think about it more and do some more simulations. Beyond the first performance decrease I mentioned above, I havent thought of another issue (but it is bedtime so Im not preforming well ATM), but let me run some test first to make sure.

Edit: spoke too soon. It will need to recalculate the checksums of footage retained on the drive, regardless of if it has been copied already. So if in the middle of creating checksums you decide to pull it, it will need to start that process over. This is because the checksums for the data can no longer be trusted to be accurate after they have (perhaps) been modified by another file system. So in reality, in the situation where you decide to pull it before the initial checksum completes, then it should really re-copy the entire drive in addition to re-checksum the drive, as perhaps the footage was screwed up by the other file system.

So you'll need to have 2 copies of the footage and 3-way verify the last version before you manually(!) delete the first copy. I dont like it, but thats what Im thinking would be necessary to completely mitigate all the risk when pulling the drive before the initial checksums are complete.