Released: R3D Data Manager for Mac v6.1

[Conrad Hunziker]

BTW. Is it possible to use r3d manager on network-volumes.

G

R3D Data Manager will read/write to any mountable filesystem. If your comuter sees it, R3D Data Manager will read/write to it. I wouldnt advise writing to MobileMe, however.

 

[joe hedge]

Good luck with that. Because you had to drag through all sets of copies:

1) It took you longer to complete the entire process. Because we use checksums, you know that both files are the same on both copies, and therefore only need to check one copy.

2) You may have missed frames or partial frames that have errors, even if these errors were not caused by the copy process. For proper data management you must watch the footage in real time, regardless of what software you use to copy it.

I'm sure R3D does lots of stuff SPP doesn't, and I will probably be getting it sometime in the near future, but for on-set back-ups, SPP has never let me down. And it hauls ass. That log I posted shows 124 gigs written in 26 mins from FW800 to 2 eSata drives, which I didn't even think was possible.

 

[Brandon Kraemer]

I'm sure R3D does lots of stuff SPP doesn't, and I will probably be getting it sometime in the near future, but for on-set back-ups, SPP has never let me down. And it hauls ass. That log I posted shows 124 gigs written in 26 mins from FW800 to 2 eSata drives, which I didn't even think was possible.

All I have to say on this subject is if you are not check-summing your media on set from the original camera master drive/card, before you recycle your drive/card... you are asking for problems. Speed is nice, but corrupted frames are a nightmare and most often the media on the camera master is fine, it's getting lost in the transfer. Our production company mandates that DIT/DMT's use a checksum to at least three separate volumes before media gets recycled. We've been burned by bad card readers and sloppy DMT work once. Once is all it took.

bk

 

[joe hedge]

All I have to say on this subject is if you are not check-summing your media on set from the original camera master drive/card, before you recycle your drive/card... you are asking for problems. Speed is nice, but corrupted frames are a nightmare and most often the media on the camera master is fine, it's getting lost in the transfer. Our production company mandates that DIT/DMT's use a checksum to at least three separate volumes before media gets recycled. We've been burned by bad card readers and sloppy DMT work once. Once is all it took.

bk

"Bad card readers and sloppy DMT work" being the operative phrase here...

I'm usually syncing sound to picture too, on-set, with the transcodes from the day before, that I made overnight. Admittedly that's too late to catch copy errors, but out of the hundreds of camera drives and cards I've backed up, the only errors I've seen were unreadable files that came that way out of the camera, or transcode errors, which are fixed simply by re-transcoding. I've never, repeat never, had a shot end before the director's "cut" that was the fault of the file transfer. All with SPP. If it had failed even once I would have ditched it.

And there are many times when speed is not just nice, but critical - not a higher priority than proper DMT, but if a drive/card goes down or becomes unreliable, and now you've only got 1 drive or a couple cards to work with...you get the picture (no pun intended).

So I don't doubt that R3DM is a great product. There is such a thing as overkill, though...

 

[Brandon Kraemer]

I've never, repeat never, had a shot end before the director's "cut" that was the fault of the file transfer.

Problems that would have been caught by checksums that I have personally been handed were not truncated scenes where the end of takes are missing, rather single corrupted frames with in whole takes... not necessarily noticeable by reviewing playback. This was due to (likely) to a faulty card reader. Crew had a finite amount of CF cards, so media was recycled before the sun set on that day, blowing away chances to recover the takes.

The result is a nightmare in post, where the proxy crashes FCP anytime it runs over a bad frame, or crashes Redcine in the transcoding process, or renders a funky looking frame in place of the intended image and is not noticed until late in the game by an editor.

Just wanted to share my experiences and why we only use checksums. Since we mandate them, we make sure the crew is aware of the turn around times so the expectations or resources are in place to accommodate them. To each their own methods.

bk

 

[Gunleik Groven]

So I don't doubt that R3DM is a great product. There is such a thing as overkill, though...

I had SPP reporting two copies as "good" from a transfer to two esata disks, where there was a bug on one of the transfers (not on the card) and the other was good. SPP reported "copied and verified".

Thus, I have convinced the production to give r3d manager a try...

 

[joe hedge]

Problems that would have been caught by checksums that I have personally been handed were not truncated scenes where the end of takes are missing, rather single corrupted frames with in whole takes... not necessarily noticeable by reviewing playback. This was due to (likely) to a faulty card reader.

Would a checksum tell you if there were bad frames on the Red drive/card by comparing it to the transferred footage? Wouldn't they be the same? And if so, wouldn't the only way to catch it be by visually QC'ing before reformatting?

 

[David Battistella]

How would you treat this if it was a film negative?

Would you say a black bag with a tiny hole in it "good enough" for production. There are always demands, there are always compromises.

Personally, I would not choose speed over accuracy in this case.

This is the negative, why take chances.

To me it's not even debatable.

David

 

[Jody Neckles]

Wonderful! Testing now.... nice to see CF reader downloading EVEN FASTER!!! heh heh heh

 

[Harry Clark]

Conrad,
Thanks for the update. Very generous to keep it free whenever possible.
Not sure why some people bash this program. It's a really great tool...
Cheers,
Harry

 

[Tim Bradley]

I as a DP have done a few shoots where a data guy is supplied that knows almost nothing, but can use a computer. I always bring my Mac Bookpro with R3Ddata manager on it as well as a few drives and cables etc. With all this I can be sure that the data has been transfered and checksummed properly. Its a great product that takes away any doubts to the data copying. It means I can spend my time doing what I should be doing rather than checking the data guy. Thanks Conrad!!!

 

[Brandon Kraemer]

How would you treat this if it was a film negative?

This is the negative, why take chances.

A good analogy David, and we are actually spoiled for opportunities at redundancy thanks to technology. As a former AC I'd always worry about the film until I knew it looked good on the telecine. We didn't have clones or a tool that could verify what we had was whole and without flaws.

 

[David Battistella]

A good analogy David, and we are actually spoiled for opportunities at redundancy thanks to technology. As a former AC I'd always worry about the film until I knew it looked good on the telecine. We didn't have clones or a tool that could verify what we had was whole and without flaws.

Yeah,
To me this is a complete no brainer. Protect your negative with a really reliable piece of software that is written for the task.

If a DMT/DIT does not have this inexpensive piece of software in their kit I just can not take them seriously.

David

 

[James Burton]

As a data guy who knows almost nothing, as Tim rightly points out, what I would do to make a reliable and fast copy is to use the system commands directly.
For example, open apples terminal utility and type :

cp <source file name> <destination file name>

then verify that the files are EXACTLY the same by :

diff -q <source file name> <destination file name>

This is better then a check sum, as it checks EVERY byte.
It is possible that two files can be different and still have the same check sum, though this is unlikely, but hay, I like to be careful with computers because I know from experience that they don't like any of us people.

 

[Conrad Hunziker]

Conrad,
Thanks for the update. Very generous to keep it free whenever possible.
Not sure why some people bash this program. It's a really great tool...
Cheers,
Harry

When I first started selling R3D Data Manager I offered free lifetime upgrades. Now I am sticking to my word, and lots of people are reaping the benefit.

 

[Conrad Hunziker]

As a data guy who knows almost nothing, as Tim rightly points out, what I would do to make a reliable and fast copy is to use the system commands directly.
For example, open apples terminal utility and type :

cp <source file name> <destination file name>

then verify that the files are EXACTLY the same by :

diff -q <source file name> <destination file name>

This is better then a check sum, as it checks EVERY byte.
It is possible that two files can be different and still have the same check sum, though this is unlikely, but hay, I like to be careful with computers because I know from experience that they don't like any of us people.

Actually, I considered that when I was writing version 0.9 (over two years ago). But theres a bunch of ways that is actually worse:

- It does not provide a method to re-verify footage at a later date. Once the original is gone, you have to trust that the copied data is correct. With a failing hard drive and a valuable shot, would you trust it?

- This method is actually more resource intensive in my studies earlier. Its harder for the system to calculate a binary diff than it is to calculate a checksum.

- This method is limited to the bandwidth of the slowest device, since it needs to read in series on both devices at the same time. For small files its not noticeable, but on larger ones its quite evident.

Plus, you insinuate that the checksum method doesnt check every byte - which is FALSE. Checksums are a specific method of adding and dividing the binary value of each and every zero and one to come up with a representation of that number. If a single zero or one is out of place, the representation is vastly different.

Checksum values are better because they are saveable so you can re-verify later, can run at the speed of the bus the device is attached to, and is less resource intensive.

There is a chance that two identical files with different content have the same checksum value - but that chance is 10^-32 with md5. Thats a 1 preceded by 32 zeros, or one in 100 nonillion (100 thousand billion billion). Its 10^-128 with sha1 and its not yet known with sha256 because noone has done it.

In addition to the above, R3D Data Manager also checks the source with itself so that you know if the source is having issues reading the same data twice.

 

[James Burton]

Hi Conrad,

I have no argument with you. Obviously your program works well. Just saying that I would do it differently, thats all.

>> - It does not provide a method to re-verify footage at a later date. Once the
original is gone, you have to trust that the copied data is correct. With a failing hard drive and a valuable shot, would you trust it?

Once the original is gone, sure, no way anyone can recheck it. For me, being the very careful guy that I am, I would therefore keep the original for as long as I could. Drives cost very little after all.


>> - This method is actually more resource intensive in my studies earlier. Its harder for the system to calculate a binary diff than it is to calculate a checksum.

Of course, but then again, computers have no feelings and people do, so I prefer the computer to work hard if needed. Comparing each byte individually is more work sure, but can be done in a multi processor, if time is important. As I think you are saying, using finder is not such a great idea.

>> - This method is limited to the bandwidth of the slowest device, since it needs to read in series on both devices at the same time. For small files its not noticeable, but on larger ones its quite evident.

Sure, if what is being copied is just one large file, this would be hard to break up ( though possible ), but typically, what is being copied is many small files.

>> Plus, you insinuate that the checksum method doesnt check every byte - which is FALSE.

Sorry, I meant that a check sum is an abbreviation for a byte, not an exact bit by bit compare.

>> Checksums are a specific method of adding and dividing the binary value of each and every zero and one to come up with a representation of that number.

Exactly.

If a single zero or one is out of place, the representation is vastly different.

As you say further on, that is usually the case, but not always.

>>Checksum values are better because they are saveable so you can re-verify later, can run at the speed of the bus the device is attached to, and is less resource intensive.

Yep, it's often faster.

>>There is a chance that two identical files with different content have the same checksum value - but that chance is 10^-32 with md5. Thats a 1 preceded by 32 zeros, or one in 100 nonillion (100 thousand billion billion). Its 10^-128 with sha1 and its not yet known with sha256 because noone has done it.

OK, we agree then, there is a chance that a checksum might not pick up a difference, ever so small a chance.
Probably because I have seen so many problems with computers over the years, I get very careful with them.

>>In addition to the above, R3D Data Manager also checks the source with itself so that you know if the source is having issues reading the same data twice.

Good idea.

For a computer guy like me, I'd do that myself, but for non computer people, I am glad you have a program for them to run.

All the best,
james

 

[Conrad Hunziker]

Once the original is gone, sure, no way anyone can recheck it. For me, being the very careful guy that I am, I would therefore keep the original for as long as I could. Drives cost very little after all.

Um - there are only very specific drives that work on the Red One camera, ones manufactured by Red. These are quite expensive compared to normal consumer drives that youd be copying to. Unless you can afford to buy a couple thousand 16Gb CF cards (and supposing Red had a couple thousand lying around to sell you), you *MUST* overwrite the camera original at some point.

Id rather have checksums rather than a diff printout. Because I used checksums, I know footage I shot over 2.5 years ago is still as the camera wrote it, even though Ive overwritten the original several thousand times by now.

Of course, but then again, computers have no feelings and people do, so I prefer the computer to work hard if needed. Comparing each byte individually is more work sure, but can be done in a multi processor, if time is important. As I think you are saying, using finder is not such a great idea.

Taking the more expensive route (in terms of work to be done) is generally not good programming. I dont know about your computers, but my 8-core on-set mac has 5 minute load averages in the 5's and 6's. Not only am I doing data copies, but Im transcoding material and viewing material, generally at the same time. So Id prefer to that the less computationally expensive route with checksums, so more processor time is devoted to other tasks. Checksums wont even take up a core of that, whereas diffs do.

Sure, if what is being copied is just one large file, this would be hard to break up ( though possible ), but typically, what is being copied is many small files.

Actually, no. The Red One records at ~2Gb/min for 4k. The only reason the files are not larger is because of the FAT32 filesystem. The important files are the large ones. Yes, theres a bunch of smaller files (proxies, RMD), but R3D Data Manager has a specific option to ignore errors with those smaller files, thats how un-important they are.

On the current show Im working on, ONE copy of the footage will be around 60TB. That doesnt include transcodes, second copies, copies to VFX, archives, etc. These are large files, and theres a LOT of them.

Sorry, I meant that a check sum is an abbreviation for a byte, not an exact bit by bit compare.

But a checksum is a bit by bit compare, when compared with another checksum.

OK, we agree then, there is a chance that a checksum might not pick up a difference, ever so small a chance.
Probably because I have seen so many problems with computers over the years, I get very careful with them.

You realize that my 80Tb raid doesnt even have 1 nonillion bits. For perspective, I could write a 1-bit file to fill my entire raid 80 thousand times before I run into the statistical chance that that the checksums might match. Google might be coming up to the statistical limit where they have 2 files with the same size and different checksums on their massive storage system.... might. And if you want better odds than that, use SHA1.

 

[James Burton]

Hi Conrad,

I think you raise an interesting issue with what is good enough.

>> You realize that my 80Tb raid doesnt even have 1 nonillion bits. For perspective, I could write a 1-bit file to fill my entire raid 80 thousand times before I run into the statistical chance that that the checksums might match. Google might be coming up to the statistical limit where they have 2 files with the same size and different checksums on their massive storage system.... might. And if you want better odds than that, use SHA1.

For me it is reminiscent with the whole analog vs digital thing. What is "good enough" for the audience is basically an emotional ( analog ) experience for them, so whether the frame size is 4k or 5k may or may not be that important; it is a matter of opinion I suppose.

However, computers are logical, no emotions and software bits are either on or off. There is no grey. So for me, when my life or job is on the line, I do what I can to be as safe as I can. No excuses. I want to be in a position if asked, could I have been more careful with my copy, I want to be able to say that I could not have done any better. Sure, maybe we don't usually have to be as careful as possible, but when so much is at stake .... like when I drive my car along a cliff road, I am one of those people who drive as far away from the cliff as possible even though many tell me I need not bother, and as checksums have a small chance of failure, a tiny tiny chance of failure as you say; I don't like it. But that's just me.

Same goes for that re-reading of a drive that you do. A good idea. The mean time between failures of a good hard driven 24x7 might be 5 years, but I agree with you that it does happen ( even if only every 50 years ), its worth checking.

Good luck with your product sales. As I said, it should be used by most people, especially if they are not computer people.

cheers
james

 

[Conrad Hunziker]

...and as checksums have a small chance of failure, a tiny tiny chance of failure as you say; I don't like it. But that's just me.

Your diff method is used by other software, with negative results. They use a read-back method, which is also bit by bit comparison. I dont know if they specifically use the diff binary, but its a very similar methodology.

For all the black and white computers are, Im sure you know its actually a world of grey. There is no absolute 100% works all the time, never fails method. Something will always fail, be it the reader, the system, the ram, software, whatever. The process of on-set DAM is mitigating failure.

Same goes for that re-reading of a drive that you do. A good idea. The mean time between failures of a good hard driven 24x7 might be 5 years, but I agree with you that it does happen ( even if only every 50 years ), its worth checking.

You arent familiar with doing on-set data management, are you?

CF cards/readers/system fail to read the same data twice the same way often enough to cause serious issues for those that are not prepared. This isn't a once-in-5 years event, but on some (particularly bad-luck) shows a daily/weekly occurrence. There are whole threads here on reduser about corrupt frames from bad CF reads. Reading twice from the source is a way to combat the possibility of a bad read.

I realize what you are saying about your computer skills being good enough for your job. But for the specific purpose of on-set data management, my experience, both with shows Ive worked on and on set visits for other shows, shows those to be bad methodologies.

And one last fact: if your show is bonded, the bond company will have specific requirements for how the data is dealt with. Diff's do not satisfy most bond company requirements.

Checksums are a (computer) industry standard method for comparing data. They are used from the most basic CRC check reading data from platters to ECC in RAM to the TCP stack. They are used in high level associations like checking downloads werent tampered with or code signing. Those are all different implementations of the basic checksum methodology. None of those use a diff methodology. Diffs are a good way to make two files on two different systems exactly the same - like a local and remote file. Otherwise, they are not the greatest for what you are trying to do.