Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Help - Offloading Red Drives - Speed vs. Safety

Dave Weber

Dave Weber

Well-known member
Joined
May 17, 2007
Messages
907
Reaction score
0
Points
16
Location
Orange County, CA
I was on a shoot the other night shooting interviews. When I went to offload the first drive through R3D Data Manager to 2 drives it gave me a time of about 2 hr 30 minutes for the Sata drive and over 4 hours for the FW800 for 126 GB. I ended up just doing a drag and drop transfer which took less time but not as safe without sum check.

What are you finding as the Fastest/Safest way to offload RED Drives?
 
Hi Dave,

Many will call me insane, but I always just do a drag copy to a SATA drive, and then clone the drive. Never had a single issue with corrupted data. Twice I have had Red Drives fail, but nothing to do with the copy process and was before Red Undead.

Having said that, I always check the footage before allowing the drives to be erased.

Since I work on multi cam productions I work with up to 10 Red Drives at a time, usually full.

As I said, never a problem so while the risk is always there (as with anything) my personal experience is that there isn't a measurable danger in copying.
More about insurance and peace of mind.
 
Just do a get info on folder you copied over and the copy and make sure the bit sum (the long number) is identical - this is just as accurate as the fastest checksum that R3D data manager offers. If the bit sums are different, copy over again.
 
Alexis Hanawalt said:
Just do a get info on folder you copied over and the copy and make sure the bit sum (the long number) is identical - this is just as accurate as the fastest checksum that R3D data manager offers. If the bit sums are different, copy over again.
Click to expand...

Hate to say it, but this is EXACTLY wrong.

You can have two different files with the same exact size with different contents. Are those contents important to you?

The 'bit sum', which I think you are referring to the folder size, is just a addition of all the sizes of the files contained in that folder. That says nothing about the content of each folder being exactly the same. The folder size is NOT the same as a checksum. Anyone who says that is dead wrong.

The only way to ensure that the data is exactly the same is to read it back on all destinations, and ensure that it reads exactly as the source read. Ask anyone who has had errors how important that is.

So the question really is, how important is your footage? If its you and your buddies, then who cares. If its a network show, Id certainly make sure all my ones and zeros were correct before you erase the source.
 
Dave Weber said:
When I went to offload the first drive through R3D Data Manager to 2 drives
Click to expand...

The issue is how you had those drives hooked up, and the throughput on the source. I dont know how you had the drive hooked up, but Im assuming it was FW800. In that case, copying to multiple drives at once divides the total bandwidth available to copy, so it will take a little more than twice as long. So in this case if you copied to a single drive twice, you would see it go significantly faster.

The next version of R3D Data Manager, due out shortly, will allow you to copy to several destinations at once, with no speed penalty. So the above would not apply.
 
My apologies, it's "byte sum" under get info. The odds two folder's having this same number yet having different content are REALLY low, right? What is the quickest option in R3D data manager doing differently? My guess was that is was checking the byte sum of each individual file, rather than the whole folder.
 
R3D Data said:
The issue is how you had those drives hooked up, and the throughput on the source. I dont know how you had the drive hooked up, but Im assuming it was FW800. In that case, copying to multiple drives at once divides the total bandwidth available to copy, so it will take a little more than twice as long. So in this case if you copied to a single drive twice, you would see it go significantly faster.

The next version of R3D Data Manager, due out shortly, will allow you to copy to several destinations at once, with no speed penalty. So the above would not apply.
Click to expand...


Conrad,

Yes that is the case. I had eSata to one and FW800 to the other. I wish you could daisy chain esata drives.

Looking forward to the next version. Free upgrade? =) I love your product it's a great tool.

thanks again,
 
Alexis Hanawalt said:
My apologies, it's "byte sum" under get info. The odds two folder's having this same number yet having different content are REALLY low, right? What is the quickest option in R3D data manager doing differently? My guess was that is was checking the byte sum of each individual file, rather than the whole folder.
Click to expand...

I think you may be confusing what sums are.

Byte sum is the addition of the number of bytes in a file. This would give you a size for a folder, but is not unique to that folder. There may be multiple folders with different content but the same byte sum.

A checksum is the mathematical method to add the zeros and ones of a specific file to create a unique number for that file. Its basically reading a file, taking the zeros and ones, adding them up and dividing in a specific method, to come up with a 32 (or 128 or 256) bit number that represents that file. If a single zero or one is out of place, switched or otherwise mangled, the 32 bit representation is vastly different.

By using this method we can say that the content of two files are exactly the same, as opposed to saying that two files are the same size.

Ive used R3D Data Manager personally to transfer 6Pb of material to date (rough estimate). In that time Ive had 9 errors. Sometimes it was a stupid human error. The majority of the time it was a hardware failure - a reader error or a write error. Things that drag & drop simply dont have the ability to detect.

Computers are just machines. They fail. By using a checksum method, you will know when it fails when you can still do something about it. To me thats far better than the editor calling.....
 
I know

I know

Brook Willard said:
The things that I've seen go wrong with drag & drop transfers would make your skin crawl.
Click to expand...

I'm sure. I hate doing it, but (knock on wood) I've had no trouble the times I have done it. :shocked:
 
Dave Weber said:
I'm sure. I hate doing it, but (knock on wood) I've had no trouble the times I have done it. :shocked:
Click to expand...

I know that not all of us agree on this one. There are many of us working professionally on all levels of productions that have various perspectives on data transfer. To date, I do not know of a single frame of R3D files that has been lost with the method I choose to use. So, for me that's what allows me to sleep at night. That method happens to be R3D Data Manager and the MD5 Checksum process. Other great minds, such as Ian Bloom have argued for drag and drop as being a perfectly valid method. Ian is a very smart guy. He knows a lot of the under-the-hood stuff. He has his methods.

Whatever works for you, I suppose that's what you go with. If there are errors, regardless of the method used, you will have to answer for it. So that's why it's important to have confidence in the method you choose because if that day comes when you get a call from post saying that File XYZ has corrupted frames, they will not really care whether it was done with R3D Data Manager or drag and drop. They'll want to know why they don't have their frames, i.e. why didn't you make sure the frames were all intact on set.

Again, how you get there is up to you. Drag and drop with visual inspection, a 3rd party verification system such as R3D data, or by telepathy. Do lots of tests. There are those of us that have seen drag and drop fail. There may be others who have seen the 3rd party apps fail. I know which one works for me. That's why I've stuck with it.
 
Steve Sherrick said:
Again, how you get there is up to you. Drag and drop with visual inspection, a 3rd party verification system such as R3D data, or by telepathy. Do lots of tests. There are those of us that have seen drag and drop fail. There may be others who have seen the 3rd party apps fail. I know which one works for me. That's why I've stuck with it.
Click to expand...

I agree Steve.

So what I'm getting from you all is that as of now, I am using the "quickest/safest" method through R3D Data Manager.

thanks for all of your input. I appreciate the work you do and the suggestions you give!
 
Dave Weber said:
I agree Steve.

So what I'm getting from you all is that as of now, I am using the "quickest/safest" method through R3D Data Manager.

thanks for all of your input. I appreciate the work you do and the suggestions you give!
Click to expand...
In terms of speed, making sure your data pipelines are as fast is possible will get you faster transfers. If you are using a laptop and it has an express34 slot, then you can add an esata card. This is a much better solution than using the one FW800 bus on the motherboard. When you move over to a tower, there are even greater possibilities, as you have far more options for expansion. You can create quite powerful data transfer pipelines.

In terms of safety, again R3D Data is what I use and it is what I consider safe for myself, but I can't say this is the only method. There are others out there. And a lot of this can be done if you know how to get under the hood of your system, but Conrad has provided a GUI that makes it more practical for on set, pressure cooker scenarios. And there are several useful tools that come with the app as well. The price is also a no brainer.

Now, if you run into a situation where time just won't allow checksum verification - find out if there are ways to make the transfers more efficient throughout the day, i.e. swapping media more often. You still have to find a way to verify that the data is there and there are no errors. So if you use drag and drop, you'll have to at the very least use scrub playback verification and in the best case scenario you will watch through the clips in realtime. A sure way to find yourself on the phone with an upset producer is to just drag and drop, never inspect/verify the integrity of the data, and then hand it over. Not a good idea, but one that is done.
 
Yeah I'm with Steve 100%- one frame of footage lost can mean your entire reputation blown. Nobody wants that.

Noah
 
As I've come to find it really does depend on the eSATA card you use. Bought the Sonnet Tempo for at work, and its blazing fast, reliable and haven't had a problem since buying it last year no matter what i throw at it. I bought an el cheapo one for personal use, and it was/is garbage, after one 500gb transfer between drives its now topping out at a whopping 2mb/sec.
 
This is like a filmloader telling me they don't use a changing tent because doing it in the light is faster.

IT'S THE GODDAMNED MASTER FOOTAGE! DON'T TAKE CHANCES.
 
In my opinion whatever method you want to verify your data is fine...but just make sure you do so. The way I've sped up my workflow recently has been through clever scripts and hardware. I'm now using a LEMO to ESATA cable that I find to be about 30% faster then FW800 when connected to a newer Mac Pro. I also have a ESATA CF reader.
 
If I cannotuse R3d DM, there is another potentially 'safe' option. I DO NOT Drag and Drop. I use cp -R in Terminal , as a Backup. Finder can crash, gui's are goofy sometimes. A system command is a tiny bit faster transfer than Finder and will continue transferring even if Finder Crashes. Plus it can be done without an operating system loaded to a MacPro... by booting single user. I do cp -R for any media that is not R3d, like EX3, P2, 5D, etc... And in the situation where Multiple Cameras exist I try to get enough drives that I can Checksum through Terminal after the transfer is complete to save a little bit of time and save the md5 output of Source and destination to a txt file and leave it in the root of the Mag.

The other check I have is doing an 1/8 res Debayer on an entire Mag after transferred. If a corrupt clip exists the frame count will not match. I create a snapshot of the finished Batch window of RedRushes to prove that footage is not corrupt. I never offer these files to Production, they are usually horrible.
 
The last show I did I used R3D and then scrubbed through all the footage. If something hinky happened with the camera, transferring or when I scrubbed I would transcode a 1/8 debayer just to be sure. R3D takes 33% longer I believe but it's well worth the added security. If you catch a problem while you are still at the location you look golden. If the assistant editor catches it....death :)
 
You must log in or register to reply here.
Back
Top