Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

R3D data manager???

Trial Version

Trial Version

Just an FYI, we now have a trial version available for anyone who wants to see it. The trial version is good for 30 days.

If you would like to get you hands on a copy, please use the contact form on our website: http://www.r3ddata.com/contact/

Thanks.
 
Photocon, is there any way to speed up the md5 checksum? at the moment, I'm trying to speed up the data moving process as much as possible. Currently thinking of using 3 xserves with caldigit's hd raid to move stuff around and speed it up.
 
ayarbro said:
Photocon, is there any way to speed up the md5 checksum? at the moment, I'm trying to speed up the data moving process as much as possible. Currently thinking of using 3 xserves with caldigit's hd raid to move stuff around and speed it up.
Click to expand...

The MD5 checksums are bound by the speed of the wire (firewire, sata, etc...) and the speed of your processor. The MD5 process is simply an addition of all the ones and zeros in a file. So it has to read each bit of data to create that MD5 sum. So the faster you can make the read speed, the faster it will be.

Thanks.
 
Thanks for the reply! I'll get super fast machines with separate channels of ESATA then!

Photocon said:
The MD5 checksums are bound by the speed of the wire (firewire, sata, etc...) and the speed of your processor. The MD5 process is simply an addition of all the ones and zeros in a file. So it has to read each bit of data to create that MD5 sum. So the faster you can make the read speed, the faster it will be.

Thanks.
Click to expand...
 
Right now R3D manager calculates the MD5 checksum by reading the entire mag first, before copying. What if the software read the mag as it was copying, calculated the "read value" sum then calculated the "written value" sum on the destinations.

This would speed the copy process by 50% or more (on my system the Mag is the bottleneck - I'm writing to two independant eSATA drives and a Fibrechannel RAID)
 
Actually, that is already in the works. If you look on our development page, we are already working on reading/writing files to/from RAM. This would be the way to get around this issue.

The biggest issue in the entire copying process is the firewire connection to the RED Media. And if you have multiple destinations, the problem is exacerbated by every destination needing to access the same data. But by reading the files initially to RAM, you solve 2 issues:

- creating initial checksums is faster
- writing to each destination is at the DESTINATION line speed.

However, it creates a huge reliability issue. RAM is not exactly the best place to keep important files, and certainly easily exploited. In addition, you would need to have 2.5+Gb of RAM available after all the programs you are running are loaded and the base OS.

In our tests, it only increases the transfer speed by 30%. I know that 30% is a large percentage when you are dealing with time (and especially considering that multiple copies would be made during that time), but we need to program around the issues of reliability first. Because reliability is a major cornerstone of our program's success, we need to throughly work out all the possible scenarios with this before we release it.

Thanks.
 
You guys rock! Thanks for all the thinking photocon! I religiously use your software on every gig.

Oh...one more question...but I think you already answered it...
The CPU percentage devoted to r3d manager is fairly low, is there any way to make the r3d process a higher cpu priority? Would this speed it up at all? The reason I said you probably already answered this, is that I'm guessing the the firewire is still the bottleneck??? (like you said)
 
ayarbro said:
You guys rock! Thanks for all the thinking photocon! I religiously use you're software on every gig.

Oh...one more question...but I think you already answered it...
The CPU percentage devoted to r3d manager is fairly low, is there any way to make the r3d process a higher cpu priority? Would this speed it up at all? The reason I said you probably already answered this, is that I'm guessing the the firewire is still the bottleneck??? (like you said)
Click to expand...

Thanks for the compliments.

Calculating the MD5 sums (meaning actually doing the math) is a fairly simple process, easy enough for any of todays modern processors. However, reading the numbers to add is much slower. So increasing the priority of the checksum or copy process wont increase the speed at all.

One thing to consider in addition to that, is that typically people dont just use our software on set. In addition, they may be playing back RED footage. By keeping the priority lower, more processor resources are devoted to Quicktime, and therefore you get a higher playing fps (assuming you dont have a bottleneck in reading the RED files). So it is actually beneficial to keep the priority of the R3D Data Manager right where it is at.

Thanks
 
we bought R3D Datamanager. The idea is awesome, finally somebody built something that can do that AND copy to multiple destinations.
Using it however is a different story. Having an 8core and writing to a 600MB/sec. SAN while reading from the Mags is probably among the fastest ways to work with R3D-Datamanager.
STILL, the MD5 process takes ages and in my bloody eyes way too long.
Might not be fault of R3DData/Potocon whatsoever (no pun intended, really), but we dropped/stopped using the checksum stuff.
55 shooting days, 140hrs or more, 23TB of data and 1 defect clip.
right off the drive, had nothing to do with the copy process.

Standard OS copy routine (speaking of OSX here) does the job.
Well.
 
You could uncheck the MD5 checkbox and still use the R3D Data Manager's ability to copy data to multiple places without any MD5 checks.

However, you need to remember that a MD5 check is not just for the first copy, but for EVERY copy. Normally you'll copy that footage again and again, over the life of the project. Backups, duplicates, SPFX, archives. By having a MD5 sum, you know everything was copied accurately - as if it were the original mag they were using.

Which leads to the next point - if you are doing any sort of bonded project, they will require some way to verify the data is valid. Having a MD5 sums fits the bill. And combined with the audit trail, it satisfies most bond company requirements.

MD5 sums may not suit your exact workflow, but you can still use R3D Data Manager. And for those that are doing bigger shows with checksum requirements, they can use it also.
 
two questions.

two questions.

I have two questions about your software Photocon....

When I queue up my two backup drives to offload media from my REDdrives in R3D Data Manager, it seems as though there are two simultaneous copy processes going on...

What I gathered from the earlier posts was that a sequential copy method is faster than a simultaneous one due to the hard and fast transfer speed limitations of firewire400,800, eSata etc.

Why, then, do I see two status bars indicating two copy processes in when I click START COPY?

Lastly, and totally unrelated, the software doesn't seem to be recognizing my REDdrive as such. I am not able to edit any of the project info nor check anything in "When copying from RED media:". How might I get the program to properly recognize my RED media...

note: I haven't tested the CF cards for this though. Perhaps I'll try that out tomorrow when I get a good opportunity (today's takes would have been waaaaay too long for CF)

thanks.
 
ASG said:
I have two questions about your software Photocon....

When I queue up my two backup drives to offload media from my REDdrives in R3D Data Manager, it seems as though there are two simultaneous copy processes going on...

What I gathered from the earlier posts was that a sequential copy method is faster than a simultaneous one due to the hard and fast transfer speed limitations of firewire400,800, eSata etc.

Why, then, do I see two status bars indicating two copy processes in when I click START COPY?
Click to expand...

Im not sure where you see me say that R3D Data Manager does sequential copies, other than the upcoming queue feature. In the future version it will process sources sequentially. It always copies to destinations simultaneously.


Lastly, and totally unrelated, the software doesn't seem to be recognizing my REDdrive as such. I am not able to edit any of the project info nor check anything in "When copying from RED media:". How might I get the program to properly recognize my RED media...

note: I haven't tested the CF cards for this though. Perhaps I'll try that out tomorrow when I get a good opportunity (today's takes would have been waaaaay too long for CF)
Click to expand...

Most likely you are not selecting the root of the drive. You need to select the root of the drive to get the program to recognize its a Red Media. For example, /Volumes/A001_0707XS/

Thanks.
 
Quick Q: I want to move a whole portable drive, containing several mags onto my edit storage.

If I've copied a bunch of mags using R3D Manager onto a portable drive, can I then select the entire volume on the portable drive and have R3D Manager copy the whole thing over for me with checksums? Or do I have to select one mag folder at a time on the portable drive?
 
Forgive my rather simplistic questions, but I want to make sure I am using your software most efficiently...

The fastest way for me to move my data around then would be to transfer to my first backup drive, run checksums, then begin a second transfer to the second backup drive, correct?

If this is the case then an automated sequential transfer function would be most welcome in a future version! Also, in addition to a % complete indicator, I'd also like to see a "time remaining" indicator as when you copy via the Finder.

thanks Photocon.
 
Trevor Meier said:
Quick Q: I want to move a whole portable drive, containing several mags onto my edit storage.

If I've copied a bunch of mags using R3D Manager onto a portable drive, can I then select the entire volume on the portable drive and have R3D Manager copy the whole thing over for me with checksums? Or do I have to select one mag folder at a time on the portable drive?
Click to expand...

I've run into the same issue. I ended up using rsync for this task. It does make new checksums though but it is quite fast and fully automatic.
I even used the program to sync up two drives that had media scattered on each of them.
Rsync is a terminal program. No GUI. But free.

I whish there was a small utility that came with R3D manager that took care of copying media from the backup drive to other systems.
 
Trevor Meier said:
Quick Q: I want to move a whole portable drive, containing several mags onto my edit storage.

If I've copied a bunch of mags using R3D Manager onto a portable drive, can I then select the entire volume on the portable drive and have R3D Manager copy the whole thing over for me with checksums? Or do I have to select one mag folder at a time on the portable drive?
Click to expand...

The best way, right now, is to do one mag folder at a time. I havent had the chance to explicitly test what you are trying to do.
 
ASG said:
Forgive my rather simplistic questions, but I want to make sure I am using your software most efficiently...

The fastest way for me to move my data around then would be to transfer to my first backup drive, run checksums, then begin a second transfer to the second backup drive, correct?

If this is the case then an automated sequential transfer function would be most welcome in a future version! Also, in addition to a % complete indicator, I'd also like to see a "time remaining" indicator as when you copy via the Finder.

thanks Photocon.
Click to expand...

Fast is a relative term - because though it may be a couple of percent faster to do things sequentially, it is also slower because you actually have to be there to do it, waiting for one thing to finish and then start another. Also, there is the human factor, which is what the R3D Data Manager is trying to remove.

In a future version, we are planning on distributing r3d files via RAM, so that you get line speed (or at least system speed) on all your transfers. This should increase your transfer time by about 30% overall.

As for amount of time remaining - thats not so easy to figure out. If it was just figuring out how fast you are currently transferring and then simple division for time - that would be easy. But you have to factor in the checksum process, which heavily depends on the same processor that you are trying to calculate the time with. So while it may be better, it would also slow down the entire process.
 
sbaechler said:
I've run into the same issue. I ended up using rsync for this task. It does make new checksums though but it is quite fast and fully automatic.
I even used the program to sync up two drives that had media scattered on each of them.
Rsync is a terminal program. No GUI. But free.

I whish there was a small utility that came with R3D manager that took care of copying media from the backup drive to other systems.
Click to expand...

You could even do a finder copy then use R3D Data Manager to validate the files via stored checksums. Just select the mag folder on the final destination as the source and select "check checksums".

Theres so many ways to use the R3D Data Manager, that it really does fit into just about every workflow.
 
You must log in or register to reply here.
Back
Top