Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Released: R3D Data Manager for Mac v6.1

Sam Crutsinger said:
When it works, it seems to knock it out fantastic, but I've started to occasionally get straight errors in the logs. I always get Copy OK, but then:
"Source verify after copy Error!" and "Destination verify Error!"

So I delete the folder and run it again. This time I get straight down the line:
"Source verify after copy Error!" and "Destination verify OK"
Click to expand...

The source verify after copy error means the source did not read the same data twice the same way. Something in the file changed so it wasnt the same checksum.

The destination verify error means the destination checksum didnt match the source checksum.

Its up to you if you want to ignore them, but please make sure you are visually verifying the footage.

Really, it sounds like either your system is dieing, or there is some other process interrupting the copy. Or I suppose it could be a bad stick of RAM, since 6.1 is more RAM dependent.

But I would advise moving off this equipment until you get it all sorted.

BTW - 6.0 and 6.1, the new copy process, is pretty well tested. I use the code on all the red shoots, and theres a select bunch of beta and heavy-use testers, in addition to all those who have upgraded since NAB. None have reported this issue.
 
Sam Crutsinger said:
Oh, and I'd like to request that eventually R3DDM should get tweaked for multithreading. I dump my footage to a RAID-6 capable of 700MB/s access and the Mac has 16 virtual cores (8 actual) available to chew up those destination checksums. It looks like the program never goes over 100% when I have 1600% available. I can understand the FW800 bottleneck, but once I get it onto the RAID, this box has seriously wide bottles.

Never satisfied. Always wanting faster. It's a testosterone thing.
Click to expand...

Just BTW, R3D Data Manager is very multithreaded. During a destination verify theres about 20 to 30 threads going on, depending on how many destinations you have.
 
I just did a quick test with R3D Data Manager vs Finder. The setup was:

Source: RED 16GB CF, 4.39GB of media, SanDisk FW800 reader

Destination A: 8 drive SCSI RAID5

Destination B: 2 drive SATA RAID0 (internal to Mac Pro)

R3D Data Manager copy time (MD5 checksums): 3'35"

R3D Data Manager copy time (no destination checksums): 3'00"

Finder copy time: 2'15"

I do realise that the Finder copy is unverified, and that R3D Data Manager is going on to do a bunch of other stuff after it finishes copying.

However, I would say that my take-out from this is that the claim that R3D Data Manager is "faster than Finder" is somewhat misleading. It would appear to be based on a specific set of test conditions designed to play to R3D Data Manager's strengths, and Finder's weaknesses. But hey, that's marketing!

That said, my initial impression of v6.1 is generally good compared to my experience with older versions. So I will probably end up buying a copy. The big change for me is the new "copy first and then verify" philosophy. To me it should always have been this way.
 
Sticking with 5.4

Sticking with 5.4

R3D Data said:
Its up to you if you want to ignore them, but please make sure you are visually verifying the footage.
Click to expand...

I'm not ignoring anything. If there's not a green check, they don't get the drive back. I'll hold up the shoot before I'll let them burn a drive without a validated copy.

R3D Data said:
Really, it sounds like either your system is dieing, or there is some other process interrupting the copy. Or I suppose it could be a bad stick of RAM, since 6.1 is more RAM dependent.

But I would advise moving off this equipment until you get it all sorted.
Click to expand...

If it was one computer, I'd think you were on to something, but it's happening on 2 very distinct systems. I've started marking the drives that cause problems to see if there's a flaky RedDrive that's at fault, but just now a 2nd drive has erred out while the drive that I marked last night as giving me grief had no problems.

I tried another test. I captured a card with 5.4 and captured the same card with 6.1. Same computer. Same card. Same cables. Same everything but the software. 5.4 gave a green check. 6.1 gave straight errors all the way down the line. Then I flopped the R3D and proxy files so the files from the 5.4 copy were in the 6.1 folder and vice versa and I had 5.4 validate the checksums. Both passed. I'm pretty sure that means the files are identical.

I'm just sticking with 5.4 for the rest of this shoot. With the alternate folder for the initial checksums, it's a read only process just like 6.1. I can live with a little slower copy if it means I'll not deal with whatever voodoo is causing this mayhem. After this shoot ends, I'll revisit.
 
2 eSata 7200 1 tb drives via express card 34:

15.14gb, full CF card via lexar fw800 CF reader

R3d dual destination with md5.... 5 mins.

Very well done Conrad.
 
Conrad,

It seems that R3D Data manager has a limit to how fast it can copy. Why is that?

I just did a test copying from one RAID to another.

Finder is showing 120MB/s in Activity Monitor
3cP is showing 120MB/s in Activity Monitor
R3D is showing 65MB/s in activity Monitor

The other day when I was downloading some footage on set, I noticed that R3D was getting 65MB/s

Why does R3D Data Manager seem to have a speed limit?


Thanks,
Dusty
 
Dustin Cross said:
Why does R3D Data Manager seem to have a speed limit?
Click to expand...

It doesnt have a speed limit in the code.

But R3D Data Manager is doing things those other programs are not. For each bit of data it needs to read from the source into RAM, calculate the checksums, deliver it out to the various destinations and verify the data made it to each destination. The limitations when you have unlimited bandwidth become the speed of your RAM and processor.
 
J. Eric Camp said:
Two 7200 RPM 3.5" drives in sled enclosures connected to 17" MBP via Sonnet pro FW800 express card.
R3D data manager
2 copies with checksum : TRT 17:34
Click to expand...

J. Eric Camp said:
2 eSata 7200 1 tb drives via express card 34:
R3d dual destination with md5.... 5 mins.
Click to expand...

eSATA does make a huge difference. I suspect that finder would be just about the same amount of time for those eSATA destinations concurrently.
 
Nick Shaw said:
R3D Data Manager copy time (MD5 checksums): 3'35"

R3D Data Manager copy time (no destination checksums): 3'00"

Finder copy time: 2'15"

I do realise that the Finder copy is unverified, and that R3D Data Manager is going on to do a bunch of other stuff after it finishes copying.
Click to expand...

Theres also stuff going on during the copy - calculating checksums, verifying source checksums, ensuring the data was written to all destinations. None of that happens with finder.
 
R3D Data said:
It doesnt have a speed limit in the code.

But R3D Data Manager is doing things those other programs are not. For each bit of data it needs to read from the source into RAM, calculate the checksums, deliver it out to the various destinations and verify the data made it to each destination. The limitations when you have unlimited bandwidth become the speed of your RAM and processor.
Click to expand...

But I am seeing the exact same speed limit when using a 15" Macbook Pro and Firewire as I am using a tower with eSATA RAIDs.

There should be a difference between those two systems. Firewire should not be fast enough to saturate my RAM and the processor is not getting used.

Also, if I turn off check sums, R3D still does check sums?


Dusty
 
There is nothing in the code that creates a speed limit. But the RAM is getting used multiple times for the same piece of data. Its not about the firewire bus, but that there is a lot of RAM reading for different purposes.

There are several different places you need to turn off checksums for all checksums to be disabled - but thats only one less read from the RAM per piece of data.
 
If there is not something limiting speed why does R3D max out at the same speed on two completely different systems?

Firewire - Laptop memory - single eSata drive

eSata Raid 1 - desktop memory - eSata Raid 2

both max out at around 65MB/s.

The desktop should be faster. It reads from the drives faster, has faster memory, writes to totally separate drives faster, and the processors are faster. Everything is faster than the laptop, but R3D data manager is not faster. Do you have any idea why?


Dusty
 
Your RAM is not an order of magnitude slower on the laptop as opposed to the desktop.

As I said before, and Ill say it again, there is a lot going on in the RAM/processors that would place an artificial limit on speed. Its not in the code.

EDIT: Ok - it IS in the code - as we do have a lot of internal verification going on, between checksums and ensuring all the data is written to all the right places. If we took all that away, we'd have........finder or 3cp.
 
OK, now I'm baffled.

I have 2 SSD's - 256GB (Indilinx), 1 SSD 50GB (SandForce).

I have one folder full of R3D files, total of 30.8GB. These files are on the 50GB SSD.

My process

When I drag and drop this folder through Finder to both 256GB SSD's at the same time (give or take the three seconds it takes to start the second copy), it takes a bit under 5 minutes to complete the copy to both drives. This is amazing. On a Macbook Pro.

When I copy this folder using R3D Data Manager v6.1 - without any checksum .txt files created - just to see the real world copy process results - it takes 14 minutes.

Plus 2.5 minutes for "Source verify after copy" and another 3 minutes for "Destination verify" for a total of 19.5 minutes, using the fastest media available.


1) What part in my computer can I upgrade to speed this up a bit?

2) Is this the performance I can expect when Epic 1.8" SSD's become available?
 
Conrad,

Quick question... we have proxies that were successfully copied off of camera media to a hard drive, and now we are trying to copy those files and their associated .R3D files from that hard drive to a RAID. We can play the proxies off of the hard drive, but when we copy them to the RAID we get a "Caution" warning inside of .R3D Data Manager and the proxy files will not open. We tried copying with the finder and we get the same result. Any ideas what is going on? This has not happened before for us.
 
Conrad,

Any known issues using R3D DM to copy/checksum files to a SAN?

We were just running a process from an edit suite that seemed to time out after the first mag... and simultaneously on other edit suites connected to the SAN, our other write to SAN processes (like project saves, or finder copies) got crazy slow. Quitting R3D speed things up a bit... and then when we rebooted the machine running R3D... the pipes opened up completely.

Ideas?
 
R3D Data said:
There are no known issues with writing across SANs. I know of several posts houses that are doing it successfully, day in/day out. It sounds like a driver issue to me, but Im not familiar with your setup.
Click to expand...

Ok, thanks Conrad.

bk
 
You must log in or register to reply here.
Back
Top