Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Copy & md5 Shell Script

Dustin I'd love to test the hell out of a windows version! I was very disappointed with a demo of Shotput Pro V4. Very slow and lots of errors with checksums, even though they were actually perfect.
 
Hey Dusty, had a new little issue pop up today. I am copying from a Red Mag SSD directly to (2) G-Drive Usb3 hard drives and using DaVinci Resolve 9 on my Mac Pro to view and CC the footage before transcoding. After the copy to the first drive is completed, I am playing back those files in Resolve from that same hard drive to show the client while your Copy Verified script is continuing to backup to my second G-Drive. Sometime after the second copy is completed and during the Destination Checksum, my computer shuts down and restarts. I just did a test where I am not playing back the footage from the first drive and just waited for the entire process to finish first, and there was no error. The computer stayed up and running. Any ideas? Last week, I did my first copy to my internal RAID first and then to the 2 backup drives and I had no issues playing back the footage once the first copy was done.
Thanks,
Matt
 
A G-drive is way too slow to be copying and playing R3Ds at the same time, so you should probably continue using your RAID for that. But a reboot is pretty severe. I'm guessing the drive spontaneously ejected itself. What USB 3 card are you using? Some cards freak out with lots of simultaneous I/O. Check sudo dmesg for errors.
 
Thanks Elliott. I'm using the Cal Digit Fasta Usb3/esata card and I have had issues with it causing a kernel panic when I tried to use a USB3 hub. I just purchased the new updated card from CalDigit but haven't installed it yet. That was probably the issue you mentioned. I'll use the Raid or just wait until the copy and checksum have completed before trying to playback the files.
 
Yeah, you know something's wrong with it when they come out with a new version that quick. I never had any issues though until recently.
 
Matt,

I have not experienced anything like that. Elliott's response sounds pretty good to me. I have noticed a huge difference in USB3 cards. Some are okay and some are terrible and none are good for bus power.


Dusty
 
setcopy is giving me favorable speeds compared to Silverstack SET — between 5 and 26% faster for Codex data packs over SAS. But I've only done a couple of data packs so far, so this may not be significant. And if I use CRC32 instead of MD5+, then of course the speed is even greater. Good work Dusty.

However, I'm still disappointed with overall speed of the Codex VFS. I haven't been able to get a straight answer from Codex about the bottleneck, but I suspect it's their VFS implementation in FUSE.
 
Without checksums, the latest version of setcopy actually seems a bit slow, which makes Silverstack look even worse. For example, setcopy moves at 225MBps with "-h none" copying from Codex data pack to a fast RAID. Command-line cp is the same. But drag-n-drop in the Finder moves at 235MBps in the same situation. I'm using command-line purge in between each test to clear the cache.

From an SxS card to the fast RAID, setcopy with "-h none" moves at 115MBps, while Finder does 121MBps. DoubleData also gets 121MBps.
 
Elliott,

Thanks for the feedback about the new setcopy app.

Did you compare with RSYNC like CODEX recomends? I have not tested that.

I have noticed Finder is faster in single copy. We looked a little to find out what Apple was doing to make Finder faster, but our focus was on multiple destinations and checksums. Copy the same data to two or three destinations in Finder and with Setcopy or DoubleData. Finder is much slower then.

That is interesting to hear DoubleData is same speed as Finder. So they figured it out. I guess we should look deeper and find that extra little bit of speed.

I have attached a screenshot of an early GUI of Setcopy. Things are still changing, but it will be pretty close to this. I hope to have a testing version of the GUI ready for NAB.

Thanks for all your feedback testing Elliot.



Dusty
 

Attachments

  • SetcopyGUI.jpg
    SetcopyGUI.jpg
    90.8 KB · Views: 0
For anyone interested in testing Setcopy and providing feedback, here are the features we currently have:

Options:
--help Print usage and exit
--version Print the version and exit
-v [ --verbose ] Execute with verbose output
-f [ --force ] Force setcopy to overwrite the destination
-d [ --delta ] arg (full|fast) Do a delta copy on a source directory to multiple targets
-r [ --rename ] arg Tell setcopy to rename the target folder
-s [ --source ] arg The source directory to copy from (requires quotes around source directory)
-t [ --target ] arg The target directory to copy to (requires quotes around target directory)
-i [ --target1 ] arg The intermediate target directory to copy to first (requires quotes around target directory)
-h [ --hash ] arg (none|crc32+|md5|md5+) The hash function used to verify the copy (default: md5)
-l [ --log ] arg Generate log files in parent target directories or to specified directory (requires quotes around log file directory and filename)
-u [ --user ] arg The name of the user (requires quotes around user name)
-o [ --owner ] arg The name of the production company (requires quotes around production company)
-p [ --project ] arg The Name or Project (requires quotes around the project name)


Some examples:

setcopy -v -h md5+ -s "/path/to/source/" -t "/path/to/destination1" -t "/path/to/destination2"

This will copy the /path/to/source/ folder and all files within to both /path/to/destination1 and /path/to/destination2.
This will perform an md5 checksum on the source and on the destination files
This will also print verbose information to the screen
This will write a log file and a source and destination checksum file to each destination


setcopy -v -h none -s "/path/to/source/" -t "/path/to/destination1" -u "Dusty" -o "Sandust" -p "setcopy"

This will copy the /path/to/source/ folder and all files within to /path/to/destination1
This will perform no checksums
This will print verbose information to the screen
This will write a log file to the destination
In the log file it will list Dusty as Data Manager, Sandust as the Production Company,and setcopy as the Project


setcopy -v -h md5+ -s "/path/to/source/NO NAME/" -t "/path/to/destination1" -r "Canon5d_001"

This will copy the /path/to/source/NO NAME/ folder and all files within to /path/to/destination1
This will perform an md5 checksum on the source and on the destination files
This will print verbose information to the screen
This will write a log file to the destination
This will rename NO NMAE to Canon5d_001 on all destinations


setcopy -v -h md5+ -s "/path/to/source/" -i "/path/to/destination1" -t "/path/to/destination2"

This will copy the /path/to/source/ folder and all files within to /path/to/destination1
Then setcopy will copy from /path/to/destination1 to /path/to/destination2
It will perform an md5 checksum on the source and on both destination files
It will also print verbose information to the screen
It will write a log file and a source and destination checksum file to each destination
This can be useful if you have a fast raid for destination 1 and slower drive for destination 2


setcopy -v -h MD5+ -s "/path/to/source/" -t "/path/to/destination1" -d fast

Will look at the source and destination checksum files on "path/to/destination1"
This will only copy files with different checksum in destination checksum file than source checksum file
It will perform an md5 checksum on the source and on both destination files copied
This will print verbose information to the screen
This will append the existing log file in the destination
This will append the checksum files in the destination


If you are interested in testing please send me your email address and I will add you to my beta testers distro.



Mahalo,
Dusty
 
Dustin, hit me up. I'd love to test on the Mac....Ed
 
Dustin Cross said:
Did you compare with RSYNC like CODEX recomends? I have not tested that.
Click to expand...
I can't get rsync to move faster than 150MBps, so that's the worst of the bunch.

Dusty, regarding the GUI, I'd like to request an easy way to enter a shoot day number, to be used in paths. And abundant keyboard shortcuts.
 
How's the GUI coming along? I wanted to request two more features, not sure if you've already covered this:
Auto-start when a mag is inserted
Remember destinations based on media type (so SxS has a set of destinations, while Codex packs have a different set, and Sound has a different set, etc.)
 
Elliott Balsley said:
How's the GUI coming along? I wanted to request two more features, not sure if you've already covered this:
Auto-start when a mag is inserted
Remember destinations based on media type (so SxS has a set of destinations, while Codex packs have a different set, and Sound has a different set, etc.)
Click to expand...

Elliot,

GUI is coming along well. Taking a lot more time than we expected though. We had an issue downloading multiple source mags at once. One mag at a time is blazing fast, but two mags and things slow to a crawl. It seems we have that fixed. It just took an complete rewrite!

I do not like automatic things! In my opinion it creates laziness and that causes errors. So I have no plans of ever having Setcopy automatically copy files with zero interaction. Telling it to copy this mag to these locations with this checksum type is not much work. Plus it would be very hard to have the software predict what automatic means to everyone. I hate when Apple automates some feature I used to use and now it does not work how I did it any more.

The GUI has multiple tabs so you can set up presets for different destinations and just jump between the tabs. If you are wanting something more automatic, see above.

I hope to have a GUI ready for testing soon, but with this complete rewrite of everything, I am not sure how much longer.

Here is a screencap of the GUI.


Thanks
Dusty
 

Attachments

  • Setcopy_GUI_screencap.jpg
    Setcopy_GUI_screencap.jpg
    72.5 KB · Views: 0
Ah I see what you mean about the tabs, that's good enough.
I like how DoubleData starts automatically, but I realize it would be harder to set identifying criteria when you support more than 2 cameras.

By the way, I noticed when copying image sequences (ARI from a Codex mag), CopyVerified is extremely slow at verifying checksums (like 300% slower). setcopy does not have this problem. Keep up the good work!
 
You must log in or register to reply here.
Back
Top