Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Copy & md5 Shell Script

D

Dustin Cross

Well-known member
Joined
Mar 6, 2007
Messages
1,240
Reaction score
0
Points
0
Location
Austin, Texas
Website
www.sandust.com
I have been playing around with a little shell script to make copies of footage and do md5 checksums on them. Mainly because things like R3D Data Manager only copy Red media and things like Shotput Pro don't do md5 checksums.

Basically the script copies any folder to one to four destinations. Then makes md5 checksum of the source and each destination and compares them.

Run it from the command line and then simply drag and drop folders from Finder to tell the script where things are. Pretty simple.

I give no warranty and I wouldn't use this on important footage unless you have played with it and know what it is doing.

I wrote this on OSX so some commands it uses might not exist on Linux or other systems. So test.

Just thought I would share and let people play.


Dusty


edit -- I made a minor change to the script. I was trying to speed it up and compare the checksums in parallel and that was too fast for small files.
edit -- changed the script so md5 deals with sub directories correctly (02/02/11)
edit -- changed script so errors print to the screen instead of to /dev/null (02/02/11)
edit -- cleaned up the code a little (02/10/2011)
edit -- fixed a problem where the copy didn't work on files with "." in the initial directory name (02/17/11)
edit -- made the md5 part ignore folders so it doesn't give errors on folders. md5 now only works on files and not directories or any special files (02/17/11)
edit -- the script has problems with doing md5 on things like OSX applications which are actually some kind of hidden folder. The copy works, the md5 has issues. (02/17/11)

edit -- added a new version (CopyScript_percent.sh) of the script for testing that has a config file, tells percent complete, and keeps timestamp of the source files. (03/10/11)
edit -- Cleaned up things a little. Improved performance a little by having source checksum happen during second copy. Also made errors write to a log file. Will add more for this log file in the future. (03/11/11)
edit -- added more info to the log file. /var/tmp/copyscript.log (04/19/11)
edit -- modified the script so you can run multiple copies at once without conflict. So you can copy multiple different drives at the same time (04/19/11)
edit -- fixed the source checksum 100% complete message. (04/22/11)
edit -- changed the order of the questions at the beginning so the source drive is last.

edit -- added link to GUI front version of the shell script. (01/18/12)
http://www.sandust.net/SandustTools/SandustTools.zip

edit -- link to the new GUI version of the script Chris created. (01/27/12)
The latest version can be downloaded here: https://bitbucket.org/chriswayg/copy_verified/downloads
If you find bugs please report them in the bug tracker here: https://bitbucket.org/chriswayg/copy...ied/issues/new
 

Attachments

Nice. Thanks for sharing, this will be nice for copying audio files, etc, especially since we will still have the hard copy of that.

I might try to make a little automator app that runs this with a little interface. If I do I'll post here for the same reasons.

Along the same lines, I made a little app that opens a new instance of R3D Data Manager so you can put it it the dock and click it to open each instance you want to use. I'll get it off my desktop and upload it if anyone is interested.

Tim
 
I was thinking about writing a Applescript GUI frontend for the script. I didn't think about trying something with automator.

Right now I have the script set to copy to the first location and then copy from the first location to the second through fourth locations in parallel. This is because I always copy to my fast internal raid first and from the fast raid to all the shuttle drives and such.


Dusty
 
Thanks!
 
You should be a little careful about redirecting standard errors to /dev/null; I ran this script on Linux (Ubuntu) out of curiosity, and it ran without errors - even if md5 does not exist on Linux (md5sum on the other hand). It copied the folders OK - but the checksum files were empty - false security.

And a footnote on your development style - there is a thing called "for loop" that can get rid of lot of the copy-and-paste code.
 
The last time I ran a script through automator, it worked a lot better than this time, so I guess I'll stick with terminal, unless you can get it to work better with a GUI.

Thanks for sharing this.

Tim
 
Erik Larsson said:
You should be a little careful about redirecting standard errors to /dev/null
Click to expand...

Doh. I did that because md5 was giving me lots of errors today on the sub folders. I should change it back for sharing. But that's why I said play with it and know what it is doing before using it for anything important. I changed it to print errors to the screen and reloaded it above.

You are right. I should clean up the code, but it was easy to copy and paste. I guess it chould also check for things like md5, but that comes standard on OSX.


Dusty
 
Dustin Cross said:
Doh. I did that because md5 was giving me lots of errors today on the sub folders. I should change it back for sharing. But that's why I said play with it and know what it is doing before using it for anything important. I changed it to print errors to the screen and reloaded it above.

You are right. I should clean up the code, but it was easy to copy and paste. I guess it chould also check for things like md5, but that comes standard on OSX.


Dusty
Click to expand...
Well, you could redirect errors to /dev/null - but make sure you exit with an error when things go to wrong - like not having md5 installed.

As it is now, everything does look to work correctly - and it is kind of hard to know that it does something wrong when it does not tell you. So - if I did not know better, I would test this: No errors, copy OK, but the MD5 validation step failed without telling me ... Potential "deep shit situation".
 
Just loaded a new version of the script. See first post.

I fixed a problem where the script wouldn't copy directories with a "." in the name like and RDM folder.

Cleaned up the code a little so it should be easier for other people to modify for their needs.

Also made md5 only look at files. It ignores anything that is not a file, like a directory. So again, please test and make sure it does what you need before using. Also, I always manually verify my md5 at the end, even if the script didn't give me any errors.

I have been using this for everything lately and not having any issues. Used it for all Red media on a commercial recently and I was very happy. I mostly wrote it for Alexa media and sound, but I have been using it for a lot of stuff lately.

I just found out it has a problem doing the md5 on OSX application. They are actually some type of hidden folder. I knew that, but never really looked into it. I was just grabbing folders to copy for tests and found this out.

I am looking at ways to speed up the script now. If anyone has any ideas on how to improve the script let me know or just do it and share.

Here is a link to another copy script on Reduser:
http://reduser.net/forum/showthread.php?t=55584&highlight=copy+md5+script

I haven't tested this and it is much more complicated than my very simple script.


Mahalo,
Dusty
 
Any way to get it to not modify the Creation date of the copied file?
 
Mike,

I think cp -p will preserve the original file attributes. Atleast the attributes it can. You would just need to change "cp -R" to "cp -Rp" in the script.

I wonder if that is something that should be a default part of the script?


Dusty
 
Great script been thinking about writing something like this for ages myself. It might be great to have an option to save a temp file of the chosen destinations for the day so that they can be recalled instead of having to re-drop each time.
 
Rick,

I thought about adding that, but I always have finder windows open with those folders for verify and QC, so it is super easy for me to drag them into the script. I would need to think about the best way to have the script deal with that temp config file.

If you have an idea, feel free to add to the script and share your improved version.

I am more focused on trying to make the copy faster right now. comparing cp vs cpio vs rsync vs tar piped to untar. I have found all kinds of recommendations online, but so far cp seems to be the fastest. I was looking for something to make a progress bar, but I haven't found a good solution for that yet.


Dusty
 
Dustin, I noticed that the script returns an error with any drive or folder names with spaces in them. Is there any way to get around that? I know I can change drive and folder names, just wondering if there's anything you can do with the scrip instead.

Tim
 
Tim,

Spaces in file and directory names are always a bad idea. Lots of app have problems with that.

Let me see if I can come up with a solution.

I have a new version of the script that tells you percent complete of each step. The code is all ugly. Once I get it cleaned up, I'll post it. I'll also look into the spaces thing.


Dusty
 
I added a new version of the script (CopyScript_percent.sh).

This version has a config file to keep track of where you want files copied to.

It also tells you percent complete of each step of the copy process.

I also changed the copy process to preserve the original file attributes.

I am still testing this version. Everything seems to be working well. Please test this version out and let me know if anything needs to be fixed.


Mahalo,
Dusty
 
You must log in or register to reply here.
Back
Top