Copy & md5 Shell Script

[Chris Wayg]

I am fine with GPL.

Ok good, I will put it in the file.

In the GUI version I made, at the completion of the copy it went back to the main screen. That made it easy to start the next copy. And from there it is easy to quit the app if I don't want to make more copies.

I like that; where can I find that in your code?

I would like to see the output window not open until you start a copy and close once you acknowledge the completion.

Hopefully my lazy code didn't make it too annoying.

The output window is openend by Platypus. I'm not sure, if I can change its behavior. Maybe something similar could be achieved by just using Pashua.

Your code was quite easy to follow, and your bash skills far exceed mine. But it was harder to modify in regards to allowing spaces in paths and filenames. Almost all path variables had to be quoted and some awk statements modified. I really need the app to handle paths with spaces, because my naming convention is 'YYYY-MM-DD My Video Shoot Name'. Every time I forgot to replace the spaces with underscores, the original copy script really messed up and went all over the place, even being hard to stop. I hope I have achieved a robust handling of spaces with this version.

I was wondering why you copy/pasted much of the code for copy 1 to 4 instead of looping over it in some way? It seems to me, that this would have made the code more concise and easier to maintain and modify.

[Dustin Cross]

I think you would just need to add a line at the end of each of the "onecopy () .... fourcopy ()". Just before exit, have it run the script again. I think that should work.

I copied and pasted so much because I was being lazy. it was easy.

I need to look at what you did to allow spaces. I didn't know of any solutions for that and could find any. The original script was not happy with spaces.

What I really need to fix is the code for percent complete of checksums. It needs to be changed so the math only looks at files and ignores all directories and non-files. That way it can count up to 100% and not have to stop counting at 95%. It is easy to fix, I just have not done it. I should really come up with a new way to calculate percent complete of the checksums, that doesn't go from 0 to 100% when you copy a single file.


Dusty

[Chris Wayg]

I zipped up a new version which I'm currently using for my CF cards and external hard drives. I tested it with some folder trees with thousands of media files and it appears to be free of serious bugs. Additional to the verification done by the script, I verified correctness of the copies by using other MD5 tools, like this freeware http://www.eternalstorms.at/md5/index.html. Please test and verify that the script does what you need it to do, before using it with any important files. The script can be opened in a text editor by finding it inside the app bundle using "Show Package Contents": /Applications/CopyVerified.app/Contents/Resources/script

I followed Dustin's suggestion regarding the workflow: "...at the completion of the copy it went back to the main screen. That made it easy to start the next copy. And from there it is easy to quit the app if I don't want to make more copies."


The latest version can be downloaded here: https://bitbucket.org/chriswayg/copy_verified/downloads
If you find bugs please report them in the bug tracker here: https://bitbucket.org/chriswayg/copy...ied/issues/new

Changelog for Version 20120123.1 beta:
# 01/22/12
# - configurable file name extension for MD5 files (.txt or .md5)
# - added GPL license
# 01/23/12
# - uses the unmodified Pashua.app again; renaming it was not needed, as it is hidden anyways.
# - added GUI loop until user exits by clicking Cancel
# - added GUI loop until all user input is OK
# - determine the intention of the user as to the number of copies without a dropdown box

I need to look at what you did to allow spaces. I didn't know of any solutions for that and could find any. The original script was not happy with spaces.

See the diff here: https://bitbucket.org/chriswayg/copy...1=c79df3141dd2

Some examples of the main changes for dealing with spaces

# quoted all folder path variables like this in dozens of places
cp -Rpn "$COPYPATH1"/"$DESTPATH" "$COPYPATH2"/ 2>>~/Library/Logs/CopyVerified/copyverified_"$DESTPATH".log &

# folder paths are saved comma separated to the config file:
193 echo "$NUM_COPIES, $COPYPATH1, $COPYPATH2, $COPYPATH3" > ~/Library/Logs/CopyVerified/copyverified.config ;
...
750 then file1=`cat ~/Library/Logs/CopyVerified/copyverified.config | awk 'BEGIN { FS = "," } {print $1}'`

177 # the first version of this gets truncated, if the path has spaces
#DESTPATH=`echo $SOURCEPATH | awk -F\/ '{print $NF}' | awk '{print $1}'` ;
DESTPATH=`echo "$SOURCEPATH" | awk -F\/ '{print $NF}'` ;

565 # the change was made to accomodate path/file names with spaces
#for FILE in $(find -s * -type f 2>&1) ;
find -s * -type f  | while read FILE

[Dustin Cross]

Chris,

Awesome work!

The new md5 seems about 5% faster. That is great.

Cool solution for spaces. Seems to work great.


A couple things I would look at.

- change sequence of error checking. have it write the data in the fields to the config before the error check, so when the main window pops back up after error check message it has the data you filled in. Like it does at the end of the script.

- move source input field to top of page. That is the field I want to fill out first. The reason I did the destinations first on the original script is so I could get the script all set up and ready and as soon an I connected the drive I just had to give it the source path and go. With the GUI it feels like the source should be first.

- I would add a button to view the log file instead of the checkbox. That way you can look at the log any time you want.

- A checkbox in the GUI to select .md5 or .txt would be great.


Since the script is now a GUI and much more accessible to people I think it needs proper error handling for comparing the checksums. Right now it will say that the files did not match, but then it will print checksums completed successfully. People comfortable with command line could figure that out and see there was a problem. With the GUI it just needs at the end of the compare checksum, if successful print successful, else print there was an error. I never added this because the diff print all the errors on the screen and it is VERY rare that two copies don't match. Really proper error handling for everything really.


Great improvements to the script.




Dusty

[Tim Sutherland]

Hey Guys, thanks for your work on this.

For anyone who hasn't used this yet, I've transferred literally thousands of terabytes of data successfuly using this script. For me, it is definitely the fastest way to move data through my cart.

Some things about the UI - Is there a way to make it update the same line like in terminal instead of printing a new line every second or so? It's kind of a pain the way it scrolls so much. Also, Can you make the window print the source and destination at the top? I usually have 3 to four windows of the script running at once, and the GUI makes it hard to know what is running and what is finished.

Also, I have to use terminal to open multiple instances, where as the straight script only requires opening the file from the dock to launch multiple instances.

If you could do that, then this would rock even more than it already does.

Tim

[Dustin Cross]

Chris,
The new md5 seems about 5% faster. That is great.

Chris,

I misunderstood what you were doing with the freeware MD5 app. I thought it was a more threaded and faster command line MD5. Your GUI is running about 5% faster than the command line script when I compare the two. Very nice. Need to look at why? Have you done anything or is it just you cleaning everything up?


Dusty

[danu prasd]

Dustin and Chris,

Excellent work, guys !!!

Kudos to both of you...

If I can give some input, it will be more OK if you guys can put some time remaining...so we can exactly tell someone (ie : noisy producer ) that how long do it takes to get the files done for copying.

Above that, thank you again for bringing in such a wonderful apps.

[Fred Beahm]

Great work, awesome share. Any future plans for an interface? I can help design if you need.

[Fred Beahm]

Seems to have errors a lot of the time. I'll be sticking with my r3d manager. :P

[Tim Sutherland]

Seems to have errors a lot of the time. I'll be sticking with my r3d manager. :P

I use it for several terabytes 5 days a week with no problems, what errors are you seeing? Can you post a screenshot?

Tim