Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Copy & md5 Shell Script

Dustin Cross said:
I am fine with GPL.
Click to expand...
Ok good, I will put it in the file.

Dustin Cross said:
In the GUI version I made, at the completion of the copy it went back to the main screen. That made it easy to start the next copy. And from there it is easy to quit the app if I don't want to make more copies.
Click to expand...
I like that; where can I find that in your code?

Dustin Cross said:
I would like to see the output window not open until you start a copy and close once you acknowledge the completion.

Hopefully my lazy code didn't make it too annoying.
Click to expand...
The output window is openend by Platypus. I'm not sure, if I can change its behavior. Maybe something similar could be achieved by just using Pashua.

Your code was quite easy to follow, and your bash skills far exceed mine. But it was harder to modify in regards to allowing spaces in paths and filenames. Almost all path variables had to be quoted and some awk statements modified. I really need the app to handle paths with spaces, because my naming convention is 'YYYY-MM-DD My Video Shoot Name'. Every time I forgot to replace the spaces with underscores, the original copy script really messed up and went all over the place, even being hard to stop. I hope I have achieved a robust handling of spaces with this version.

I was wondering why you copy/pasted much of the code for copy 1 to 4 instead of looping over it in some way? It seems to me, that this would have made the code more concise and easier to maintain and modify.
 
I think you would just need to add a line at the end of each of the "onecopy () .... fourcopy ()". Just before exit, have it run the script again. I think that should work.

I copied and pasted so much because I was being lazy. it was easy.

I need to look at what you did to allow spaces. I didn't know of any solutions for that and could find any. The original script was not happy with spaces.

What I really need to fix is the code for percent complete of checksums. It needs to be changed so the math only looks at files and ignores all directories and non-files. That way it can count up to 100% and not have to stop counting at 95%. It is easy to fix, I just have not done it. I should really come up with a new way to calculate percent complete of the checksums, that doesn't go from 0 to 100% when you copy a single file.


Dusty
 
I zipped up a new version which I'm currently using for my CF cards and external hard drives. I tested it with some folder trees with thousands of media files and it appears to be free of serious bugs. Additional to the verification done by the script, I verified correctness of the copies by using other MD5 tools, like this freeware http://www.eternalstorms.at/md5/index.html. Please test and verify that the script does what you need it to do, before using it with any important files. The script can be opened in a text editor by finding it inside the app bundle using "Show Package Contents": /Applications/CopyVerified.app/Contents/Resources/script

I followed Dustin's suggestion regarding the workflow: "...at the completion of the copy it went back to the main screen. That made it easy to start the next copy. And from there it is easy to quit the app if I don't want to make more copies."


The latest version can be downloaded here: https://bitbucket.org/chriswayg/copy_verified/downloads
If you find bugs please report them in the bug tracker here: https://bitbucket.org/chriswayg/copy_verified/issues/new

Changelog for Version 20120123.1 beta:
# 01/22/12
# - configurable file name extension for MD5 files (.txt or .md5)
# - added GPL license
# 01/23/12
# - uses the unmodified Pashua.app again; renaming it was not needed, as it is hidden anyways.
# - added GUI loop until user exits by clicking Cancel
# - added GUI loop until all user input is OK
# - determine the intention of the user as to the number of copies without a dropdown box
Dustin Cross said:
I need to look at what you did to allow spaces. I didn't know of any solutions for that and could find any. The original script was not happy with spaces.
Click to expand...
See the diff here: https://bitbucket.org/chriswayg/cop...fied.sh?diff2=5b550aad15a7&diff1=c79df3141dd2

Some examples of the main changes for dealing with spaces
Code: 
# quoted all folder path variables like this in dozens of places
cp -Rpn "$COPYPATH1"/"$DESTPATH" "$COPYPATH2"/ 2>>~/Library/Logs/CopyVerified/copyverified_"$DESTPATH".log &

# folder paths are saved comma separated to the config file:
193 echo "$NUM_COPIES, $COPYPATH1, $COPYPATH2, $COPYPATH3" > ~/Library/Logs/CopyVerified/copyverified.config ;
...
750 then file1=`cat ~/Library/Logs/CopyVerified/copyverified.config | awk 'BEGIN { FS = "," } {print $1}'`

177 # the first version of this gets truncated, if the path has spaces
#DESTPATH=`echo $SOURCEPATH | awk -F\/ '{print $NF}' | awk '{print $1}'` ;
DESTPATH=`echo "$SOURCEPATH" | awk -F\/ '{print $NF}'` ;

565 # the change was made to accomodate path/file names with spaces
#for FILE in $(find -s * -type f 2>&1) ;
find -s * -type f  | while read FILE
 
Last edited:
Chris,

Awesome work!

The new md5 seems about 5% faster. That is great.

Cool solution for spaces. Seems to work great.


A couple things I would look at.

- change sequence of error checking. have it write the data in the fields to the config before the error check, so when the main window pops back up after error check message it has the data you filled in. Like it does at the end of the script.

- move source input field to top of page. That is the field I want to fill out first. The reason I did the destinations first on the original script is so I could get the script all set up and ready and as soon an I connected the drive I just had to give it the source path and go. With the GUI it feels like the source should be first.

- I would add a button to view the log file instead of the checkbox. That way you can look at the log any time you want.

- A checkbox in the GUI to select .md5 or .txt would be great.


Since the script is now a GUI and much more accessible to people I think it needs proper error handling for comparing the checksums. Right now it will say that the files did not match, but then it will print checksums completed successfully. People comfortable with command line could figure that out and see there was a problem. With the GUI it just needs at the end of the compare checksum, if successful print successful, else print there was an error. I never added this because the diff print all the errors on the screen and it is VERY rare that two copies don't match. Really proper error handling for everything really.


Great improvements to the script.




Dusty
 
Hey Guys, thanks for your work on this.

For anyone who hasn't used this yet, I've transferred literally thousands of terabytes of data successfuly using this script. For me, it is definitely the fastest way to move data through my cart.

Some things about the UI - Is there a way to make it update the same line like in terminal instead of printing a new line every second or so? It's kind of a pain the way it scrolls so much. Also, Can you make the window print the source and destination at the top? I usually have 3 to four windows of the script running at once, and the GUI makes it hard to know what is running and what is finished.

Also, I have to use terminal to open multiple instances, where as the straight script only requires opening the file from the dock to launch multiple instances.

If you could do that, then this would rock even more than it already does.

Tim
 
Dustin Cross said:
Chris,
The new md5 seems about 5% faster. That is great.
Click to expand...


Chris,

I misunderstood what you were doing with the freeware MD5 app. I thought it was a more threaded and faster command line MD5. Your GUI is running about 5% faster than the command line script when I compare the two. Very nice. Need to look at why? Have you done anything or is it just you cleaning everything up?


Dusty
 
Dustin and Chris,

Excellent work, guys !!!

Kudos to both of you...

If I can give some input, it will be more OK if you guys can put some time remaining...so we can exactly tell someone (ie : noisy producer ) that how long do it takes to get the files done for copying.

Above that, thank you again for bringing in such a wonderful apps.
 
Thanks for this great little helper!
One question: if I do a copy after the first transfer from the destination folder to another drive - can I proceed the same way or does it create md5 of an md5?
best wishes and kind regards,
martin
 
Been using this nifty script on a feature and it's been fantastic. Let's me start making dailies while my RAID copies to backup drives. Thank you so much for creating it!

I do have a bit of feedback and possible bugs:

1) Sometimes it seems to hang between the 2nd & 3rd copy and MD5 checks. I've had to cancel it and just do a check a different way. This seems to happen if I've had the computer on a bit and done a few copies, as the first ones usually work fine.

2) The window with the folder selections hides behind everything, so I'm constantly having to hide all other windows to get to it.

Requests:

1) Multiple copy windows (offload from 2 cards at once, for example)
2) Copy on RedMag mount
3) Progress bars, or updated lines instead of a thousand lines indicating percentage
4) Presets - I'd love to use this for sound cards as well, but going through and changing all the destinations and source every other transfer is a hassle

However, this tool is amazing. I've built a system where I can offload a full 128gb RedMag in less than 10 minutes, and have dailies ready shortly after along with 2 backups of footage. Absolutely invaluable - the copy first to raid, then copy from raid function is genius.
 
David,

Are you using the GUI version or the command line version? I assume the GUI version since you are asking how to run multiple copies.

- hang on md5 I will need to look into. I mostly use the command line version, so I have not experienced this with the GUI.

- window hiding. on Mac F9 or Function F9 should show you all available windows so you can easily select the window you want.

- Multiple copy windows. I think you would have to duplicate the application and that would allow you to run multiple instances of the application. With the command line version I just open a new tab in Terminal and run the script again. On a job last week I have 8 downloads going at one time in 8 tabs. Thought I was gonna kill the client drives :)

- Copy on RedMag mount is not something I plan to add. I use this with so many types of media that auto copy on RedMag mount is not planned. The code of the script is right there, so you can add that if it is really important. My favorite part about the shell scripts is they are easily modified to your special requirments.

- Progres bars would be nice. The command line over writes lines until it finishes. The GUI writes a new line for each update with no over writing. Just limitation of the tools used to make the GUI. There is a little progress bar tool, but I didn't think it worked as well as the command line script text output, so I didn't use it.

- presets is an interesting idea. It does remember the config of your last copy, but having it remember multiple configs. I will have to think about how to do that without making it too complicated. I use the command line with sound all the time and every other type of media. Just have to remember it doesn't work with spaces lots of mags have in their name. Quick rename of the mag and copy away. I really need to add the changes Chris did to the GUI for spaces to the command line version.

- copy to raid first is only a benefit for those that have a fast raid to take advantage. I think it works great, but I still use RDM or Shotput on jobs where I don't have a raid.

- A huge advantage of this script is you can modify it to meet your needs. You could add a redline command to the script and have the script automatically transcode your footage as soon as it is verified on the raid.

Glad it was useful.


Dusty
 
Hi Dustin,

Brilliant Tool. I am using the GUI version and it often hangs on the verify at 95% Any way to fix this? And where do I send my donation to?

Many thanks

Georg
 
Georg,

Not that brilliant if it hangs often. :( What are you copying when it hangs? what type of media and how many MB?

I thought I had fixed this problem. I need to redo the math it uses to calculate % complete. Different file systems measure file sizes differently. Mostly it is with folders.


Dusty
 
Dustin Cross said:
David,

Are you using the GUI version or the command line version? I assume the GUI version since you are asking how to run multiple copies.

- hang on md5 I will need to look into. I mostly use the command line version, so I have not experienced this with the GUI.

- window hiding. on Mac F9 or Function F9 should show you all available windows so you can easily select the window you want.

- Multiple copy windows. I think you would have to duplicate the application and that would allow you to run multiple instances of the application. With the command line version I just open a new tab in Terminal and run the script again. On a job last week I have 8 downloads going at one time in 8 tabs. Thought I was gonna kill the client drives :)

- Copy on RedMag mount is not something I plan to add. I use this with so many types of media that auto copy on RedMag mount is not planned. The code of the script is right there, so you can add that if it is really important. My favorite part about the shell scripts is they are easily modified to your special requirments.

- Progres bars would be nice. The command line over writes lines until it finishes. The GUI writes a new line for each update with no over writing. Just limitation of the tools used to make the GUI. There is a little progress bar tool, but I didn't think it worked as well as the command line script text output, so I didn't use it.

- presets is an interesting idea. It does remember the config of your last copy, but having it remember multiple configs. I will have to think about how to do that without making it too complicated. I use the command line with sound all the time and every other type of media. Just have to remember it doesn't work with spaces lots of mags have in their name. Quick rename of the mag and copy away. I really need to add the changes Chris did to the GUI for spaces to the command line version.

- copy to raid first is only a benefit for those that have a fast raid to take advantage. I think it works great, but I still use RDM or Shotput on jobs where I don't have a raid.

- A huge advantage of this script is you can modify it to meet your needs. You could add a redline command to the script and have the script automatically transcode your footage as soon as it is verified on the raid.

Glad it was useful.


Dusty
Click to expand...

I'm using the GUI version. I duplicated the app like you recommended and am now having no problems downloading multiple cards at once. I have a pretty fast raid (1 gigabyte/sec) so offloading 2 cards and exporting dailies and backing up to 2 client drives is no problem.

Having to hida all or using Fn+F9 is merely an annoyance - I wish to be able to Alt-Tab to the app and have all windows come to the front.

Instead of copy on RedMag mount, perhaps a nice option would be a list of drives that pop up and you could choose which one copies. Perhaps with the presets idea, you could even enter in the usual cards that you use - for example, if a card called 788T comes up (for sound) it would fill in the blanks for where to copy sound.

Progress bars arent essential, but at least having the % line overwritten would be good.

Do you have a resource for learning how to script a bit? I'd love to add a sync audio and transcode bit with redline if possible.

By the way, I used to do software User Interface design, so if you'd like to collaborate on the GUI version I'd be more than happy to help.
 
David,

There are tons of resources online to learn bash scripting. Here are a couple simple scripts I posted on Reduser that are easier to understand than the copyscript.

http://reduser.net/forum/showthread...cture-software&p=793307&viewfull=1#post793307
http://www.reduser.net/forum/showth...nes-with-audio&p=852354&viewfull=1#post852354

and here is an old redline script with each line explained:
http://www.reduser.net/forum/showthread.php?15499-RED-Line-Batch&p=246746&viewfull=1#post246746

Basically a shell script is just a list of command line commands written out in order. You can write them out on a single line with ; between each command or you can save them in a text file with each command on a new line so it is easier to read.

You can use If Then Else statements to tell the script If this is true Then do things A Else do thing B.

You can use a For loop to tel the script For each of these things Do something.

The most complicated thing in my scripts is Sed and Awk. These are two convoluted command line tools for editing text.

You can download the copyscript from the first page of this thread. Or in GUI app, command click and click "Show Package Contents". In CopyVerified.app the actual script is located in the Contents/Resources folder and is called script. Open that is a text editor. In SandustTools.app there are several scripts in Contents/MacOS folder. Open them in a text editor.

Hopefully that helps you get started. Let me know if I can answer any questions.



Dusty
 
You must log in or register to reply here.
Back
Top