Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Copy & md5 Shell Script

Wanted to make this look nice in my doc, where I have it for launching, so I made a quick icon for it. This plus telling osx to always open with terminal makes it nice and clean.

Tim

copyscripticon.png
 
That way find ignores anything that is not a file (-type f). the only problem with that is I have to find all files, du to get their size, then add all the sizes to get the total. It was so much easier to change 99% to 90% and be done.
Click to expand...

find path/to/your/folder -iname \* -exec ls -l {} \;|awk '{print $5}'| awk '{ sum+=$1} END {print sum}'

will do all your quoted needs in one line of bash code if you want it!

and i've just tested a folder size across HFS+ and FAT32 and it comes back exactly the same.

HFS+ volume (folder on desktop of mac)
find /Users/jamie/Desktop/The_Best_Film_Ever/A003_R1P7_AMA -iname \*.mov -exec ls -l {} \;|awk '{print $5}'| awk '{ sum+=$1} END {print sum}'
74252205

fat 32 volume
find /Volumes/FATTY32/A003_R1P7_AMA -iname \*.mov -exec ls -l {} \;|awk '{print $5}'| awk '{ sum+=$1} END {print sum}'
74252205


Haven't tested UDF though. Would be good if someone here could do that...then i'd know as well!
cheers
j
 
Jamie,

Thanks. I was joking and being lazy. I know its pretty easy to do.

One thing I haven't found a good solution for is spaces in filenames or path. Being an old school unix guy, I never use spaces so I never even think about that, but they are common enough these days to be annoying. I have found a couple really ugly solutions for dealing with spaces. Do you have anything nice and neat for that problem or do you just ignore it, like I do?


Dusty
 
Actually, after playing with the script a bit, it would be nice to have it notify you when the checksum of the source is complete. I don't think I'd want it to auto eject, though.

The Log looks great, can you put a line in at the end stating wether or not all checksums matched? It's possible that it does report bad checksums but I haven't had any in my testing yet to check. I'll force a bad checksum and see what happens. And it would be great if the log data was added above previous data, so that you don't have to scroll down forever to see current info after transferring hundreds of mags.

Also, it looks like the LOG file doesn't update until the operation is complete. If the computer crashes during a copy operation, no information is stored in the LOG. Is there a way to have it update the log file at the completion of each copy step? That way after a crash and reboot, you can quickly see what operations finished and what operations didn't. I know you can manually check the md5 values, but it would be handy to have in the LOG as well.

I'm also wondering if the script could check for already generated md5 value txt files, to save time on recalculating md5s in event of a crash or additional copy after the original copy operation.

And it would be cool if it were stored in a location like /User/Library/Logs/CopyScript.log or similar so that those without hidden files turned on can easily locate the log. Same for the config file.

Again, great work, this thing is getting better and better.

Tim
 
Tim,

All of that is doable.

The Source checksum complete was there, but I think the second copy is overwriting it. I will make it create a new line so it is visible along with the second copy percent complete.

I can have a line that confirms the diff had no errors or tell what the errors are. I have been meaning to add this, but I always manually verify the checksums, so it hasn't been a priority.

Writing the log in reverse order. I will have to look into the best way of doing that. Shouldn't be a problem.

The log makes a temp file while it is working and at the end puts the temp file in the main log. This is because of multiple instances, so each mag is in its own section and not all mixed up. So if the copy fails in the middle, look at those temp files and you should find what was completed.

Checking for existing md5 files should be easy.

Changing the location on the log files and config files is easy. I will put everything in /User/Library/Logs/CopyScript/

Anything else you can think of to put in the log?


Dusty
 
Maybe in the confirmation line, have a nice little summary, telling how many folders and files were copied and what the results were, something like.

(A001_0807RX) -> Copied (1) folder containing (12) files to (2) destinations on (Dest Drive 1) and (Dest Drive 2), with (no errors).

where each () contains a variable

That way it's something to quickly show as proof of a good copy if necessary, since when it hits the fan, the person managing the data is often the first to be blamed.

Also some option to re copy bad checksummed files without copying the whole thing again would be nice, but that might be more complicated.

Tim
 
I can add a nice summary at the end.

Do you like how it is keeping each mag together, or would you be fine with everything mixed together with just time stamps?

I think I will hold off on the recopy stuff. I want it to notify when there is a problem, but I think fixing the problem should require manual intervention. I will look into it though.


Dusty
 
I can see that about the re copy. Maybe sometime you could do a separate script to recopy non matching checksums for when you're in a pinch.

The way mags are grouped is fine, although maybe some tabs to make it easier to read.

Tim
 
---------------------------------------------
Hi Dustin,

I hope you don't mind me contacting you privately but I really like your shell and wouldn't want to cast a stain on its reputation by publicly reporting a problem which is almost certainly related to deficiencies in my IT skills.

I was using it on an Alexa job a few days ago (after testing it and coming to the conclusion that it is fast and reliable as you said - at least in relation to the not fast and definitely not reliable shot put pro).

I tried to copy an Alexa folder to two destinations and it reached 96% then froze for several minutes even though the data was all copied. I checked the bits and they matched up and played through a couple of clips and everything seemed fine. I would have liked to transcode and ultimately retry your shell but it was in the heat of battle (I was data wrangling and ACing) so I just had to revert to the hated shot put pro and press on.

I'd love to have another crack of the whip with your shell but would feel more confident about using it if you had any idea why it had a little freak-out the other day. I'm quite busy and the lions share of jobs these days are on the Alexa so I will have the opportunity pretty soon.

On a related note - if you were thinking of developing it further, one thing I would dearly love to use is something that lets me copy within finder (drag and drop) then run a checksum retrospectively

i.e. First folder (Drag 1st folder here) 2nd folder (drag folder here) original folder (drag folder here) - perform checksum.

In my wildest dreams I'd do it myself but, as you have probably worked out from this message, I'm far too thick.

All the best from England

Have a good weekend

Alex Parkinson
---------------------------------------------

Alex,

I know exactly what happened. It is the problem Jamie and I were discussing on earlier in this thread about different file systems measuring directors differently. In the UDF file system used for Alexa SxS cards, directories actually have a different file size than in HFS+ used by OSX. So you run into a problem where the destination never gets large enough to end the loop I use to measure percent complete. The latest version of the script solves that problem is a really dumb way. I have the measure end at 90% instead of 98%. I really need to change the script so it only measures files and ignores directories.

Try the latest version of the script on some test footage and see how it works for you. It stops the loop at 90% now to prevent that problem. It could still happen if the source and destination are more than 10% different in size, so I really need to rewrite that whole section of the script.

A script that just does MD5 checksums would be pretty easy. I am about to start a new feature, so it might be after that before I could get to it. I need to make a few things about this script better first.



Dusty
 
I applaud you Dustin. Just found this thread, cause I haven't had enough time to search for this until now. Something I've been wanting to do for sometime, but haven't had the time for. Way to go!

Hope you don't mind if I make some edits! I'll still keep your name in the comments of the script though.
 
Thanks Dustin for this very useful script! At first I considered buying Shotput Pro, but when I tested your script I found out it meets my needs fully and I can adapt it easily to various use cases. Hopefully someone here could help me with an issue I ran into:

Thankfully the script unsurfaced a problem with corrupted video files from my Canon EOS 7D when copied to my desktop Mac. (Strangely the Shotput Pro demo did not catch the corruption.) I copy from SanDisk Ultra 16 GB CF cards using a USB card reader to two internal hard drives. Every time about a dozen video files out of 50 to 100 clips are corrupted, meaning the MD5 do not match even though the files usually play fine in Quicktime. I suspect some crashes in FCP-X while editing may be caused by these corrupted files. I double checked with other MD5 software like checkSum+ and they confirm the corruption. Usually the files copy successfully after a second or third try of re-copying just the damaged ones.

I've checked this with three different card readers, various CF cards and cables and different hard drive targets. The problem only shows up on the desktop Mac and not on my 2010 MacBook Pro using the same card readers (- everything copies perfectly onto the MacBook). I've run Prime95 on the desktop, stress testing the CPU and memory for many hours and it did not show any errors. This leads me to believe, that probably memory or CPU errors are not the cause of the corruption.

What else could it be? I'm really out of ideas on what else I could check :confused:

Chris
 
Not sure why I have not posted this yet, but I added a little GUI front to make the script easier to store in your Apps folder and launch. It is still a command line script. The GUI just launches terminal and runs the script. The GUI also has a couple other scripts I have written.
http://www.sandust.net/SandustTools/SandustTools.zip

The first thing it does when you run the app is check the website to see if there is a newer version. If there is, it will ask if you want to download and install the new version. You can test this by downloading the old version and running it. The script is written to be install in Application folder so that is where it will install the new version.
http://www.sandust.net/SandustTools/...ls20110726.zip

right now it looks like nothing is happening when it is downloading the new version, but the new version will pop up once the process is complete. I should add some kind of counter to let you know it is doing something.

If you look inside the application package you can see all the scripts and tools that are in there, so you can still modify them to your needs.

Hope you find this useful.

PLEASE TEST and let me know if you find any problems.


Dusty
 
The only advice I could offer, Chris, is that if the issue is only located on one computer, then you know it is either hardware or software on that computer that is causing it.

Maybe that sounds pretty obvious. But if you were to re-install the operating system and still get the issue, then you know it's entirely hardware related. Of course, since it's a somewhat inconsistent issue, it's kind of tricky to diagnose from there.
 
Nick Pasquariello said:
The only advice I could offer, Chris, is that if the issue is only located on one computer, then you know it is either hardware or software on that computer that is causing it.
Click to expand...
Thanks for your replies and suggestions. Its hard to pin down the exact cause, as the issue cannot be reproduced consistently. I removed some RAM which I had added recently and I tried different USB ports. Now the issue seems to have been reduced, having copied two cards in a row without issues. I suspect that it is either the RAM or a faulty USB port.

It goes to show that we cannot rely on the operating system to protect us from data corruption during a copy operation, and that verification of checksums is essential to alert us to problems, before they become incorrigible.
 
Dustin Cross said:
I added a little GUI front to make the script easier to store in your Apps folder and launch. It is still a command line script. The GUI just launches terminal and runs the script.
Click to expand...
Dusty, thanks for your updated script. I wouldn't have noticed the corruption of my video files in a timely manner without it.

I really like your idea of integrating the CopyScript with Pashua as a GUI frontend. I modified your script to use Pashua for all data input and added a Platypus wrapper for making it an app and for terminal-like output, thus eliminating the need to start a separate terminal session. I added more robust user error checking and modified it throughout to allow file paths with spaces. (I did not integrate the other scripts or your auto-update mechanism, as I'm mainly interested in the CopyScript.)

I wanted to ask under what license you consider your code? If its not public domain I would suggest an appropriate open source license such as the GPL or MIT. I uploaded my changes to a git repository at bitbucket. Please give it a try and tell me what you think. It hasn't been tested yet extensively, thus it may still contain bugs.

https://bitbucket.org/chriswayg/copy_verified
Here is my changelog:
# - added a Pashua GUI frontend for inputting the file paths and data
# - added Platypus wrapper for making it an app and for terminal -like output
# - added support for spaces in path and file names
# - added the -n option to cp to prevent from accidentally overwriting files
# - added sanity check for validity of paths
# - chanage MD5 files from .txt to .md5 to facilitate verfication with other tools
# - renamed to 'CopyNverify' to have a name that is less generic
 
Chris,

Nice.

I made a more complicated GUI version of the script using Pashua, but I got too complicated and it really slowed things down, so I scrapped it.

I am fine with GPL.

I made the MD5 files .txt so it was easy to review them. Simply select them and hit the space bar. Originally they were .md5, but .txt made life so much easier and I can easily change them to .md5 if I ever needed to.

In the GUI version I made, at the completion of the copy it went back to the main screen. That made it easy to start the next copy. And from there it is easy to quit the app if I don't want to make more copies.

I would like to see the output window not open until you start a copy and close once you acknowledge the completion.

Hopefully my lazy code didn't make it too annoying.



Dusty
 
You must log in or register to reply here.
Back
Top