DIT intern question: Do you ignore the difference log files make when transferring?

[Melih Engin Eskier]

There was a mismatch in between file sizes of my folders on the SSD card and the MAC hard disk.

I have heard this is because hard disk format types. What do you prefer when transferring? Is there a way to hard disk formats match?

Is there an exact quantity difference for this mismatch?

How do I make sure the transfer is exact?


Thank you.
Engin

 

[Brian Merlen]

Copy pasting mags should not be an interns job, if you lose the shot the productions screwed. You can do check sums with red data manager. or you can do playback/scrub each clip. that tends to verify there is no corruption

 

[Melih Engin Eskier]

Copy pasting mags should not be an interns job, if you lose the shot the productions screwed. You can do check sums with red data manager. or you can do playback/scrub each clip. that tends to verify there is no corruption

Thanks for your reply. I'm not doing the transferring yet. I was just enthusiastic to learn more than I was taught. I'll certainly do playback/scrub each clip when I have the time from now on.

I was talking about the difference when you check the folders with "command+i". I was already told seeing the green tick was enough on r3d data manager.

I was hoping there were other ways to verify the data.

Thanks again.

 

[Marc Wielage]

Beyond using Red Data Manager, I know of people who use Shot Put Pro to verify copies (and copy to multiple drives simultaneously). If you're just learning, my advice is 1) don't be in a hurry, and 2) check everything three times.

 

[jimhare]

Hi Engin,

You are correct, there will be a discrepancy in folder sizes so this is not the way to confirm a correct transfer. The best way is to use a CheckSum, which ensures the copied file matches the original. This is best done with R3D Data Manager, which can do multiple copies simultaneously as well as confirm and report on all of your transfers.

I suggest you have a look:

http://www.r3ddata.com/

 

[Melih Engin Eskier]

Beyond using Red Data Manager, I know of people who use Shot Put Pro to verify copies (and copy to multiple drives simultaneously). If you're just learning, my advice is 1) don't be in a hurry, and 2) check everything three times.

Thanks Marc. I will hold onto your advices.

 

[Melih Engin Eskier]

Hi Engin,

You are correct, there will be a discrepancy in folder sizes so this is not the way to confirm a correct transfer. The best way is to use a CheckSum, which ensures the copied file matches the original. This is best done with R3D Data Manager, which can do multiple copies simultaneously as well as confirm and report on all of your transfers.

I suggest you have a look:

http://www.r3ddata.com/

Thanks Jim, that's the software we're running. I was checking for alternative rituals or softwares. I guess R3D Data Manager is more than sufficient.

 

[Will Keir]

You can checksum, use software to copy and paste but if you really want to check the footage, WATCH IT. If it looks the way you shot it and nothing crazy happens, you're good.

R3d Manager makes downloads 2x as long and is a little better than copy and paste. I have not heard of the SSD folder being a different size than the target drive. If I saw that, I'd suspect errors and recopy + visual inspection.

If you are going to use R3d Manager and are shooting a couple 128s or 256s on the day, you gotta have eSata type speed. Even FW800 doesn't cut it anymore.

 

[Cory Petkovsek]

What you have heard about disk formats is true; but not a concern.

Harddrive formats are broken down into allocated blocks, like blocks in a city street. We decide to have a fixed grid of streets in our city, so if we want to build one more building, we have to create a whole block for that building. The block will remain mostly unused until we can build additional buildings.

A file might take up 123,456,789 bytes. One disk format like NTFS defaults to 4k allocation blocks. Another disk format might allocate 32k allocation blocks. Parts of the file might be spread throughout the disk and waste unused portions of blocks (defragmentation). The file has a fixed size of 123,456,789 bytes, but the space it takes up on the disk might be 123,459,000 bytes. Copy the same file to another disk format and it takes up less space. If you are comparing file sizes, they should match. If you are comparing disk space used, they won't match. Both are shown on the windows or mac file properties windows.

Another way to verify exact content of a file is to use an md5sum on the command line. The R3D Data Manager uses something similar in a GUI.

Open the terminal, type md5, and drag an r3d to the terminal. The path fills in. Press enter on the terminal and it will process the entire file and spit out a mathematical calculation (an md5 hash). Do this on the copy and the hashes should match. You can even drag the source and copy to the terminal window, putting a space in between to do both together. These two files do not match:

$ md5 file1 file2
MD5(file1)= 5c4a2bdccf48c3e7bf7489f24ac5fcb1
MD5(file2)= 7e06cbbb761e90e2e059657927b43f5c

Cory

 

[Dustin Cross]

Engin,

The reason for the difference in files sizes can be due to different hard drive file system types (HFS+, FAT, NTFS, UDF, etc).

I am pretty sure the reason you are seeing the diffetences with R3D Data Manager is because RDM put the checksum files in the folder with the footage. So there are more files in the destination folder than in the source folder.



Dusty

 

[david_winters]

I've been using this lately, with great success. I shoot a lot of small jobs where I'm the Dir, DP, and dumping cards at lunch at after wrap at home.


http://www.reduser.net/forum/showthread.php?54803-Copy-amp-md5-Shell-Script

 

[Melih Engin Eskier]

Engin,

The reason for the difference in files sizes can be due to different hard drive file system types (HFS+, FAT, NTFS, UDF, etc).

I am pretty sure the reason you are seeing the diffetences with R3D Data Manager is because RDM put the checksum files in the folder with the footage. So there are more files in the destination folder than in the source folder.



Dusty

That's so true. I can see those files but I thought I'd better double-check if there is another reason. Thank you.

Engin

 

[Melih Engin Eskier]

What you have heard about disk formats is true; but not a concern.

Harddrive formats are broken down into allocated blocks, like blocks in a city street. We decide to have a fixed grid of streets in our city, so if we want to build one more building, we have to create a whole block for that building. The block will remain mostly unused until we can build additional buildings.

A file might take up 123,456,789 bytes. One disk format like NTFS defaults to 4k allocation blocks. Another disk format might allocate 32k allocation blocks. Parts of the file might be spread throughout the disk and waste unused portions of blocks (defragmentation). The file has a fixed size of 123,456,789 bytes, but the space it takes up on the disk might be 123,459,000 bytes. Copy the same file to another disk format and it takes up less space. If you are comparing file sizes, they should match. If you are comparing disk space used, they won't match. Both are shown on the windows or mac file properties windows.

Another way to verify exact content of a file is to use an md5sum on the command line. The R3D Data Manager uses something similar in a GUI.

Open the terminal, type md5, and drag an r3d to the terminal. The path fills in. Press enter on the terminal and it will process the entire file and spit out a mathematical calculation (an md5 hash). Do this on the copy and the hashes should match. You can even drag the source and copy to the terminal window, putting a space in between to do both together. These two files do not match:



Cory

Thanks a lot for your comprehensive explanation of hard disk workflow Cory. I now feel more comfortable.
However I have problems running md5 command. I open the terminal type md5 and drag two copies of the same file to the window before I press enter. It says there is no such existing file when I do that. I want to use md5 for every transfer I will make in the future. I'd be glad if you could help.

Engin

 

[Will Steer]

Thanks a lot for your comprehensive explanation of hard disk workflow Cory. I now feel more comfortable.
However I have problems running md5 command. I open the terminal type md5 and drag two copies of the same file to the window before I press enter. It says there is no such existing file when I do that. I want to use md5 for every transfer I will make in the future. I'd be glad if you could help.

Engin

Try this GUI version of MD5, a bit nicer than using terminal:

http://www.macupdate.com/app/mac/21726/md5

 

[Scott Crawley]

BTW, scrubbing is useless. It is possible to scrub over a bad frame all day long and never see it. Checksum baby... The only way to fly.

 

[Nick Shaw]

BTW, scrubbing is useless. It is possible to scrub over a bad frame all day long and never see it. Checksum baby... The only way to fly.

It's not an either/or. A checksum will tell you that one file is a perfect copy of another, but has no idea about the content of the file. What good is a perfect copy if the original file is corrupted somehow? You need to do checksums and a visual check to be sure. In an ideal world you would watch everything in real time, but in the real world scrubbing is often all you have time for.

 

[Scott Crawley]

It's not an either/or. A checksum will tell you that one file is a perfect copy of another, but has no idea about the content of the file. What good is a perfect copy if the original file is corrupted somehow? You need to do checksums and a visual check to be sure. In an ideal world you would watch everything in real time, but in the real world scrubbing is often all you have time for.

Good point and an important distinction, thank you. The wise DIT does a checksum to cover his own ass so it should be one of his primary concerns. If his actions were directly responsible for damaging the files he would be out of a job. Failing to alert the Camera department of a camera fault may be viewed as a lesser infraction, though none the less important. It depends upon how the team defines the position. Is that fair to say?

 

[Cory Petkovsek]

Thanks a lot for your comprehensive explanation of hard disk workflow Cory. I now feel more comfortable.
However I have problems running md5 command. I open the terminal type md5 and drag two copies of the same file to the window before I press enter. It says there is no such existing file when I do that. I want to use md5 for every transfer I will make in the future. I'd be glad if you could help.

Engin

You need to do some troubleshooting. Type md5, space, and drag one file and see if it gives you an md5 hash. Then do it with both files again. Make sure there are spaces between the command and each parameter. You are giving commands directly to the operating system and there is a specific syntax required. I showed you an example of what it looks like when it is working correctly. You can also read the manual page for md5 by typing 'man md5'.

Cory

 

[Melih Engin Eskier]

BTW, scrubbing is useless. It is possible to scrub over a bad frame all day long and never see it. Checksum baby... The only way to fly.

Haha. )

 

[Scott Crawley]

To be fair, scrubbing isn't completely useless. you can catch large errors that way, <ahem> like some camera firmware issues and completely corrupted files, but a bad frame from a write error is less than likely to be caught by scrubbing.