Checksums are dumb?

[Kenneth C Merrill]

According to Mystery Box checksums are a waste of time and give you a false sense of security. Any thoughts?

http://www.mysterybox.us/blog/2017/7/5/protecting-your-digital-assets-part-1-checksums-digital-errors-debunked

I don't like going against anything Sam or the rest of the Mystery Box crew have to say about imaging technology because I feel like they have a lot more knowledge and experience than I do. However, I've had 3 instances of corrupted footage in the last 2 years. These three instances also happened to be among the very few times that I copied footage from a mag without a checksum. Now when I'm making the decisions I ALWAYS use checksum software because the few times I didn't, I got burned.

It certainly could be a correlation-not-causation situation, but that's just my experience.

 

[Martin Stevens]

I use "Allway Sync" a lot and unfortunately there is no checksum option.

 

[Cameron McKinlay]

Not sure who mystery box is....but you can't argue facts.
MD5 Checksums validate the copied file against the original file to check for corruption or inconsistencies.
Much like yourself, I had lost some valuable footage just using finder to copy the footage. I now ONLY use shotput pro or Hedge, which just picked up through the group buy. Haven't had a corrupt file in months.
So take that info from Mystery box with a grain of salt...it may not be the best advice.

 

[Kenneth C Merrill]

Sorry, I thought I pasted the link. It's now in the OP.

They use some pretty convincing language to put down the reliability of MD5 and hash checksums. I'm curious if anyone has any counterpoints to their technological argument. I'm sure we all have anecdotes we can cite, but those could technically be coincidental.

Thoughts?

 

[Stephen Strangways]

I worked on a show years ago where a metal-bodied Mac laptop was being plugged into an ungrounded AC outlet in a third-world country, and it was electrically live - users reporting feeling the laptop "vibrating" as they touched it. AC current was passing into their hands!

Copying footage off of RED RAID and CF media (yes, this was the RED ONE) with R3D Data Manager was throwing up checksum errors. So, they "solved" it by doing drag-and-drop copies with Finder, which threw up no errors at all. They thought everything was fine, that they could trust Finder and that R3DDM must have a problem. When they got the footage back to the office... 75% of the clips were corrupt. The grounding issue was corrupting files copied to and from external media.

So... you want to tell me again that "If your computer tells you things copied right, they copied right!" Bullshit. I know that's not true, on Mac or Windows, from many different experiences.

If I copy footage, it's with Double Data or ShotPut Pro. I run TeraCopy as the default file copier on all Windows machines that I use.

And as for checksums... I can go back to that footage we shot on the Red One all those years ago, grab one set of drives from the triplicate backups we have, stored in several different locations, and I can verify the checksums on just one set to make sure the footage is still fine. If I didn't have checksums, I'd have to get two different sets of backups and compare them to each other, and after all of that, if there was a difference, I wouldn't know which one was the uncorrupted copy.

Checksums are useful.

"when Windows or MacOS tells you the file’s copied, it’s actually copied" is not true.

 

[Marc Wielage]

One thing I'd ask the Mystery Box people is this:

have you ever encountered a file that was bad when you did a Checksum copy, and the Checksum indicated it was good?

I have absolutely had corrupted files copied in the finder, so it happens. I would agree with them that it's extremely rare, but it does happen. But I've never had it happen with a Checksum, not ever.

I think they're trying to establish a conclusion and then coming up with an argument to justify it. I also think the only argument you can make against Checksums is that it takes more time.

 

[Patrick Tresch]

On documentry work I don't use checksum but drag and drop (and I didn't have any corrupted files) but check the bit size... BUT allways make two copies DIRECTLY from the SSD card. Two copy errors on the same card would be strange.

 

[Cameron McKinlay]

"the file will go through a multitude of bit-for-bit checks designed to make sure that when Windows or MacOS tells you the file’s copied, it’s actually copied. No additional hashing or checksums needed."
Doing some digging this is what I found about MacOS and Checksums...
MacOS does use checksums, but it is only for metadata structures such as volume header and journal header, not for the actual data that you really want to verify....like footage.

 

[Stephen Strangways]

Well done, Cameron. I think that's exactly the counterpoint to their technological argument that Kenneth was looking for.

The biggest gap I see is in verifying the read - Teracopy is far from perfect, because it reads the data from your source once, generating a checksum in the process. It then reads the destination back to verify against the checksum that the data was written correctly... but it never re-reads the source to make sure it was read correctly in the first place! R3D Data Manager always did that. I don't know about Double Data for sure. I don't think ShotPut Pro does. From Hedge's website it appears that it doesn't - "Hashes are created when reading from the source disks, and while the copy is being written to the destination."

Ideally, what I want is a system that reads from the source and writes to the destination, generating an xxHash64 checksum as it is reading the source. Once the data is written, it simultaneously reads from both the source and destination to generate new xxHash64 checksums, and then verifies they both match the first checksum. So, three checksums: one from the first read of the source, a second from a second read of the source, and a third from a read of the destination.

 

[Martin Stevens]

Ideally, what I want is a system that reads from the source and writes to the destination, generating an xxHash64 checksum as it is reading the source. Once the data is written, it simultaneously reads from both the source and destination to generate new xxHash64 checksums, and then verifies they both match the first checksum. So, three checksums: one from the first read of the source, a second from a second read of the source, and a third from a read of the destination.

I wish for this as well.

 

[Marcus Struzina]

Is there any reason not to still use R3d Data Manager? And how many people are using Resolve to copy cards are there any problems with that?

 

[Bérenger Brillante]

Double back up from the card itself is fine for everyday job.
But.
I once shot in a abandonned powerplant and told my assistant to stay away from walls and cable, and everything magnetic/electric. But he didn't recognized the electrical coil beside his cart, and they where still charged. Every hard drives he mounted had erratic behaviour, but no errors.

After a few copies, he tells me there is something wrong with the footage and I follow him near the cart where I see the coils. We had no Checksums, but I bet if we had one, It would have said somethings. The minutes we backedup away from the coils and used new disks, everything was fine. It costs 4 external hard drives and 1 internal, the content of 2 backups on a 3 cameras shoot.

For a short about Warhammer 40 000 with a space marine played by K1 star Jerome Le Banner, and about 10 stuntmen. I'm happy it was Red One with 8gb CF only, as we lost only 3 (critical) shots, and managed to redo the others.

Checksum on hard location. In studio, copypaste and playback. You have to trust the computer you are given. I use only mine. If it's the production mac book air, checksum. if it's rental, checksum.

Also, use something like supercopier or ultracopier on windows.

 

[Mark L. Pederson]

So... you want to tell me again that "If your computer tells you things copied right, they copied right!" Bullshit. I know that's not true, on Mac or Windows, from many different experiences.

Yup.

I can tell you 100% - from experience - he is dead wrong on this topic. It's very rare - but finder can 100% fuck you. Mac and Windows. When it happens - boy ... does it hurt.

And for the record - Windows does not perform error detecting when copying files & OSX only verifies metadata & headers.

 

[Alex.Mitchell]

Hopefully with the advent of APFS data security will be easier for newbies to manage but until then I cannot recommend Silverstack, Shotput, Hedge, etc. enough. They're all going to make your life so much easier and more secure.

 

[jacob.schwarz]

As a member of Mystery Box I think you guys all bring up valid points and undoubtable experience about this issue. I'll leave our post guy Sam answer and defend our blog post (i just told him to get on the forum). Hopefully it will lead to an insightful discussion, which is the intent of our blog.

Speaking for myself personally, I have never had an issues with data offloading media, but i do agree it happened. I've had a ton of issues with record media going bad (Cfast Cards especially!), but there was nothing that i could do to prevent or predict that beside offloading often, which we do, and buying official and certified record media (like buying directly from Red or certified CFast Cards by Arri).

All my errors and issues with data have all originated from the source media, and so I would rather copy and verify in a editor or RedCine-X once it's copied that it all there and working than take the extra time to use checksum, and if it not i always check the RAW files on the source media.

Again, this is my personal experience and don't reflect the vast amount of experience of all you and of our Post-Supervisor who wrote the article. I agree take the article with a gain of salt, just sharing our knowledge and experience.

We have been using Hedge lately a lot and LOVE it! Just bought another 4 more copies. It copies files faster with checksum information faster than finder copy just in case you want that extra piece of mind. However, i always check my media once it copied in RedCine-X or editor to make sure it all good.

 

[Cameron McKinlay]

We have been using Hedge lately a lot and LOVE it!......For piece of mind

How ironic

 

[Samuel Bilodeau]

Hi guys, thanks for your comments on the post. I appreciate reading all of the concerns about our blog post that's anti-checksum. I want to respond to some of them directly, which I'll do in a minute here, but first I want to state a couple of things upfront that I think got lost:

First, the major premise of the post is that a reliance on checksums give a false sense of security - implying you don't need to verify your footage in other ways to make sure the camera, computer, hard drives, or everything in between is working. The vast majority of data corruption happens from camera errors, card errors, card reader errors, and electrical interference, and with the exception of electrical interference checksums usually will not indicate any problems with the footage when files are damaged in this way: when you copy a corrupt file, it'll checksum completely after the copy and say everything's okay.

Second, I used pretty strong language to emphasize the insecurity of checksums. I'm not anti checksums, especially when they're being used by professionals - I understand that many people find great comfort in watching the numbers say that it's fine. But again, when this is the only form of verification it's grossly insufficient for detecting errors. So I chose to use that language to emphasize that you cannot rely on checksums or automated verification procedures to 'check' your footage.

Third, there are exceptions to every rule. Bérenger Brillante makes an important point about your equipment and I'm always more hesitant when using borrowed or rental equipment, though as often as possible I still make sure to thoroughly test the equipment and make sure it's working as expected in advance. I will still never use checksums alone to verify the footage, even on location. Visual inspection and playback are a must because they'll reveal more errors than any set of checksums.

Lastly, unless something bigger is going wrong with your hardware (or software, but usually hardware), at worst you'll get a single bit or burst error in footage if things aren't 'bit for bit' accurate. I didn't discuss this in the post because it was slightly off topic. But the effect of single bit errors and even small bursts are usually limited to a single block within a single frame, and in most cases are indistinguishable from noise. Even if the whole block is lost, the time and cost calculation says that it's far easier to on a very, very, very rare occasion patch a single block using information from adjacent frames than to spend additional hours and hours on a regular basis running checksums.

Okay, that said, I want to answer a few specific comments below:

However, I've had 3 instances of corrupted footage in the last 2 years. These three instances also happened to be among the very few times that I copied footage from a mag without a checksum. Now when I'm making the decisions I ALWAYS use checksum software because the few times I didn't, I got burned.

It certainly could be a correlation-not-causation situation, but that's just my experience.

You can still get burned even with checksums. It's only a part of the confidence package. Visual inspection is the only way to make sure that you're not getting corruption from problems with the camera, card, card reader, or drives. I'm not saying don't checksum because it's hard to argue when you've had that kind of experience, even if it's very likely that it's only correlative.

I worked on a show years ago where a metal-bodied Mac laptop was being plugged into an ungrounded AC outlet in a third-world country, and it was electrically live - users reporting feeling the laptop "vibrating" as they touched it. AC current was passing into their hands!

Copying footage off of RED RAID and CF media (yes, this was the RED ONE) with R3D Data Manager was throwing up checksum errors. So, they "solved" it by doing drag-and-drop copies with Finder, which threw up no errors at all. They thought everything was fine, that they could trust Finder and that R3DDM must have a problem. When they got the footage back to the office... 75% of the clips were corrupt. The grounding issue was corrupting files copied to and from external media.

So... you want to tell me again that "If your computer tells you things copied right, they copied right!" Bullshit. I know that's not true, on Mac or Windows, from many different experiences.

I'm not going to disagree with you that checksums were good here. Visual inspection would have worked too. The most likely scenario here though was that the data on the cards was being corrupted by the electrical issue so the card was changing, or the data was being changed in the transition from one ECC format to another (while in RAM). And in either case, vibrating hands? I don't think checksums weren't the first indicator something was wrong.......

One thing I'd ask the Mystery Box people is this:

have you ever encountered a file that was bad when you did a Checksum copy, and the Checksum indicated it was good?

YES. I was dealing with a client just last month where this happened, and I've had it happen internally too. Usually it has to do with hardware errors on the camera, card, card reader, etc. Visual inspection has caught errors like this and made sure that we don't shoot all day on a camera with corrupted firmware.

I have absolutely had corrupted files copied in the finder, so it happens. I would agree with them that it's extremely rare, but it does happen. But I've never had it happen with a Checksum, not ever.

I think they're trying to establish a conclusion and then coming up with an argument to justify it. I also think the only argument you can make against Checksums is that it takes more time.

I'm not defending Finder or Windows Explorer here, but I also have been working IT for the better part of 18 years, and experience has taught me that things usually aren't what they seem. In almost every case where I've seen Finder or Windows Explorer silently have a problem with a copy, it's been because of other root causes - hardware issues, memory leaks in another application that's running, too much IO on a bad controller that got its commands crossed, things like that.

I think that's a fair point of view though to assume that I have a conclusion about checksums. My next blog post will make clear the reasons for why I value copy speed and time over checksumming, but the short version is that your odds of data loss drop by many orders of magnitude with the first duplicate copy you make. When your choice is between getting a copy 4-5 x faster, or even 1.5 - 2x faster and running checksums, you're better of getting the copy faster and doing the visual inspection, right away. Most applications that run checksums, with the exception of programs like Hedge that trades off RAM caching the data for not adding separate read operations to the original media for checksum calculation, need additional time to prep a MHL or other set of hashes to act as checksums - there's a lot that can happen in just an additional 10 minutes when it comes to copying footage, and if you want to reduce the actual risks of data loss, minimizing the time it takes to get that second copy is essential.

Doing some digging this is what I found about MacOS and Checksums...
MacOS does use checksums, but it is only for metadata structures such as volume header and journal header, not for the actual data that you really want to verify....like footage.

Correct. However, every protocol in between does. You get the first step of error correction by the media controller on read, then the data is reencoded into self correcting error code for USB or Thunderbolt or Network or PCIe transmission at the hardware level. Every time its moved from one place to another it's error corrected, and then finally stored as self correcting error code on your destination drive. MacOS and Windows themselves usually don't actually 'access' the data directly on modern computers. It's rarely held in RAM and is instead passed from device to device. The volume header and journal headers are checksumed by MacOS because they're created by the OS as part of the HFS+ file system, which it really doesn't want to get corruption in so it does that additional data verification pass and stores the checksums as data.

To be clear and to reiterate: I'm not saying that any of the operating systems run checksums during copy or checksum the data. I'm saying that in every step except for the RAM itself data isn't stored or transmitted as 'pure data', but self correcting error correction code that represents the data and heals itself of any transmission or storage errors. And even then, if you're using enterprise / server/ workstation hardware with Xeon or the equivalent processors and ECC RAM, such as you'll find in a MacPro, the forthcoming iMac Pros, or workstation laptops, it's stored as error corrected data there too!

Continued.....

 

[Samuel Bilodeau]

Continued..... Part 2

Well done, Cameron. I think that's exactly the counterpoint to their technological argument that Kenneth was looking for.

See above.

Ideally, what I want is a system that reads from the source and writes to the destination, generating an xxHash64 checksum as it is reading the source. Once the data is written, it simultaneously reads from both the source and destination to generate new xxHash64 checksums, and then verifies they both match the first checksum. So, three checksums: one from the first read of the source, a second from a second read of the source, and a third from a read of the destination.

See my note about speed. This takes time, and if you cache to RAM and it's not ECC RAM you're actually introducing more risk that the copy will go wrong.

Double back up from the card itself is fine for everyday job.
But.
I once shot in a abandonned powerplant and told my assistant to stay away from walls and cable, and everything magnetic/electric. But he didn't recognized the electrical coil beside his cart, and they where still charged. Every hard drives he mounted had erratic behaviour, but no errors.

Erratic behavior is usually the first indicator something's wrong!!!!!

After a few copies, he tells me there is something wrong with the footage and I follow him near the cart where I see the coils. We had no Checksums, but I bet if we had one, It would have said somethings.

That's an interesting what-if. The likely reason that it said it copied fine was that the data was corrupted not on write (it would have written okay and passed internal verification), but shortly after it was depolarized or altered because of the strong EM. Depending on how quickly checksums were read after that write it still may not have caught it. Consider adding a grounded Faraday cage to your DIT cart or hard drive array next time you're around that much power.......... Also most RAID enclosures act as Faraday cages so copy to a RAID when possible. There are other reasons that's always the best idea, but I'll be covering those in the next blog post about best practices.

Checksum on hard location. In studio, copypaste and playback. You have to trust the computer you are given. I use only mine. If it's the production mac book air, checksum. if it's rental, checksum.

Visual playback as well on location, but I think this is good to remember - always double check your equipment, and always visual inspect even if you checksum.

I can tell you 100% - from experience - he is dead wrong on this topic. It's very rare - but finder can 100% fuck you. Mac and Windows. When it happens - boy ... does it hurt.

And for the record - Windows does not perform error detecting when copying files & OSX only verifies metadata & headers.

See above RE:Windows and MacOS. I'm not going to disagree that some of my points were misunderstood or unclear, and I whole heartedly agree that some standalone copy programs run better than Finder or Windows Explorer.

As for being dead wrong, I'll respectfully disagree and present a counterpoint. I think it's important to not put so much faith in checksums that you ignore the other threats and best practices because of them, which is the major concern which prompted me to write the post and to use such strong language against them. If I've put doubt in anyone's minds about the usefulness of checksums and they start taking additional steps to take care of their footage than just checksums alone, then I think I've made my point. After reading the post, I think most people will have the same reaction that Kenneth did: start questioning their efficacy. For professionals and quality DIT establishments, I doubt anyone will stop using checksums. But when you consider the number of independent producers, cinematographers, small production companies etc who treat checksums as the gold standard of DIT and never go any step further in asset protection there's a lot of value in encountering a counterpoint opinion. Checksums are not enough. And in my opinion, it's better to not do checksums than to ignore the checks that actually make a difference in protecting your footage.


Anyone who feels more peace of mind when using checksums than ignoring them, by all means, use checksums! I'd recommend using Hedge because it gives you speed AND checksums, but it's the speed of making those duplicate copies that really makes a difference in terms of minimizing actual risk. But if you rely exclusively on checksums at the expense of other best practices, consider incorporating additional visual verification passes into your existing workflows, as well as the recommendations we make and I'll cover in a blog post next week, because that's a change that's more likely to help you

 

[Craig Parkes]

Sam I am going to counter with this - use checksums if your neg insurance policy asks for checksums.

Visual checks are good and ideal, but an insurer isn't going to not uphold your policy if a person missed an errir on set - humans are fallible. If there isn't a checksum record, then that fallibility is on you.

I'd rather be slow, miss an issue and be insured then fast, miss an issue and not be insured.

Checksums help you spot issues, visual checks only may decrease the chances of you missing issues / save a production half a day, but if you screw up and there id no checksum that failure is on you alone.

 

[Marc Wielage]

I can tell you 100% - from experience - he is dead wrong on this topic. It's very rare - but finder can 100% fuck you. Mac and Windows. When it happens - boy ... does it hurt. And for the record - Windows does not perform error detecting when copying files & OSX only verifies metadata & headers.

That is exactly my experience as well.

Checksums are not a replacement for also visually inspecting the pictures by eye and listening to the dialogue when it's synced up. Everything is important. One glitch can ruin your entire day.