Checksums are dumb?

[Samuel Bilodeau]

Hey guys - as promised here's the next blog post in the series about protecting your digital assets, that builds on the concepts presented in the first post about checksums, to look at where much higher risks lie and how best practice recommendations to reduce the risk of data loss, both in DIT and throughout your whole post-production workflows.

I appreciated everyone's feedback on the first post, and added a whole section of miscellaneous DIT advice to cover a some things that we had assumed were just "common knowledge", just in case.

http://www.mysterybox.us/blog/2017/7/21/protecting-your-digital-assets-part-2-best-practices

I've started a new thread for discussion that particular blog post which you can find here

Looking forward to your feedback!

 

[Gavin Greenwalt]

I've moved well over 100 TB of data over the years and never one single time had a corrupted copy on Windows. I've moved probably 1-2TB on Finder on OSX with RED footage and it corrupted.

Let me repeat that: If your computer tells you things copied right, they copied right!

Not true and I have the corrupt R3Ds to prove it (not actually, I've since deleted them).

The one time I've had a copy corruption (on OSX) a visual inspection was fine at one half resolution but would every 10-15 minutes completely corrupt the full-resolution wavelet level. It's great to say "visually inspect" but unless you have a red rocket you aren't going to be able to review an hour or two of footage at full debayer on a laptop in the field. By comparison a 'spot check' for camera issues at full debayer, a review at 1/2 debayer and a checksum ensures EVERY FRAME actually gets carefully reviewed.

I've done enough reviews to know that if you are expecting someone under pressure to find a fucked up frame you're asking for a pie on your face. When we deliver VFX shots we put the final shot on top of the approved shot viewable with a difference blend and watch it frame by frame. It's the only way to notice something is wrong when suddenly there is a big bright white splotch where a roto shape moved. You can't expect someone to pay that close of attention for an hour straight. They're going to get highway hypnosis and slowly pay less and less attention just watching.

The chances of a corrupt copy are low. The chances of a corrupt copy coinciding with an hash collision seem somewhat astronomical to me.

 

[Wayne Morellini]

Gavin, how did you verify the terabytes of data in the file weren't altered somewhere on the bit level? As I understand it, many programs are not set up to garrantee this, but I'm definitely interested in one that do.

Aren't we looking at thousands of terabytes over a decade or more of recopying a high volume workload to new media people have told me about in the past? Talk about a waste of time compared to get long life archival media. Give me a multiple layer meter wide archival format (upto a million terabytes+ per layer hopefully shortly) I can stick one in the office, and one in a garage.

 

[Kemalettin Sert]

Gavin i moved only 120TB data last month with late 2013 rMBP running sierra and only had 1 card problem which was fully recovered on win10 with easy MBR fix.Card died in camera because of the battery not computer related problem at all.
Btw i havent used any checksum..

 

[Wayne Morellini]

It amazes me, expecting to pick up errors merely by viewing each time you backup (as some people are transferring backups yearly). You wouldn't know unless you propperly viewed comparing bits. Even a bit level fault can be devastating unless your stuff in uncompressed unencrypted. In document files even a lack of compression or encryption a bit might not douch to real text, but corrupt a word or sentence, or a few sentences then what on earth did it mean in the whole paragraph, and then what do you mean in the whole page, and even the whole document, if that page was an important one. So, going non cowboy is preferable.