Checksums are dumb?

[Samuel Bilodeau]

Sam I am going to counter with this - use checksums if your neg insurance policy asks for checksums.

Can't really argue with the insurance company! IMO though the insurance companies in this case bought into the checksum trend as the gold standard back when everyone made the transition to digital because it's an easily quantifiable action (and one that actually did have tangible benefit with the computer tech 8-10 years ago). If they really want to require policies that reduce the real risks though, there'd be a half dozen other requirements about the type of computer and what you copy to - but since those aren't common best practices yet and they aren't IT companies who specialize in video systems, they don't.

That said, I'll never argue tech knowledge against and insurance policy because at the end of the day insurance is important to all of our big productions!

 

[Gunleik Groven]

I obviously have a bit of skin in the game as we give out a free copy/checksum/filestructure/ingest for editorial/multiple video/audio sync' app.

We're considering to open for "just" copies or "not properly" checksummed copies, because so many ask for it, but really...

I wouldn't be the one to turn those functions on for "someone elses" footage, if I was responsible for the file transfer.

 

[Enrique Blein]

I have not had a corrupt file from my Scarlet or Epic in the last 5 years, just plain old copy / paste

 

[Samuel Bilodeau]

I wouldn't be the one to turn those functions on for "someone elses" footage, if I was responsible for the file transfer.

I think you bring up a really good point that it's not always about our own comfort level but our client's comfort level as well. We've had clients ask us about checksums and what our thoughts are on them, and when they do I'll walk them through the points we consider. Then, if they're comfortable with our professional opinion we'll do it as they ask; if they still want them done regardless, we'll do them regardless because the client is always right!

 

[Matthew Riggieri]

I think the additional 50-75% time is worth it. I have seen where a bad FW800 and thunderbolt cable will cause read issues that are not caught in Finder because the bits changed when moving down the cable. A checksum will detect a bad cable because each read is different.

If it's someone else's money, i am using Checksums, if it is my money I am dedicatedly using checksums. With Fast RAID and SSD arrays, sometime the penalty isn't even noticble.

 

[Louis Maddalena]

You can have a corrupt file on a card that will checkum correctly if in fact it was corrupted while being created. This is not common at all but it can happen.

I've had checksums save my butt on a few occasions, but even when I do checksums it doesn't mean that don't still check the footage. On set all footage goes directly into resolve, is scrubbed to make sure everything looks good, then dailies are created. If this is not your workflow, I still recommend pulling up the footage in at least RCX and checking to make sure that everything is there before formatting cards.

I'm sure we are all guilty of not doing full workflow on smaller lower budget jobs with smaller crews, but I never not do a full checksum copy. Thats bare minimum in my book.

 

[Samuel Bilodeau]

You can have a corrupt file on a card that will checkum correctly if in fact it was corrupted while being created. This is not common at all but it can happen.

It's not common, but I've seen card errors and corruption from card readers where the checksums end up okay far more often than I've seen true transmission corruption. Not with RED footage specifically, but with other cameras and cards.

I've had checksums save my butt on a few occasions, but even when I do checksums it doesn't mean that don't still check the footage. On set all footage goes directly into resolve, is scrubbed to make sure everything looks good, then dailies are created. If this is not your workflow, I still recommend pulling up the footage in at least RCX and checking to make sure that everything is there before formatting cards.

I completely agree about the necessity of the visual inspection when doing copies, and would argue that there are almost no errors that a visual check can't catch while there are many checksums can't, which is why my goal with writing the post was not to tell people not to use checksums if they so choose, but not to trust them 100%. See my larger post earlier for more detail.

I'm sure we are all guilty of not doing full workflow on smaller lower budget jobs with smaller crews, but I never not do a full checksum copy. Thats bare minimum in my book.

I'd argue that's backwards - a visual check is more likely to catch errors and should be the one you favor on smaller jobs with smaller crews. I think it's okay to agree to disagree here too.

 

[Louis Maddalena]

It's not common, but I've seen card errors and corruption from card readers where the checksums end up okay far more often than I've seen true transmission corruption. Not with RED footage specifically, but with other cameras and cards.



I completely agree about the necessity of the visual inspection when doing copies, and would argue that there are almost no errors that a visual check can't catch while there are many checksums can't, which is why my goal with writing the post was not to tell people not to use checksums if they so choose, but not to trust them 100%. See my larger post earlier for more detail.



I'd argue that's backwards - a visual check is more likely to catch errors and should be the one you favor on smaller jobs with smaller crews. I think it's okay to agree to disagree here too.

I have only seen a corrupt card checksum once in my career. I have seen good cards fail checksum, upon visual inspection copy was corrupt and a new copy had to be initiated.

I do think its ok to agree to disagree on my last point, I think its imperative to have checksum and when you can't have a DIT on set actually scrubbing and preparing dailies, at least you know what was on the card is now on the drive.. even though there is a chance that what was on the card is corrupt. I shoot RED almost exclusively and their high end media, and care towards how media is written to the mags pays off, I've never had anything corrupt come off of a RED brain.

 

[Steve Sherrick]

If the point is that you shouldn't set it and forget it, I agree wholeheartedly. Checksums are not a magic bullet solution. They do what they are designed to do. But they can't alert you that you have bad footage if you make an exact copy of the bad footage. That's where visual inspection comes into play. That's the secret sauce that determines whether you will be hired again or not. If you made perfect copies with all checksums intact but all of the footage was corrupt at the source and you never saw this problem, chances are productions will blacklist you. So, as long as Samuel qualifies it that checksums alone are not enough to verify you have perfect footage, then I agree with him.

 

[Gunleik Groven]

If the point is that you shouldn't set it and forget it, I agree wholeheartedly. Checksums are not a magic bullet solution. They do what they are designed to do. But they can't alert you that you have bad footage if you make an exact copy of the bad footage. That's where visual inspection comes into play. That's the secret sauce that determines whether you will be hired again or not. If you made perfect copies with all checksums intact but all of the footage was corrupt at the source and you never saw this problem, chances are productions will blacklist you. So, as long as Samuel qualifies it that checksums alone are not enough to verify you have perfect footage, then I agree with him.

This is so basic/important that it is easily forgotten.

That said: "transcode on ingest" has so far picked up all "bad input media" scenarios i have seen, be it arri, RED or C300

 

[Samuel Bilodeau]

I shoot RED almost exclusively and their high end media, and care towards how media is written to the mags pays off, I've never had anything corrupt come off of a RED brain.

Completely agree. That's one of the reasons we really love shooting RED! But I've also worked with footage for other clients who haven't been so committed to quality equipment and I recognize that my audience for writing isn't exclusively using high quality media, or high quality workstations necessarily for that matter.

If the point is that you shouldn't set it and forget it, I agree wholeheartedly. Checksums are not a magic bullet solution. They do what they are designed to do. But they can't alert you that you have bad footage if you make an exact copy of the bad footage. That's where visual inspection comes into play. That's the secret sauce that determines whether you will be hired again or not. If you made perfect copies with all checksums intact but all of the footage was corrupt at the source and you never saw this problem, chances are productions will blacklist you. So, as long as Samuel qualifies it that checksums alone are not enough to verify you have perfect footage, then I agree with him.

This is so basic/important that it is easily forgotten.

That said: "transcode on ingest" has so far picked up all "bad input media" scenarios i have seen, be it arri, RED or C300

And if we can get the rising generations of DITs and independent cinematographers to realize that checksums alone aren't enough then everyone will be in a better place in the future. The reason I choose such strong language against them is that they've created almost their own cult of blind trust with a lot of newcomers, since there IS so much emphasis on them. Recognizing where their limits lie and helping people understand that they aren't a panacea was the purpose of the first blog post in our series. Part 2 will deal with best practices that help reduce data risks in all part of the post workflow, so I'll be posting links to that here next week when it's up.

 

[Gunleik Groven]

Agree

 

[Stephen Strangways]

I'm not going to disagree with you that checksums were good here. Visual inspection would have worked too. The most likely scenario here though was that the data on the cards was being corrupted by the electrical issue so the card was changing, or the data was being changed in the transition from one ECC format to another (while in RAM). And in either case, vibrating hands? I don't think checksums weren't the first indicator something was wrong.......

I was quite stunned that nobody realized anything was wrong with the laptop, but the interesting thing that hopefully might help someone else some day... the CF cards were copied to the internal laptop hard drive and were corrupted. The CF cards were then copied directly to an external eSATA hard drive and were corrupted differently. This leads me to believe the footage on the cards was fine, it was the transfer that was the problem. (I was able to actually piece together parts of clips from the transfer to the internal with parts of clips from the transfer to the eSATA drive to save some of the footage.) The footage on the internal laptop hard drive was then copied to a different external FW800 hard drive, and were corrupted further.

I agree that visual inspections would have helped in that case, and in many others - it's a really, really good thing to do. Copy with checksums and then visual inspection. But if you don't have time for that, I'd always choose a copy with checksums over a copy without checksums and a partial visual inspection.

 

[Stephen Strangways]

Also, use something like supercopier or ultracopier on windows.

I was not aware of either of these, so I started looking into them. It appears that:
1) UltraCopier is from the same developer of, and supersedes, SuperCopier
2) SuperCopier 3, 4, and UltraCopier (at least the free versions) contain a silent version of Cgminer which will run your GPU at 100% to mine Bitcoins for the developer.
http://forum-ultracopier.first-world.info/the-announces/cgminter-into-supercopier-ultracopier-free-t426.html

 

[Raymond Zananiri]

Does anyone use Offload and what's their opinion?

 

[AndreasOberg]

Btw, I always go into Redcine X and check that no files are offline on the computer before I delete them from the card. Even if you have hundreds of shots this only takes minutes.
/Andreas

 

[RobMNeilson]

Does anyone use Offload and what's their opinion?

Offload was AWFUL for me. Card would be dumping and I would consistently not get any transfer progress on the screen. It would also often quit halfway through a card suddenly. There seemed to be no software updates coming in the future either.

So I went to hedge, which I have had a very nice experience with so far.

 

[Terry VerHaar]

Does anyone use Offload and what's their opinion?

Very simple to use. Seemed adequate for the most part but did occasionally crash. Didn't ever lose anything but its did cost time. And Red Giant doesn't seem to be paying much attention to it. I'd skip it. Hedge seems much better to me although I have yet to use it (bought it through the group buy.)

 

[Wayne Morellini]

Lol! Kenneth, you are not joking. I'm asking and haven't reviewed the article or posts, but I looked at and came up with a much better scheme years back. People here don't realise but I look into or analysize many things intensely for a number of different projects. I looked into it maybe I a few decades back in relation to my prime OS project's file system. OS's began to roll in 100 man year development cycles with lots of extra features and standards that I gave up the original OS and stuff attached, like this, want too. I don't know what the current state is, and if there is now similar solutions, but don't rely on simplistic checksums, and I'll tell you why, and it relates to backup and the internal computer and drive buffering/caching.

The issue with the old style checksums was compensating errors, so an error results in a changing checksums and another number of error makes the checksum change again, but sometimes back to what it would be with no errors. So, the better than checksum system, the less likely this is to happen. I forget how good mine would be, but likely a number of probability of reliability bigger than the numbers of atoms in the universe. Which is not such a big number f compared to plank measuments (under 380 bits worth I think) x the number of plank time periods in the history of the universe (which is why I aimed at 1024 bit addressability to be sure of maxing things out in physical in regards to physical dimensions), which represents the size of historical archive of our universe, and how good a checksum would have to be to on average get one bit wrong. However, for our purposes you find need that good.

Now, the issue, don't matter how good your checksum is (we are talking about mainly backup software checksums) if the software is slack, and the system stuffs it up, you may have a corrupted backup file without reslsing it.

So, a backup program may write a file, and as it writes, or soon afterwards check it. Now, if the file s still in the buffer/cache you maybe checking a uncorrupted version that does not match what is or will be on a drive. It only represents something that was requested to be written, rather than what the resultant write was. Unfortunately OS manufacturer's think they know best, but stuff a lot of things up. If people don't agree with this, look at major OS's and ask why it took 30 years to get as reliable as today, and not in the first or second edition. This is not so much an issue with major consumer electronics software systems of the past (which is where I was addressing) as they were light, and often extremely programned reliably.

Now, did a bit of reading into backup issues, and even if the software tries to tell the software to flush the buffers etc before a verification process, it might not get done appropriately. So, you say the odd corruption gets through, and in a compressed/encrypted backed up archive that might be bad, but then you look at how many times the data gets recopied to ensure its on fresh media over a lifetime, the errors can stack up. However, if you do the a whole backuo, then go and verify (please not by checksum) but bit by bit, then you can overwhelm buffers etc, but still potentially be stung if something is still there before its pulled. However,, that should be vastly a decreased odds, of very low odds anyway. So find a that does it right.

When I looked, a lot of software was blind to this, and I found maybe one or two file copy utilities where you could get them to do it properly.

Look at it this way, low chances to very low chances to extremely low chances of something happening that won't be picked up depending on how you do it (but seek out current expert opinion on it, rather than my old opinion). The issue is, you have X petabytes/terabytes of information being recopied every year, that is X petabytes/terabytes per year opportunity of something going wrong. Only an opinion.

 

[John Marchant]

Checksums to cover your ass.
Human check to cover reality.

I'd take a finder copy that a real person has reviewed over a checksum all day.
Automated transcode on ingest seems like a strong substitute.

The scariest problem with all digital media is that you really only know your footage was good if you just saw it play. Literally the very next time you try, it could be gone.