13 threads on CORRUPT frames in less than 2 MONTHS

[Eirik Tyrihjel]

I apparently had serious issues on two 16 Gb cards today, using build 30 with MX upgraded cam, shooting 4.5 K, RC36.
I regularly checked most of the files on camera throughout the day - so I felt fairly confident - like I always do (I have lost very very few takes in the last 2 and a half years - and always had an ERROR displayed at the end of the takes I lost), but upon return from my shooting day I went ahead to unload/back up the data from the 16Gb cards.

I got errors, the first 5-6 takes where ok, the rest were unreadable.
I tried REDundead on one card, it found a lot of old stuff, but only the 5 first takes of 12 from the material I shot last, and needed.

I then tried the card with a different Cf reader of the same brand (Sandisk Cf reader 800FW), and all of the sudden THERE IS NO PROBLEM!

Everything is there!

I was really mislead by all the stuff REDundead was able to uncover (I didn´t think there was place for the files I was missing - but I was wrong), and I was also mislead by the fact that all the 8Gb cards worked as expected.

Weird stuff - but in my case:
One CF Card reader produced the errors, all of them, I could easily have been fooled - and lost the "lost" takes, luckily I tried the other CF reader and all ended well, nothing lost!

The important thing is: Check everything, try a different cable, a different computer, a different CF reader
and never assume anything! And then check again!

Accidents do happen, I acknowledge that data is lost from time to time, but more often than not it is caused by faulty components or user error. In my case it was very very easy to look at the 16Gb cards and the camera system and think: "there must be some bug in there" - while all along it was obviously the Cf reader.

I have one CF card reader now with a RED TAPE on it, I will check it out a bit (has no bent pins) but most likely it´s retired!

When I downloaded the files, I got half of them before the errors came up

 

[Zakaree Sandberg]

Zak, remind me never to hire you. :thumbsup: What Zak is saying that the human element is always there. We don't count frames as they speed by before our eyes. All it takes is one of our blinks to line up with the wrong frame, and it all goes out the window. And he is also reminding us that footage checkers are not flawless either, unfortunately.

All of which supports my point, which makes me a genius, and I will be expecting my money in the mail. ;P

I did a visual check.. did not do a transcoding check.. they didnt want to pay for a kit rental.. so they got 2 macbook pros. I wasnt going to transcode each clip for free. I was hired to make sure the cameras were all set up properly, check exposure, and data manage.. Which I did.. Like I said, everything came out looking good.. but there was a single frame somewhere in there that was corrupt, didnt show up on any of my visual checks or checksum checks.

Besides.. this is all what I heard about from post.. I never confirmed that there was any corrupt frame, they just told me that on a 2 or 3 clips, they werent able to finish a transcode.. could have been on their end.

the only way for me to catch would have been to transcode each clip...
not happening on the rate they wanted to give.

 

[Roberto Lequeux]

Just to be 100% clear, I was totally kidding. I know that the usual RedUser suspects know the camera in and out and many of them make for some of the best RED DITs out there. In fact I am only on this thread to learn.

 

[Kim Frank]

I did a visual check.. did not do a transcoding check..
Like I said, everything came out looking good.. but there was a single frame somewhere in there that was corrupt, didnt show up on any of my visual checks or checksum checks...
the only way for me to catch would have been to transcode each clip...
not happening on the rate they wanted to give.

Zakaree, that really frightens me. Of cause no one can exept you to transcode the clips on a small job. And if that really would be the only way to be 100% sure, that would really be a pain.
I don't want to blame you in any way for doing something wrong, cause you obviously didn't but could you share how you veryfied visually.
Because what I know is that I could always scrub through the Footage in RedcineX but the clips with corrupt frames wouldn't play. Did you scrub or play? Aslong as checksumms can be correct even with corrupt frames, there MUST be a way to at least check the clips visually and be 100% sure without needing to transcode every single take. I'm mainly shooting music videos and on a fast moving set with a lot of setups and takes of often over 3 min. length this would be an absoloute nightmare. I'd need tons of CF cards to be able to wait for a finished transcode before deleting the original media but be able to keep on shooting. There has to be another way and if not RED has to invent a way. Wouldn't it be somehow possible to have redcinex scan the R3Ds to make sure it can transcode?

 

[Scott Crawley]

So the lesson here is:
checksum, scrub, transcode, real-time visual check of transcoded footage, 2x backup, only then should you reformat media.

Conclusion: Red Rocket in the hands of an intelligent and conscientious DIT on set is a MUST for a reliable and economic workflow.

Sound right?

 

[Kim Frank]

Not completely, because the Red Rocket is most of the time able to transcode clips with corrupt frames. So you would need to backup, transcode than do a real-time visual check. If you spot some corrupt frames (and you'd have to be concentrated to do so) you'd need to start the whole thing all over again. And that realy isn't a solid and realistic on set offloading workflow. Even with real time red rocket transcode (that still needs to be stored and back uped) you would end up with about three times the length of each clip for offloading - transcoding - verifying
not possible on most sets in my opinion

edit: just reread your post and saw that this was exactly what you wrote. So yea

 

[Greg M]

The key is to have enough media in field so that you dont have to reformat immediatly, giving the DIT time to manage it properly. We constantly have rental requests with a package that only has 2 drives...foolishness.

Generally when I shoot, I have enough media to get me thru a complete day. This way we wrap with 2 complete sets of copies + the original media intact (or as much of it as possible). If you are re-formatting each drive immediatly after download, your chances for loss are greater.

 

[Kim Frank]

Is there a way of thinking of an application that would compare the original data with the offloaded data. Maybe something like an waveform or Histogram comparison after offloading, or maybe even while offloading. Of cause that would need processing time, but it could happen in the backround. Epic will have in camera crruption check so this would eliminate this problem forever. Maybe the camera could produce a kind of histogram data, like the Quicktime reference files, that then could be used by the application for comparison. Would that be possible in any way or is this idea complete bullshit

 

[Kim Frank]

The key is to have enough media in field so that you dont have to reformat immediatly, giving the DIT time to manage it properly. We constantly have rental requests with a package that only has 2 drives...foolishness.

Generally when I shoot, I have enough media to get me thru a complete day. This way we wrap with 2 complete sets of copies + the original media intact (or as much of it as possible). If you are re-formatting each drive immediatly after download, your chances for loss are greater.

Most of the time I shoot on CF cards. By an daylie output of over 300 gig (musicvideo) I would need a minimum of 10 16gig cards. Maybe I will have to consider this...

 

[Greg M]

Most of the time I shoot on CF cards. By an daylie output of over 300 gig (musicvideo) I would need a minimum of 10 16gig cards. Maybe I will have to consider this...

Correct, minimum.
Extra media is a requirement imo. Every time you turnover your media you are significantly increasing the risk of errors. You will need to balance this risk with budget, but I keep media at the very top of my priorities list.

 

[Gunleik Groven]

Not completely, because the Red Rocket is most of the time able to transcode clips with corrupt frames. So you would need to backup, transcode than do a real-time visual check. If you spot some corrupt frames (and you'd have to be concentrated to do so) you'd need to start the whole thing all over again. And that realy isn't a solid and realistic on set offloading workflow. Even with real time red rocket transcode (that still needs to be stored and back uped) you would end up with about three times the length of each clip for offloading - transcoding - verifying
not possible on most sets in my opinion

edit: just reread your post and saw that this was exactly what you wrote. So yea

Rocket reports bad frames, (@ least RocketCine-X) even though it manages to transcode around it.

Love Rocketcine-X. Stable. Working properly. Can be conformed to. Renders around bad frames. Reports about bad frames.

Love it!!!
Wish it was the codebase for RedCine-X and not the other way around.

 

[Dustin Cross]

Most jobs I do, don't have enough media to not cycle media and I almost never have enough time to play all footage in realtime. A job I jut did shot 6 hours of footage in one day. Just getting everything downloaded to three locations and all transcoded to MXF files took all the time I had and couple hours after wrap. Scrubbing was the only option.

Red CF readers will hopefully help. A Red download device for the Red Drive would be great, but I don't see that happening, since it seems Red has abandoned drives.

I am not sure how something like R3D Data Manager can do a md5 checksum without errors on two files and only one be corrupt. Conrad, is that possible?


Dusty

 

[Erich Ocean]

Every time you turnover your media you are significantly increasing the risk of errors.

You're absolutely right, but the problem isn't "the risk of errors", it's that RED has no end-to-end data integrity scheme. It should.

The truth is, hardware components consistently fail in ways that are both easy and hard to detect. No amount of conscientiousness or good practice can prevent errors from happening to you. With end-to-end data integrity, you switch from *avoiding* problems to *expecting* problems -- and designing the system as a whole to detect and correct those problems automatically and in an end-to-end fashion -- for the entire path from the RED sensor to your portable, redundant archival medium. That's what a computer does best, and it's what humans do the worst. This is the end game, this is where we *must* be long-term.

As I stated above, end-to-end data integrity is a solved problem. REPEAT: it's a solved problem. The technologies work, they're cost effective, and they're not encumbered by patents or other licensing issues. RED needs only to do the engineering work to implement them. I'm certain they will eventually (and it sounds like they've already started down that path with Epic).

P.s. For those who are thinking "end-to-end data integrity is a pipe-dream", I humbly submit the Internet and TCP/IP as an example of a system designed with end-to-end integrity. Somehow, trillions of bits are transmitted accurately over then internet day-in, day-out without issues -- and we're having problems copying data from one volume to another.

 

[Kim Frank]

Rocket reports bad frames, (@ least RocketCine-X) even though it manages to transcode around it.

Love Rocketcine-X. Stable. Working properly. Can be conformed to. Renders around bad frames. Reports about bad frames.

Love it!!!
Wish it was the codebase for RedCine-X and not the other way around.

Where or how does Rocketcine-X report bad frames?
Do you have to transcode to get this report?
Does Redcine-X do the same?

 

[Kim Frank]

As I stated above, end-to-end data integrity is a solved problem. REPEAT: it's a solved problem. The technologies work, they're cost effective, and they're not encumbered by patents or other licensing issues. RED need onlyto do the engineering work to implement them. I'm certain they will eventually (and it sounds like they've already started with Epic).

Erich, could you elaborate where or how this Problem is solved?

 

[Zakaree Sandberg]

extra media for sure... this production gave me 3 16 gig cards so it was a hustle to quickly dump and check..

To visually check i opened up in redcine-x, scrubbed to make sure there were no obvious issues, and played them back in a low resolution (not all of them though, as I didnt have much allotted time.. I just had to randomly pick a few to watch fully)

 

[Erich Ocean]

Erich, could you elaborate where or how this Problem is solved?

Sure! Adopting ZFS on the camera (http://en.wikipedia.org/wiki/ZFS#Copy-on-write_transactional_model) would be the simplest, fastest way for RED to do it. ZFS is a valid Linux filesystem, and it can be run in user space on any OS (Windows, Mac OS X). It's open source, free, and works great. It's incredibly robust. (AFAIK, the RED One runs Linux.)

Additionally, RED would make a transfer app for Windows and Mac OS X that would read the data off of a CF card formatted with ZFS and copy it to whatever volumes people were using (NTFS on Windows, HFS+ on Mac OS X). Ideally the copy would expand the size of the files with additional redundancy blocks (there are lots of schemes for doing this, including some that wouldn't required changes to how .r3d files are used by other apps), so that errors on the new volume (which is not as robust as ZFS) can be repaired later if needed. All tape drives (e.g. LTO) do this automatically internally, as well as optical discs. Hard discs with non-ZFS filesystems don't (in general). On Mac OS X and Linux, users could also use ZFS as their filesystem if they want to maintain the data integrity after the data is out of REDs ecosystem.

For RAM, RED should be using ECC RAM on the camera, and users should be using ECC RAM on their computers. Mac Pro's have ECC RAM today.

Beyond these changes, RED should allow people to record the same images to two (or more) CF drives at once. This would guard against major card and/or drive failure. Each CF drive should be 100% redundant and share no internal components.

RED, if you're interested, I'd be happy to stop by and walk your people through how to do this. Although I don't advertise it here, I've been a computer scientist for many years and consult with major, well known companies all over the world on issues like this -- something a quick Google search would reveal. Hot rodding existing systems is my specialty; if computers were cars, I would be the guy helping teams design F1 racers.

 

[Don Harris]

I had one of my 32GB SanDisk CFs go bad during a time-lapse shoot a couple months ago, for example. The Nikon D3 didn't notice or care, it just kept snapping pictures..

Well, since Nikon provides two card slots so you can record your shots to both cards, if you didn't use it, it's hardly the fault of the camera now is it?

 

[Zakaree Sandberg]

its probably a good Idea to zero out data on the cards before each shoot..

 

[Gunleik Groven]

Where or how does Rocketcine-X report bad frames?
Do you have to transcode to get this report?
Does Redcine-X do the same?

Wouldn't know about Redcine-X
We're not friends, yet.

As to Rocketcine-X, you'll have to transcode - which is realtimeish depending on format and hardware, but can happen at the same time as you're backing up the next media (given that you have sufficvient disks and ports to handle that).

Rocketcine-x reports bad frames in the render report (bad frame at XXXXXXX)

Works really sweet.

This can be due to... bad frames... or just a render-error, so usually I just re-throw that file on Rocketcine_x, then have a look at it from the original media, then re-copy.

Absolutely no bad frames so far with this attitude. But you never know when the ugly green hits you in the neck... -