13 threads on CORRUPT frames in less than 2 MONTHS

[Erich Ocean]

its probably a good Idea to zero out data on the cards before each shoot..

About the only thing that will do is wear out your CF cards faster...resulting in more errors, earlier. Not recommended.

The cards are reformatted so that new files can be written in continuous blocks on the CF card (this is more of an issue for a hard disk).

 

[Scott Crawley]

Not completely, because the Red Rocket is most of the time able to transcode clips with corrupt frames. So you would need to backup, transcode than do a real-time visual check. If you spot some corrupt frames (and you'd have to be concentrated to do so) you'd need to start the whole thing all over again. And that realy isn't a solid and realistic on set offloading workflow. Even with real time red rocket transcode (that still needs to be stored and back uped) you would end up with about three times the length of each clip for offloading - transcoding - verifying
not possible on most sets in my opinion

edit: just reread your post and saw that this was exactly what you wrote. So yea

To be absolutely certain, I guess I would copy 1, then checksum, then do the real-time check... Before the transcode and second B/U. If the crew is pumping out six hours of recorded material in a day, I would definitely want a full day worth of media.

 

[Erich Ocean]

With RED One cameras, someone could use the SDK to write an automated "visual inspection" utility. Such a utility would:

1. work at the wavelet level (without debayer or anything else)
2. measure the change in overall levels in each channel independently on a per-frame basis, at each wavelet level
3. display in the UI each frame segment where the change in levels was above some threshhold, for the user to inspect
4. work in a "faster than realtime" manner

This would catch virtually all of the corrupt frames I've seen on the forum that aren't currently being flagged by RED's utilities, and do it automatically. Sure, there would be a few false positives, but the DIT can examine those manually. If we find new frames it didn't catch, we could add new strategies.

This would be way less labor intensive (and error prone) than manually analyzing each clip!

 

[Gunleik Groven]

With RED One cameras, someone could use the SDK to write an automated "visual inspection" utility. Such a utility would:

1. work at the wavelet level (without debayer or anything else)
2. measure the change in overall levels in each channel independently on a per-frame basis, at each wavelet level
3. display in the UI each frame segment where the change in levels was above some threshhold, for the user to inspect
4. work in a "faster than realtime" manner

This would catch virtually all of the corrupt frames I've seen on the forum that aren't currently being flagged by RED's utilities, and do it automatically. Sure, there would be a few false positives, but the DIT can examine those manually. If we find new frames it didn't catch, we could add new strategies.

This would be way less labor intensive (and error prone) than manually analyzing each clip!

RED, have a chat with this guy...
(Or r3d manager Conrad could probably have a good talk with him, too...)

 

[Kim Frank]

TO SUMM IT UP FOR THE MOMENT:
By now we have collected some ideas and ways to solve or get ahead of the corrupt frame Problem.

1. RED printing in the manual, that a visual verification before formatting original media is a neccecarity, that green or destorted frames are corrupt frames caused by an offloading error that can cause application crashes or transcoding issues and that its recomended to change out the offloading Hardware. (as in other words also stated by JEFF KILGROE)

2. To have enough Media in the field to shoot a whole day without having to reformat media, or at least enough that a visual veryfication can be made. (as also stated by GREG MILNECK)

3. RED could implement a "bad frame report", like in the render report in RocketcineX (as stated by GUNLEIK GROVEN) that "scans" the Footage without the need to transcode.

4. Someone could write an application that analyses the offloaded data automaticly. One could use the SDK to write an automated "visual inspection" utility. (As also stated by ERICH OCEAN)

5. RED adopting ZFS on the camera (http://en.wikipedia.org/wiki/ZFS#Cop...actional_model)- (as stated by ERICH OCEAN)

Hopefully RED likes some, or at least one, or has another one ...

 

[Jay Gannon]

Or just hire a DIT who knows to do all of the above and more for one relatively low fee? =)

 

[Kim Frank]

Or just hire a DIT who knows to do all of the above and more for one relatively low fee? =)

Thanks Jay...
I got your point the first time you posted it and of cause it's valid, but as the discussion went on, it has been said that humans can fail and that not every job does have the time or money to do as you say. And by now we are thinking about a way to eliminate this problem in a more time and cost effective way, if that's possible.

 

[Gunleik Groven]

TO SUMM IT UP FOR THE MOMENT:


3. RED could implement a "bad frame report", like in the render report in RocketcineX (as stated by GUNLEIK GROVEN) that "scans" the Footage without the need to transcode.

To be clear: The report comes AFTER transcode...

 

[Kim Frank]

To be clear: The report comes AFTER transcode...

Yes, I did understand that. It was just an idea, that it would be helpful to get something like this Report without the Need to transcode, as for me I often don't need to transcode for my post Production Workflow and it just would eat up disk space if I needed to just to get this report.
But thanks for clarifying...

 

[Jay Gannon]

Thanks Jay...
I got your point the first time you posted it and of cause it's valid, but as the discussion went on, it has been said that humans can fail and that not every job does have the time or money to do as you say. And by now we are thinking about a way to eliminate this problem in a more time and cost effective way, if that's possible.

I'm sorry but I don't think anyone should be shooting without a DIT, its dangerous and gives RED a bad name, and its something I am very passionate about. So I continue to make my point, if your shooting commerically and your not a dit, then hire one.

 

[Gunleik Groven]

I'm sorry but I don't think anyone should be shooting without a DIT, its dangerous and gives RED a bad name, and its something I am very passionate about. So I continue to make my point, if your shooting commerically and your not a dit, then hire one.

Problem is that that is not allways an option AND the DIT is the first thing many production companies want to get rid of.

I am NOT saying that is wise, or that I don't agree with the value of a DIT. But you know as well as I that the marketing for RED is "digital awsomness" and not so much "hey, remember to bring an extra man on set, and he better know his shit and be good and in demand - thus have to get paid", sorta.

 

[Conrad Hunziker]

With RED One cameras, someone could use the SDK to write an automated "visual inspection" utility. Such a utility would:

1. work at the wavelet level (without debayer or anything else)
2. measure the change in overall levels in each channel independently on a per-frame basis, at each wavelet level
3. display in the UI each frame segment where the change in levels was above some threshhold, for the user to inspect
4. work in a "faster than realtime" manner

This would catch virtually all of the corrupt frames I've seen on the forum that aren't currently being flagged by RED's utilities, and do it automatically. Sure, there would be a few false positives, but the DIT can examine those manually. If we find new frames it didn't catch, we could add new strategies.

This would be way less labor intensive (and error prone) than manually analyzing each clip!

I had already implemented a method similar to this. There are several issues with this approach:

- You have to work after the debayer to get accurate color information. Thats a must.

- Its impossible for software to detect the difference between a flash frame, a whip pan, and a corrupt frame. You end up with percentages of probability, and a human has to decide what those parameters are. Because a human has to decide, it is not foolproof and error prone. You are dealing with a set of numbers (percentages), which to a large percentage of the population is meaningless or difficult to grasp.

- You need to work in 2k or higher to accurately get information about a set of frames.

- None of the above is real-time capable. To get around some of the problems above you end up holding dozens of frames in memory, and calculating the differences of such.

- People would use said software on system not up to the task, which would become painstakingly slow. People are still using MacBooks to offload.

- A human still has to view the suspect footage in real time to find any issues. If there are multiple takes with issues, you have to either guess that some of them are fine, or watch every one of them in real time.

So with all the above, and given the same human element, it is by far faster and more reliable to watch footage in real time than it is for a software utility to watch the footage for you. Thats the reality.

Of course, there are legal issues here as well. If software pronounces the footage ok, but there is an issue that is large enough to cause the footage to be unusable, but small enough that it fell within the error of probability - whos fault is it then?

Really, watching the footage in real time, with the current level of technology, is the most reliable answer. This position used to be called the "film loader". Productions choose not to hire them at their peril.

 

[Kim Frank]

Thanks for the insight Conrad...
What about my idea, that the camera itself could produce some kind of Color Meta data while recording, that after offloading can be used by an application to really compare the information. Would that still need a debayer? Aren't the corrupt frames also visible in the reference files? Wouldn't using them be a way to speed up the process? As long as epic will report about corrupt frames while recording, some way to compare the recorded with the offloaded data would really eleminate this problem.
Excuse me if that sounds foolish, just dreaming.

 

[Conrad Hunziker]

Thanks for the insight Conrad...
What about my idea, that the camera itself could produce some kind of Color Meta data while recording, that after offloading can be used by an application to really compare the information. Would that still need a debayer? Aren't the corrupt frames also visible in the reference files? Wouldn't using them be a way to speed up the process? As long as epic will report about corrupt frames while recording, some way to compare the recorded with the offloaded data would really eleminate this problem.
Excuse me if that sounds foolish, just dreaming.

Corrupt frames are visible in the Quicktime reference files, but are harder to spot with the smaller resolutions. For example, if your using the 1/8 quicktime files, and theres a 1/8 frame corruption, you are dealing with a really small portion of the frame to try and gather a differential comparison with other surrounding frames. Thats why for any software application to detect frame corruption, it needs to deal with larger data sets.

As for the metadata idea, as per Red's conversations, that wont happen with the R1. There just isnt enough processing power in that guy to do any of the items you are talking about, in real time.

The epic should solve all these issues with the interframe checksums, depending on the exact implementation.

As a last aside, I am firmly opposed to ZFS filesystems, just from a usability point of view. I realize the benefits, but the filesystem is not universally supported, and in some respects is still undergoing fundamental changes. FAT32 is by far not the best filesystem, but (sadly) it is the most universally supported. BTW, the FAT32 system on the R1 is not windows FAT32, but a variant made for speed. ZFS is not as fast in real world testing, and requires more overhead not available, to my understanding, in the R1.

 

[Erich Ocean]

I had already implemented a method similar to this. There are several issues with this approach:

- You have to work after the debayer to get accurate color information. Thats a must.

- Its impossible for software to detect the difference between a flash frame, a whip pan, and a corrupt frame. You end up with percentages of probability, and a human has to decide what those parameters are. Because a human has to decide, it is not foolproof and error prone. You are dealing with a set of numbers (percentages), which to a large percentage of the population is meaningless or difficult to grasp.

This isn't the right forum to discuss technical issues like this, but I'll just say that I disagreed with both of these observations.

I've been building a 3D renderer that renders directly into a wavelet basis and uses that basis for all sorts of analysis during the render. I'm very comfortable with wavelets, how they work, and what you can get out of them without converting them to RGB (!) first.

Second, software is amazingly good at detecting subtle differences between things using techniques more sophisticated than a "percentage". I've got some really nice PCA software that is CUDA accelerated and can do some truly jaw-dropping analysis. I would also recommend using Bayesian inference to classify frames (teach it what both good and bad frames look like, including the "hard cases" you mentioned above). It's then a matter of ranking footage and the DIT can look at potential misses to as much certainty as they want.

With that out of the way, you're correct that processing RED 4K footage faster than realtime requires appropriate hardware. Basically, a fast enough drive array, enough RAM, and a modern CUDA card like a GTX 480. Since we're talking about helping a DIT, and since we're talking about *footage*, arguably the most valuable thing on the set, I don't see the cost as being unreasonable.

Don't forget the goal: to catch more problems than we currently do, in a large number of situations, with no changes to RED One data collection. For absolute data integrity, I've offered other mechanisms -- and even then, I don't think we'll convince a conscientious DIT not to watch the footage all the way through.

 

[Quentin Brown]

:lurk5:

Really interesting and informative thread here.

I have been wishing there was a faster and more reliable way to check for corrupt or dropped frames for a while. Conrad and Erich have obviously given this a lot of thought and have the knowledge to actually understand how such a thing might be implemented.

One thing though - the high end hardware requirements needed to get this to work might just preclude it's use in those situations where it's most needed.
It's often the low budget productions where data management has been left at the bottom of the considerations that you'd come to rely more heavily on a tool like this but then usually those same budget considerations won't stretch to extra media and paying for higher end DIT kit.

For example last week I ended up working as a digital loader on a music video under a DIT friend who was initially hired to do all the duties of 1st AC, 2nd AC and DIT! He said no way, and insisted they hired someone else too. I ended up being 2nd AC and Data wrangler - no budget for my kit, just working off my macbook pro. No time to visually check footage, just had time to set copying through R3D data manager & run back to change lenses, move directors monitor and get slate up + spot the steadycam op - though I did get to check the first mag to come off the camera for some kind of confidence and spent some time the night before going through and testing out the kit (I would have been happier if I had had a chance to visually check a full offload of footage at least once from each RED media in use). These are the kinds of situations where I would most desperately want to have a software solution flag possible dropped/corrupt frames for me to check. It wouldn't have to do it in real time - just fast enough for me to see the results when I come back to the station before my next reload.

Of course ideally I would want to be working on projects with bigger budgets so they care more about data integrity and can afford to spend money on it - but this world is far from ideal. ;-)

 

[Kim Frank]

Really interesting and informative thread here.

Really glad you found it informative, that was my purpose with this thread.
But 75 replies and not one of anyone at RED feels really disapointing and makes a Statement for itself IMHO.
Why does anyone have to find out aboutthis issue the hard way, by loosing footage. Why is it so difficult to inform and warn RED users?
Give them Tips and workflows to avoid this.
No statement is a statement, right?

 

[Stephen Williams]

Really, watching the footage in real time, with the current level of technology, is the most reliable answer. This position used to be called the "film loader". Productions choose not to hire them at their peril.

Hardly accurate, a film loader or 2nd assistant is a junior position, who loads film into the magazine. Quality Control monitoring is another thing altogether.

 

[Roberto Lequeux]

Really glad you found it informative, that was my purpose with this thread.
But 75 replies and not one of anyone at RED feels really disapointing and makes a Statement for itself IMHO.
Why does anyone have to find out aboutthis issue the hard way, by loosing footage. Why is it so difficult to inform and warn RED users?
Give them Tips and workflows to avoid this.
No statement is a statement, right?

Honestly, I think you are waaay out there and need some perspective. I do not agree with you in the least bit.

This is a professional tool, don't blame the manufacturer because they didn't see it necessary to make a 3rd grade level manual spelling everything out.

Stuff messes up. Kodak, Fuji, Arri, Panasonic, Sony, JVC, Canon, Nikon, Mamiya, IMAX, RED, they will all mess up from time to time. The nuclear clock has back-ups!! And we are talking about 99.9% reliability here!

Sorry if I am making this sounds harsh, it is not directed at you but at the general attitude of demanding and complaining over things we have no right to whine about. It is not meant to sound harsh for the wrong reasons, only as an attempt to get this point across, clearly, 'cause it feels absolutely ridiculous to dump on RED for something that isn't their fault, or for not yet having broken a new barrier!

Especially, when their prices are so stupid low and RED One is a slam-dunk of an investment for the right person, when it has been adopted by some of the biggest filmmakers ever, when it is being used by the biggest budgets ever.

Especially when they keep adding things even though it is obviously already "good enough", and adding these new tweaks without charging a penny extra for it!!

OK, fine, maybe they didn't already become THE FIRST to add check-sum in-camera, with R1... but what if there isn't enough processing power, or memory, or little red elves with supercharged abacuses still free to handle the extra operations necessary for the check-sum? There could be several things we are completely unaware of that make this impossible on the R1. After all, this is a camera that has been out for a few years already and records over 400% what other cameras coming out NOW record!

And someone already mentioned in this very informative thread that RED announced they are going to be THE FIRST to add check-sum in-camera! On Epic.

So, if you were busting your ass non-stop for several years straight, demolishing all sorts of barriers and planning to offer it all yet again for a small fraction of what the competition charges for cameras that have less than half the capabilities... and people still demanded and complained, pointing a finger at you...

You are right, silence can be a statement.

Again, this is not directed at you, but at what I consider a general lack of perspective in the forum.

 

[Kim Frank]

I think you are making a general point, a good one, but that has not much to do with my request.

It's not about making a "3rd grade level manual" it's about keeping people informed and save them some serious trouble.
I think you can't expect a RED one user to read through the forums, just to find out that green frames, black R3D Files, crashing Applications may be caused by corrupt frames that in 99% happened while offloading from the original Media.
In the Manual they already state what kind of CF Reader they reccomend and there is an POSSIBLE ERRORS ISSUES AND SOLOUTIONS Section where this information would fit perfectly!

Put yourself in my situation for a second.
I didn't know what causes these issues and because the clips played back fine in Camera and I caud scrub through the offloaded footage, I thought it would be fine. If I just knew at this point I would have switched out CF Reader and/or Fire Wire 800 cables and offload it one more time. But I thought it would be some kind of Bug, that could be solved. But after the Rented Media was formatted and I had a lot of importand takes, that I couldn't use and needed to find out how I at least could get the Clips around the corrupt frames trancsoded, I came in a tough situation on the post Production Scedule that could have cost me 20.000 Euros. At the end I took another Compositor in the team and we made it just in time.
So transcoding with a RedRocket in a post fascility cost me 1000€ and the extra Compositor 2000€. I lost 3000€ by simply not knowing something I could have easily avoided if I just KNEW!

I learned it the hard way, why do others have to?!