Spotlight corrupting frames in R3D files?

[Chris Santucci]

I had an issue a while back with bad frames in R3D files that I eventually determined was caused by Spotlight running while transferring those files from the original camera media.

Can anyone verify this phenomena?

I just got a call from a post house that's dealing with 3 days of my R3D files and they tell me they're getting errors during transcode even though we got checksums on 100% of what we gave them.

 

[Victor Lazaro]

If you had a checksum of 100% then there's a good chance that its an in camera write error.

 

[Tim Bradley]

I had an issue a while back with bad frames in R3D files that I eventually determined was caused by Spotlight running while transferring those files from the original camera media.

Can anyone verify this phenomena?

I just got a call from a post house that's dealing with 3 days of my R3D files and they tell me they're getting errors during transcode even though we got checksums on 100% of what we gave them.

Hi Chris have you got a frame (picture ) example of what the corrupt frame looks like ?

I had an issue months back where there were random corrupt frames during a shoot and I have never been able to find the cause.

Cheers, Tim

 

[Chris Santucci]

If you had a checksum of 100% then there's a good chance that its an in camera write error.

How can there be any errors if the files show as being 100%? I don't get that.

Hi Chris have you got a frame (picture ) example of what the corrupt frame looks like ?

No, I don't. The post house is telling me they got errors while transcoding. That's all I know.

From what I've been able to research, CF card readers can get hot and cause corruption and Spotlight can cause corruption also.

 

[Johnny Friday]

Did you verify the actual images after your shoot and/or transfer? Personally my method is i don't trust anyone--including a post house to transfer my images or get a disk that i did transfer to....without ME first verifying images are there...This is your work; your gold your responsibiity...I in fact find so many post houses and production companies often (more than not) know nothing or very little about the RED workflow and they wont' put much time into figuring out the problem if one occurs--well, nor should they.

Anyway, I preach to anyone working with me that at end of shoot or day or when we get a chance...we make transfers, open in RCX and just verify the images actually exist and do a fast scroll through and often pick 2 or 3 images and play them out of several hundred. EVEN when doing a checksum that gives you a green light---you can transfer garbage in just as easy....This goes for anything and everything i do---WHY, because at the end of the day YOU or ME are the guy the screwed it up regardless of who really screwed it up....I always base this on my images, my responsibility all the way to post (if i can that is), but make sure images exist when the leave my hands AND i keep a backup of ALL images i send out for stock sales, or productions---THEN, when i get word they have colored and finished the work, i recycle those backup hard drives--You will be a hero when they call you crying about losing some images or a drive crash and you have a backup--I beg to say this is part of your job as cinematographer, camerman director etc....

....hoping you did do this and if not, you will....best,

 

[Chris Santucci]

I'm with ya, John, 100%. And yes, the DIT always checks the footage files and sees image for sure. The clients were watching the files, the director was watching files, the DIT was watching the files. And yet, this particular post house has issues. They ARE re-transcoding from the drives we gave them and I haven't heard about any issues since this was started.

 

[Tom Wills]

I was the DIT on this job, and yes, every file came through R3D Data Manager OK (or was re-transferred, and then cleared the checksum OK), and I always opened RedCine X Pro whenever I had a moment and checked to make sure I was getting thumbnails and that I could scrub through every clip. Most then got played by the DP and Director in full also.

I've done several jobs with this RED, and with these cards, and using this same system, and I've only had issues on this job with this post house. My methods haven't changed, so I think it must be something on their side.

 

[Dustin Cross]

I would have the post facility run MD5 on the files and verify the checksums of the problem files they have match the checksums you got on set. The might have caused a problem copying the footage at post facility.



Dusty

 

[Steve Sherrick]

Chris/Tom, there are a lot of possibilities as to what may have happened between when the files left your system and when they were ingested at the post house. I'll throw out a couple. Assuming the files you sent them are 100% fine and have been checked thoroughly:

- It's possible the post house copied the files onto one of their storage systems and perhaps some files got corrupted during this process.
- Drive might have been damaged during shipping (if it was shipped)
- Their setup has configuration problems
- If they are using your drive to transcode from, maybe there is a bad cable in the chain.

Wouldn't be the first time I have heard of these types of things happening. One job I did called to say the drive was DOA. When it left my hands to the producers it was working 100% and had been thoroughly checked. Fortunately there was a backup drive that was then shipped and was fine but they called me to ask if I had a backup of the backup just in case something happened to that one, and indeed I had them covered.

Data can be fragile and that's why having multiple backups is critical. Ensuring that each of those backups is 100% saves a lot of time and money.

 

[Chris Santucci]

It's the post houses job to do this stuff if they want to verify our checksums. We're doing everything we should be doing to insure the integrity of the footage files. I really, really, think it's Spotlight that's causing the issues for them. I'm about to shoot a 4 day Red job with the same gear and it's being posted by the producer so we'll see if HE has any issues because the 3 day Red job I did 5 weeks ago resulted in zero negative feedback about our footage files.

 

[Chris Santucci]

Chris/Tom, there are a lot of possibilities as to what may have happened between when the files left your system and when they were ingested at the post house. I'll throw out a couple. Assuming the files you sent them are 100% fine and have been checked thoroughly:

- It's possible the post house copied the files onto one of their storage systems and perhaps some files got corrupted during this process.
- Drive might have been damaged during shipping (if it was shipped)
- Their setup has configuration problems
- If they are using your drive to transcode from, maybe there is a bad cable in the chain.

Steve, we back up to 2 drives and one is hand delivered to the post house. They copy our files to their drive and I suspect this is where the corruption is happening. They re initiated their transcode today from our drives and so far I have not heard anything from them.

 

[Mike Lary]

How can there be any errors if the files show as being 100%? I don't get that.

A checksum only verifies that your copy matches the original. It won't trigger an error for bad frames.

 

[Johnny Friday]

Chris,
I doubt that it's spotlight...as Steve pointed out, there are a trail of possibilities...bottom line if you have verification visually on the drive you sent them, then likely anything can have happened. Might have been the hockey puck used at fed-ex that day or any number of possibilities...but as Steve and i point out....having a backup of the data (even if it's not your job), is just critical to these kinds of tasks and will always be a big reason why "they" will use you again. I have about 40 drives at my office just with backup data from jobs over six months old...only once in a great while will i get a producer calling and praying i have a copy of the files. Shit happens with these hard drives and computer data....and all we can do is be prepared....all you can do is check the drive if they are not able to trouble shoot it with new cable, power supply etc....run red-undead et. al.

....AND as continually pointed out a checksum or running R3d data manager et. al. means NOTHING unless you visually verify data. again you can copy over ORIGINAL corrupt data files and checksum will verify you did JUST that....it ain't gonna tell ya if your files are corrupt or a bad frame etc....

 

[Chris Santucci]

A checksum only verifies that your copy matches the original. It won't trigger an error for bad frames.

Gotcha. Is there a way to verify the integrity of the R3D files other than by playing them?

 

[Johnny Friday]

Gotcha. Is there a way to verify the integrity of the R3D files other than by playing them?

NOPE--visual only

 

[Chris Santucci]

Chris,
I doubt that it's spotlight...as Steve pointed out, there are a trail of possibilities...bottom line if you have verification visually on the drive you sent them, then likely anything can have happened.

Why do you doubt that it's Spotlight? The first time I had this issue myself back in February, I was transferring from cards to external drives and ended up having bad frames galore and then I found out that Spotlight does in fact corrupts R3D files. Since then, I've made sure to disable Spotlight and have not had the bad frames and corruption issues - until the files are out of my hands and then who knows what they're doing or how they're doing it.

 

[Johnny Friday]

Why do you doubt that it's Spotlight? The first time I had this issue myself back in February, I was transferring from cards to external drives and ended up having bad frames galore and then I found out that Spotlight does in fact corrupts R3D files. Since then, I've made sure to disable Spotlight and have not had the bad frames and corruption issues - until the files are out of my hands and then who knows what they're doing or how they're doing it.

Well, then it could something related to your machine...I have spotlite running on 3 of my machines...one edit suite full time and two field units and NEVER have i had a problem with spotlight causing corrupt frames on any of my machines. Again, there are a million possibliities for your issues, but isolating it to spotlight? there surely would be more people with this issue...I've transferred and transcoded more material than i can think of w/out such issue. again, i think you may look else-where, but this is just friendly advise.. My wife being a doctor has to remind me countless times...NEVER diagnose the problem until you can verify it with absolute data. Computers are monsters for giving us false data....when comes to diagnosing a problem....First thing i would do is know that every time i do a transcode i get errors 100% of time -- if so, then there is a problem. Now go to another computer with spotlight running--do you get errors? My guess would be no. Then work back form there.

....another way to say this, is if Spotlight does indeed cause you errors to your transcodes, why does this not occur to countless others? Might be some configuration of your machine with spotlight running that is an issue....

 

[Björn Benckert]

This is normal. Discs are varying in quality, so is cables etc. etc... The post house might be doing things wrong, more likely they are given a disc with an error....
It's easy to check ask them what clip they got problems with see if it's transcodes well from the other backup/master drive. if so Ftp them that clip. If its a lot of clips that has problem go over there and swap drives... I run a post house, we go trough a lot of discs, thinking of them as fail safe is crazy. So simply use the backup drive, that what it is made for. Hopefully the error is not on that drive aswell.

 

[Chris Santucci]

Well, then it could something related to your machine...I have spotlite running on 3 of my machines...one edit suite full time and two field units and NEVER have i had a problem with spotlight causing corrupt frames on any of my machines. Again, there are a million possibliities for your issues, but isolating it to spotlight?

I have to land on something and really, other than a CF card reader getting hot (not this time I'd say, as we rotated through 2 readers), I'm at a loss as to what it could be. I didn't make up the Spotlight dealio and who knows, maybe OS has something to do with it also.

My fallback defense this time is the fact that we played the files and from now on we'll definitely be sure and scrub through each and every file on the drives we copy to.

 

[Chris Santucci]

This is normal. Discs are varying in quality, so is cables etc. etc... The post house might be doing things wrong, more likely they are given a disc with an error....
It's easy to check ask them what clip they got problems with see if it's transcodes well from the other backup/master drive. if so Ftp them that clip. If its a lot of clips that has problem go over there and swap drives... I run a post house, we go trough a lot of discs, thinking of them as fail safe is crazy. So simply use the backup drive, that what it is made for. Hopefully the error is not on that drive aswell.

More drives it is! Thanks!