Troubleshooting: Epic Fail

[Drew Suppa]

Hello all:

On set in Los Angeles and it's late... I don't expect a response anytime soon; I'm having my first issue of this kind.

I am using R3D Data Manager (v. 6.0) on a MBP OS 10.6.3 and dumping 16GB RED CF cards to G-RAID 500GB Shuttle and 2.0TB Archival drives through the Lexar FW800 card reader. We are running a RED One camera body with the M-X sensor. Never had issues with this type of setup before on older RED models nor upgraded ones.

One of our reels had significant data corruption; there were only 4 takes, but corruption is present in all four clips.

From the Recovered File Log:

R3D Data Manager Recovery Results
*********************
File: A057_C001_05208X_001.R3D
Shot Date: Dec 1, 0002 4:00:00 PM
Frame Size: (null) x (null)
Number of Frames: (null)
File did not copy successfully, possibly corrupt

*********************
File: A057_C002_0520GB_001.R3D
Shot Date: May 19, 2010 5:21:41 PM
Frame Size: 3840 x 2160
Number of Frames: 1310
File successfully recovered

*********************
File: A057_C002_0520GB_002.R3D
Shot Date:
Frame Size: x
Number of Frames:
File did not copy successfully, possibly corrupt

*********************
File: A057_C003_0520BA_001.R3D
Shot Date: Dec 1, 0002 4:00:00 PM
Frame Size: (null) x (null)
Number of Frames: (null)
File did not copy successfully, possibly corrupt

*********************
File: A057_C004_052019_001.R3D
Shot Date: May 19, 2010 5:32:24 PM
Frame Size: 3840 x 2160
Number of Frames: 1312
File successfully recovered

*********************
File: A057_C004_052019_002.R3D
Shot Date: May 19, 2010 5:32:24 PM
Frame Size: 3840 x 2160
Number of Frames: 1475
File successfully recovered

*********************
File: A053_C009_0519JS_002.R3D
Shot Date: May 19, 2010 1:01:05 PM
Frame Size: 3840 x 2160
Number of Frames: 432
File successfully recovered

*********************
File: A053_C010_0519BB_001.R3D
Shot Date: May 19, 2010 1:04:04 PM
Frame Size: 3840 x 2160
Number of Frames: 3
File successfully recovered

*********************
File: A042_C009_0518MV_002.R3D
Shot Date: Error (dup)
Frame Size: 0 x 0
Number of Frames: 0
File successfully recovered






Note the date malfunction in several clips, and the missing fields all together. I was able to render a ProRes file from Clip 1, but it displays a black screen and has multiple audio squelches of varying tone and intensity. Any other clip I try to view, either in Quicktime rapper or from the .R3D file, in REDAlert! or REDCine-X causes the G-RAID drives to become noisy under seek and causes the applications and my system to crash.

I have alerted production that the card is currently corrupt and it has been pulled from rotation, hopefully to be recovered by someone with better tools and knowledge than I.

Thoughts?

 

[Drew Suppa]

UPDATE:

Sorry to sound like a prick; it's very late and I'm frustrated.

The first time I tried the transfer, R3D Data Manager crashed.

The second time I tried the transfer, it gave me the error message and the clips would not play.

The third time, I let the computer, drives, reader and CF card cool down, while swapping my cables. After completing this third transfer, R3D Data Manager tells me that all files have been successfully copied. However, opening in REDAlert! or REDCine-X yields the same audio squelching, frozen frames, green frames, etc. as the first transfer, and I am unable to make usable ProRes files or even play back in any form.

Question to the R3D Data Manager team... how is a technician/loader/anyone supposed to be confident in your product if it tells them that a successful transfer has occurred, yet the footage is f**ked up, obviously by way of corrupt data from the card end? Please pardon my French, but I'm a little irked that this application has confirmed a "successful" transfer.

I would attach the annotation from the .txt file below or in a separate post, but I'm still 1500 characters too long. I have sent a crash report to R3D Data Manager team.

Input is needed from anyone with similar problems / R3D Data Manager team.

 

[Kwan Khan]

Possibly corrupted.
Can you try Cine-X (in different System) and check your RGB values?

 

[Drew Suppa]

I've been under the assumption that the files are corrupted, but after further research, they play back without error in-camera.

I'll be able to check on a different system tomorrow morning when we wrap. I only have one machine on-location, and this one crashes every time I play from Cine-X or even try to import into Alert!. Even the proxies crash Quicktime (freeze / application quit). The one clip I was able to import into REDAlert! and generate a ProRes 4:2:2 file played back in Quicktime, but ended up being a black frame with multiple audio glitches.

 

[Drew Suppa]

UPDATE:

Now 5:12AM Pacific Time; after playback in-camera, we decided to record another clip to the end of the card, overwriting any other previous clips that may not have been wiped during formatting before use.

This took care of the issue; R3D Data Manager imported the clips successfully and both the .R3D files and proxies play back in full.

Still, I'd like to open this dialogue to the guys at R3D Data Manager, because I do see a potential flaw in reporting a successful transfer when the data is corrupt. Also, I hope others can benefit from what I learned over the past 5 hours.

 

[Brian Ferguson]

Hi Drew,

I came across this thread with your posts, and I was wondering if it might have had more notice if it was in the "Tools" portion of the Workflow forums. Since R3D manager isn't just for Apple. Also hard to tell what the topic is about from the heading. Hope everything works out for you!

Brian Ferguson aka briferg

 

[Michael McLaughlin]

Check the pins

Check the pins

check the pins on your RED CF unit. Use a flashlight.

They may be straightend now because of a successfull insert.

Just because one pin (or more, don't know limit of terminal failure) is
not totally in contact at all times, probably means that the card will
still function. BUT this is of course, is at the cost of data corruption.

If you can't tell for sure, unscrew the side of the cover on the
RED (for the CF reader), and look at pins directly from the side.

Have had this happen, it sucks. And didn't happen until I got a new
RED 16GB card (had 8GB for over a year, never a problem like this).

One of my RED 16GB (on closer inspection after the misshap) had a
slight bulge, and plastic bur like thing. The other 16GB is fine.

I have to contact RED this week and get to the bottom of this, but I
bet there are probably more odd shaped (missmade) cards than just mine.

Again it is hard to tell unless you carefully inspect (like lay them on a perfect flat surface etc.).


I guess I would some it up to the end result is like having mis-seated RAM in a computer. Weird issues, random errors, hard to pin down whats wrong etc..
This seems to be the same for the CF in a situation like this.

 

[Peter Majtan]

But Michael, he did mention that the files played back fine in the camera. The corruption has obviously happened during the transfer from the CF to computer. Now I am not saying that the CFs are not the culprit - it might very well be the issue, but not in the camera module and rather in the CF card reader used to transfer the data to the computer. Just my 2 cents...

Peter

 

[Michael McLaughlin]

I know Peter...

I know Peter...

I read all his comments but, I was more locking onto this.

Now 5:12AM Pacific Time; after playback in-camera, we decided to record another clip to the end of the card, overwriting any other previous clips that may not have been wiped during formatting before use.

This took care of the issue; R3D Data Manager imported the clips successfully and both the .R3D files and proxies play back in full.

He had to at some point to do all the above taken the card out then
brought it back (maybe re-seeding the card correctly) and then formatted.
It is still hard to totally discern (or maybe I am just dense) what exactly he did. I guess the key is formatting for me, and re-seeding a problematic card.

It's a tricky thing.

Things "might" seem to be playing back in camera fine but:
maybe he didn't play "all the way thru" etc. etc.

Tough to judge with-out a super thorough indepth step by step.

I usually DIT, and I've worked with a couple of DPs and ACs that swear they did this that or the other thing; only to find out they didn't do this or that or
the other thing all the way through, or didn't have this cable at all screwed tight, or they really didn't jam-sync correctly (on and on and on).

Granted do that enough and you find yourself not working much as a AC or DP. But it still happens.

I come from old school, when funky tech voodoo (like SCSI in the old days) happen, check cables etc. first (at source), then work upwards.

It was more a safety thing, cause same thing happened to us, one minute the file played back fine in camera, next time not (I was not on the set at this
first moment, but exprienced when I checked).

AT first the AC brought me the 16GB card and I inserted into my flash copier, copied to laptop, found some files totally garbled. I then told the AC I wanted to come back with him and check the camera (and noticed the bent pins). And on carefull examination noticed the slightly mishapen card. My card
reader also has problems with the 16GB sometimes depending what computer its hooked (just to add more fun to the confusion).

More or less; just trying to help the guy cut out all externalities (cause it seemed maybe there was more to it).

BUT that could be me just projecting my own experiences onto his situation, which may or may not help.

But I doubt that they my suggestions would hurt. Even if cards are seating
fine (with no problems), a lot of times you can still see kinks in the pins where
they had been bent once before (and since been restraightened mostly by correct position re-seating).



So a quick flashlight check of the CF slot takes how many secs?

 

[Drew Suppa]

check the pins on your RED CF unit. Use a flashlight.

They may be straightend now because of a successfull insert.

Just because one pin (or more, don't know limit of terminal failure) is
not totally in contact at all times, probably means that the card will
still function. BUT this is of course, is at the cost of data corruption.

Thanks for your valued comments; upon checking the camera earlier on, I didn't see any noticeable bends in the pins, and we recorded to two other 16GB cards without issue immediately after the issue with Reel 57 (the problematic card).

That said, I did not check the pins on my primary CF card reader, but even so, I had a back-up on set which also gave issues.

 

[Drew Suppa]

Hi Drew,

I came across this thread with your posts, and I was wondering if it might have had more notice if it was in the "Tools" portion of the Workflow forums. Since R3D manager isn't just for Apple. Also hard to tell what the topic is about from the heading. Hope everything works out for you!

Brian Ferguson aka briferg

It was hard to come up with an appropriate title at 3:00AM after trying data recovery already for nearly two hours. My apologies; it seemed "Epic Fail" was appropriate.

If this should be moved, please do so; I know specifically I am using an Apple machine, so found it appropriate to post herein.

 

[Brian Ferguson]

It was hard to come up with an appropriate title at 3:00AM after trying data recovery already for nearly two hours. My apologies; it seemed "Epic Fail" was appropriate.

If this should be moved, please do so; I know specifically I am using an Apple machine, so found it appropriate to post herein.

No worries I am not a Moderator. When I saw the title, I saw the word Epic and was like "what the heck?". When I read it I was imagining your circumstances ( I have been in similar) and was thinking where you could have placed it to get the quickest answer. here it seemed buried.That was the spirit of my post which probably didn't come across. I didn't want to come off as a forum cop, just an observer watching you post without getting a response.

 

[Jay Gannon]

Drew this sounds more like a hard drive about to die in my opinion.
If at all possible rotate your raid out of your workflow and do a full check on each of the drives.

 

[AaronPicot]

and get a back-up computer! Yikes, what if something really bad were to happen to your computer by accident? Would production have to halt?

 

[Drew Suppa]

Drew this sounds more like a hard drive about to die in my opinion.
If at all possible rotate your raid out of your workflow and do a full check on each of the drives.

The G-RAID drives on set are brand new; Purchased 3 weeks ago. My MBP has an aftermarket Western Digital Scorpio Black 1-platter 320GB drive.

I have checked and done RP to all three drives and have found no errors.

and get a back-up computer! Yikes, what if something really bad were to happen to your computer by accident? Would production have to halt?

Unfortunately, this is a micro-budget feature with a very low kit rental; my backup machine lives at home unless absolutely necessary.

 

[Nir Shelter]

Agreed about the back up machine.
Always have at least 1 backup machine and twice as many HDD as will be used on set regardless.
Remember no matter how low the budget you’re only as good as your last job...

As for the error sounds like a physical corruption on the CF itself rather than readers. Best to get this card out of your rotation as soon as possible.

 

[Drew Suppa]

We've swapped 2 cards already, and now I'm having errors with another 2; this brings my total to 4 failed 16GB CF cards in 2 weeks!

I'm trying to think of where other weak links can be, but at this point, it's the media or the camera body itself. I'm quite frustrated by this.

 

[Jay Gannon]

Very very unlikely that its the cards then.
Its either your station/hard drives or the camera.
Personally I'm still betting the RAID as its brand new. Bad HD batch.
Or a problem wit the CF unit on the cam.

 

[Mark Toia]

We had the same exact problem.
It was the CF card. The funny thing was the same exact week. My 5d did the same thing.
I could view and watch the pictures in the cameras but could not extract the data with massive corruption problems.

Both cards for the red and the canon got binned and we have never seen the issue again.

But you have had 4 more cards go down. Maybe the card writer in the camera. ?

 

[Drew Suppa]

Last potential weak link in the system was taken care of... downloaded the latest R3D Data Manager. Ruled out all issues from the machine side... now it's either the CF cards or the CF Module on the RED body itself.

This issue persists.