Huge Data Loss - Confused as to what happened

[Tyler Dreher]

So I'm editing a friend's wedding that I shot. I transferred the footage when I got home and everything looked okay (here's where I made the mistake). I usually check through CineX to make sure everything is there but of course this is the time I didn't, I just went off of the folders. Well I've got 20+ empty folders and not sure what to do. If you look at the times for when the files were modified, it seems like the folders were not empty when they were transferred because the time stamp has enough time in between for the footage to transfer. Thankfully this was the project to learn this lesson on, but I'm confused as to what happened. Any help would be appreciated.

 

[Phil Holland]

How did you transfer the footage?

 

[Tyler Dreher]

Dragged the RDM onto a drive

 

[Tyler Dreher]

How did you transfer the footage?

In my 3 years of working with RED I've never had an issue doing it this way, but clearly I did something wrong here. Is there transfer software you'd recommend?

 

[Steve DiMaggio]

I know sometimes for some reason my computer will only allow some many files or some much data to be drag and dropped, I have had the same exact issue with my weddings ( I usually shoot about 1.5 tb of mixed media and 50 weddings a year).

I learned to always double check file count and disc size against the media original.

steve

 

[Tyler Dreher]

I know sometimes for some reason my computer will only allow some many files or some much data to be drag and dropped, I have had the same exact issue with my weddings ( I usually shoot about 1.5 tb of mixed media and 50 weddings a year).

I learned to always double check file count and disc size against the media original.

steve

It picked right back up after that gap though and kept going.

 

[Tyler Dreher]

I learned to always double check file count and disc size against the media original.

steve

Which is very weird in this case. Judging by the fact the RDM folder is only 290gb, the footage is definitely gone, but why did the computer take so long to transfer empty RDC folders?

 

[Joel Voelker]

Attempt a file recovery on it, maybe the permissions are bad or the files are hidden. I assume you already wiped the mag?

Buy shotput. Best $100 you'll ever spend.

 

[Jan Balster]

Attempt a file recovery on it, maybe the permissions are bad or the files are hidden. I assume you already wiped the mag?

Buy shotput. Best $100 you'll ever spend.

I'd 2nd ShotPut Pro5, which has gotten much faster with the XXHash-64 checksum. One word of caution, though, ALWAYS do a visual check in REDcine. I once had ShotPut give me the "thumbs up" after running the checksum but the footage was corrupted due to a bad cable. Something that would have easily been spotted with a visual check in REDcine.

 

[Steve DiMaggio]

Which is very weird in this case. Judging by the fact the RDM folder is only 290gb, the footage is definitely gone, but why did the computer take so long to transfer empty RDC folders?

that is strange man

steve

 

[Nick Morrison]

I'd 2nd ShotPut Pro5, which has gotten much faster with the XXHash-64 checksum. One word of caution, though, ALWAYS do a visual check in REDcine. I once had ShotPut give me the "thumbs up" after running the checksum but the footage was corrupted due to a bad cable. Something that would have easily been spotted with a visual check in REDcine.

+1000 to everything Jan is saying.

Our DIT's always use ShotPut Pro, and always visually check everything in RCX before giving "OK" to a card to go back in circulation.

 

[Tyler Dreher]

Appreciate the help guys, lesson learned.

 

[Steve Sherrick]

Silverstack would also help here as it provides very quick visual feedback as to what is going on, including the ability to view your footage right there in the UI. On a professional job, this would have been disastrous. You always have to work on a "no frame left behind" mentality. Because all it takes is a corrupted clip on an irreplaceable take to jeopardize future work. The cost of proper data management utilities is nothing compared to reputation so fortunately this happened on this type of project.

 

[Tyler Dreher]

Silverstack would also help here as it provides very quick visual feedback as to what is going on, including the ability to view your footage right there in the UI. On a professional job, this would have been disastrous. You always have to work on a "no frame left behind" mentality. Because all it takes is a corrupted clip on an irreplaceable take to jeopardize future work. The cost of proper data management utilities is nothing compared to reputation so fortunately this happened on this type of project.

Exactly what I was thinking. I am extremely careful on pro jobs, but I guess saw this as "just another wedding" and got careless. Thankfully it is mostly during pictures which I end up cutting out anyways, the only thing missing is the first look which is a bummer, but nothing as big as losing the whole ceremony.

 

[Jack Shanahan]

Try red undead on the mag even if you have formatyed. I was able to recover a half day of shooting that was accidentally formatted once. Its cray there was still footage from the last job on the card too!

 

[Michael Tiemann]

I had a Red Station replaced under warranty. The Red Station had a flaky USB connection. The first symptom was that the Mini-Mag would randomly dismount. A later symptom was that it would not mount except when I was holding it. Slight cable pressure made a connection. Perhaps you issued the copy command, the OS requisitioned all the disk space (OSX won't start a copy it doesn't think it can finish), the drive dismounted during the operation (leaving the space allocated but not filled with data) and when you popped out the mag and started copying another, you didn't notice the warning message. Your system log (on your computer) might have recorded the dismount.

 

[Adrian Jebef]

Silverstack is by far the best data management and reporting software available, though it will cost you.

Visual checks are always helpful but an easy additional step after transfer is to do a total file size count on the Mag folder vs the copied folder. Also remember that unless you are transferring the Mag to two separate hard drives you will always eventually end up with problems.

 

[Josiah Duncan]

I always just check the disk size of both RMD folders to make sure they match in finder. It's often really easy to bump the Mag reader so that it stops transferring if you're not using it on a cart.

 

[Marc Wielage]

I know sometimes for some reason my computer will only allow some many files or some much data to be drag and dropped, I have had the same exact issue with my weddings ( I usually shoot about 1.5 tb of mixed media and 50 weddings a year).

I learned to always double check file count and disc size against the media original.

This is very sage advice. I have occasionally (very rarely) caught the OS getting a little overloaded and "forgetting" to copy over a few files... sometimes more than a few. Not just Red material, but also sound, text files, you name it. I've seen this happen when the sheer number of files is pretty massive, like 500,000-600,000+ files. My guess is that there's some kind of memory limit as to how many file names the OS can keep track of, but I'm just guessing.

This is a huge reason to use DoubleData or Shotput Pro or Silverstack even an inexpensive utility like Niwa Multicopy, anything that can generate a checksum during the process so that you can verify it's a bit-accurate copy. I don't deny that we all copy stuff casually and quickly with little things here and there, but for mission-critical stuff, you gotta dot the I's and cross the T's. It pays to be paranoid with camera files and master sound tracks.