Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Volume Recover on an 8GB Red Flash Card

I'll try and take a look at this issue when I can in the next week or so, but I don't have cards available to test with at the moment.

One point is that the Mac is messy. From a network management point of view it's a huge pain it the ass, it creates all sorts for metafiles wherever it goes. I have directories on my server full of Mac dot-files. With P2 it's often recommended to write-protect the cards before putting them in a Mac, as it immediately writes various folder information onto the cards as they are mounted I believe. I think the same will happen with the RED cards.

Only deleting the Profile files is a good approach, because it means that if the Mac does write more data on to the card it can only write over the sectors previously occupied by those files, rather than writing over sectors previously occupied by RAW data.

[blatant plug]
This is almost exactly the sort of situation we're creating the SolidStore to avoid... I know first hand how easy it can be to get a little out of whack when copying files over and over again all day. Everything starts to look the same.

I think human-error is by far the biggest risk to data on set. It's so easy to make a mistake, and once it's made it's really hard to fix it.
[/blatant plug]
 
A few things I've read (in the last few minutes) suggest this tool is very effective in many cases where others fail:
http://foremost.sourceforge.net/
It's a linux application.

I will try to build myself a test image to test with tomorrow. Can you confirm exactly the circumstances? You mounted card on Mac. Selected entire contents of card and dragged to trash? Then emptied?
 
Sycophant said:
With P2 it's often recommended to write-protect the cards before putting them in a Mac, as it immediately writes various folder information onto the cards as they are mounted ...
Click to expand...

If someone were to make a Read-ONLY compact flash reader they would sell a million of them! (Every Red user, every still photographer)
 
It would probably be possible to implement at a driver level in the operating system... You can certainly write-protect a device once it's mounted, but it may be too late by then.

Alternatively I suspect it's possible to make the default mount option Read Only in OS X. I might look into that.
 
In our case, we are doing a test with a formatted card. Some clients have been curious about this and we would like to offer them a solid answer.

In a scan with Virtual Lab (made by BinaryBiz.com), the .r3d files showed up and looked to be the correct sizes. But the way Virtual Lab works, they want a license fee + you purchase a quota of MB or GB for a certain $ amount.

For an 8GB card, were talking $175 just to try and recover with no guarantee that anything will be recovered. I emailed Virtual Lab, and they are not interested in doing a test or trying to work with us. I informed them that they could have another great addition to their business with RED business, since nothing else we have found seems to show the .r3d files at the correct size, but they were not interested.

For people recovering jpgs, their business model may work, but for 2GB .r3d files, I was not going to spend the $ on just a test. Even if it were $40-$50, maybe, but paying $175 with no guarantee plus having to pay each time per GB is not the way to go.

I suppose if it worked and was the only solution, it's a small price to pay and very much worth it.

So far, the trial version scans the files (after you put in the .r3d filetype and maximum size) and reports back what looks to be the correct files and sizes.

I think the real solution here if for RED to release a small utility that does this. I really think it would be a simple thing for them to crank out. Maybe after build 15 is released?

It would be a great way to offer even more insurance and a feeling of reliability. Eventually, there will be a large production where someone accidentally formats the card in Camera and then this will all of a sudden be an emergency. Why not put out a utility that does this now? Last NLE company I worked for released a utility that re-built the clip index and it was very much appreciated by customers.

Stuart, or perhaps Deanan, can you comment on this as a possibility? REDRECOVER?
 
I spent significant time trying to recover p2 MXF files.

This software was by far the best I tried.
http://www.runtime.org/

I wasn't able to get the p2 data back because of fragmentation in the file system. The software did recover the files, but they were garbled.

Read this FAQ on their site... you may get lucky, but fragmentation on very large files may create a problem that no one can solve.

http://www.runtime.org/recoverability.htm

clip:
* Fragmentation
A very common situation, caused by file deletion, format or partition deletion is a missing FAT entry. As long as the file size is smaller than the cluster size (e.g. 32 KB, depending on the drive size) you will get a perfectly recovered file because you actually do not need the FAT entry.

When the file is larger, it is usually allocated in consecutive clusters. This is why the most promising strategy for a data recovery software is to assume consecutive clusters when it rebuilds the file without FAT entry. This works for most of the files but runs into problems for files that increase over time. These files will necessarily be fragmented if they cannot be allocated consecutively because other files meanwhile use these clusters. Sadly, many important files fall into this category: Email files, databases, large documents and directories.

GetDataBack employs several techniques in order to recover even fragmented files correctly. These techniques include taking the allocation of other files into consideration. GetDataBack also is capable of reassembling fragmented directories. But make no mistake; these efforts are doomed to fail for large and heavily fragmented files.

As annoying as it is, although their content is still somewhere on the drive, these files are unrecoverable.

There is no automated data recovery software available that can solve fragmentation satisfactory. If you want to recombine a file consisting of 10 clusters on a 20 GB drive you must analyze, given a cluster size of 32 KB, all possible combinations of one known cluster with 9 other clusters out of possible 625000. There are 6250009 possible combinations, a number with 52 digits.

The only possible and more intelligent approach is a "manual" data recovery for a particular file. With Runtime's DiskExplorer you would begin at the known cluster and search downward looking for data you know belongs to the missing part of the file. Finally, you put all your findings together into a new file. The limitations of this approach are obvious. This can only be done for a couple of files with a known content.

Even data recovery service companies will most likely not produce better results. While they might have a couple of tools, e.g. for extracting readable text, they do not have your knowledge about the content of the file.
 
I wouldn't think fragmentation would be a huge problem on RED CF cards, we're talking about relatively few large sequentially written files...

Convince the NSA that the card had terrorist secrets on it, they'll get the data back! :)
 
I have seen this happen on set once already.
Card shot and oops reformated wrong card, but we have not rolled yet.
We didnt have a fix, stressful hour yelled at by the director.

I would PAY nice money for a program that would save me on set when this happens.
[programmers]
 
I've played a little bit with a card here, and I can't get any good results really.

I think it's a problem that is going to be largely unrecoverable, certainly not in an automated 'click to fix' kind of way.
 
Unless there's some way RED can prepare the file system to cope with this I don't think it's something they're going to have any more luck with. These are issue that are inherent with the filesystem, and are made more complex by the large file sizes involved.

The best bet is products like our SolidStore and the software tools I believe Offhollywood is working on, that will help manage that flow and reduce the likelihood of human error of this sort. Even then it's still going to be important that the card get handed off to the right person and put through the process correctly.

Little things may help. Perhaps different coloured CF holders for 'exposed' cards. All cards coming out of the camera are put in a red holder for example and not used again until the DIT puts them in a green holder to signify a processed card.
 
Yes, of course.

But if there is anything else that can be done in the form of a RED utility, it is well worth looking IMO.

I have tried tests with at least 12 different recovery programs, both on MAC and PC, but they don't really know what to look for and how the file structure is built.
I believe there are options out there to recover P2 data. Has anyone experience with this?

If RED could make a custom app, there might be a decent chance to recover .r3d files.
I wonder why we haven't heard from them on this. I can't find anywhere else on the forum where this issue has been addressed. :(
 
I realize that the .R3D file system is still evolving, and that that makes such a utility difficult to make, but perhaps a utility app could be released and updated along with the firmware.

This is a very important part of the equation and I hope we see this released soon.

Anyone else have anymore luck with other File Recover programs.

Perhaps there is something still out there... low level, that can restore .r3d.
 
This definitely shouldn't be difficult to solve as long as the designers of the file format are on board with helping to make the files easily recoverable. Dylan is right that fragmentation shouldn't be an issue if files are written consecutively and no gaps are created by deleting earlier files while later ones exist. Even the creation of new files after format by the Mac shouldn't be a problem if the media was formated on Mac to begin with, since it will only write files that already existed there before, in exactly the same location, which would not overwrite any data.

FAT32 is a fairly simple file system in that it doesn't store cluster chain information with information on the file. It's all stored in the File Allocation Table (FAT). When a file is deleted, the first character of the directory entry is set to ASCII 229 and the cluster chain in the FAT is zeroed out. This means that full file start information will remain as long as no new file has overwritten the directory slot or the same clusters on the disk. Since this directory entry tells you where where the file starts and how long it is, if the file is not fragmented, recovery is about as simple as it comes.

Formating completely zeros out the root directory and FAT, so file start information will be lost for files that are located in the root directory. I don't know how files are stored on the Red, but if they're all in the root directory, then no file system metadata will exist after a format. Since these data recovery programs are having no luck, I would guess that files are being stored in the root directory, which means that a signature search will be the only way to find the files.

I have done file recovery based on signature searches for a variety of files and I'm usually able to construct an algorithm that enables me to find the start and end of files. Unfortunately, guessing at the file structure wouldn't be sufficiently reliable for putting together a tool. This is where the designer of the file system comes in. Only the author of the file type knows for sure what set of rules can be used reliably to find the beginning and ending of files of that type, particularly since those rules are subject to change. If the designer of the file type can take care to ensure the rules remain valid, or are updated, when changes are made to the file structure, then the tool would remain a reliable tool. Also, if the process of building the set of rules that define the beginning and end of the file shows that there is no way to do it -- no reliable rules exist -- the designer of the file system can add a highly unique beginning and ending signature to the file, thereby completely resolving the issue. As long as those beginning and ending signatures remain as the first and last bytes in the file, it wouldn't matter what changes occur to the rest of the file.

So here's the question: Is this a sufficiently important issue that the file system designer should become involved in defining a set of rules for beginning and ending signatures, and possibly for creating highly reliable beginning and ending signatures, to maximize the ease of recovering mistakenly formated media?

How important is it?

I'll post a poll on it.
 
OK so, an update:

I've tested more with BinaryBiz's "Virtual Lab" software, and it seems to have potential. (www.binarybiz.com)

Here are the steps taken:

When you run a scan, you must select the option for "I don't know why my files / folders are missing".

Next, select "Start a new Virtual Lab recovery session". This connects you to their server where they create you an account which you will eventually use to pay for a MB or GB "quota" of data recovery "credits". I don't think this is right, where you have to pay by the data recovered plus the software license, but I think they do this to prevent piracy of their software.

Anyway, you then will be prompted to select your media (Lexar ATA flash card), and once you do that, choose the option for "Recover Initialized Drive" and then "Start New Scan".

Next, you will be asked, "Would you like to include a filetype Search?"
This is very important. Select YES and you will now be able to add the .R3D file type to the filetype list to select from. Select Add. When you type in the file extention,Make sure it is in ALL CAPS.

The way this works is that you must at least 5 known good .R3D files to create a "signature" for the filetype. So you will next click Add again and browse for 5 or more good .R3D files to add to the signature list. Once you have done this and have at least 5, click on Start, and Virtual Lab will analyse the file format to build the signature and then allow you to name it: .R3D, of course. (ALL CAPS)

Next, select the .R3D filetype from your list (it should show up in the unsorted folder) and select it's check box. Make sure no other filetypes are selected, unless you perhaps are trying to also recover the .mov quicktime proxy files. But I wouldn't worry about those right now. Also, change the maximum size of filetypes you are trying to recover to <8 GB, as the default is <10MB. Then Click on OK to Start Scanning.

Results should show within a few minutes (as tested on my MacBook Pro with Express Media 34 reader and full 8GB REDCard).

Here are my results from the scan. Virtual Lab wants about $110 for the software plus another $65 for about 8GB. I have not paid yet, since I am not sure it's going to work, and I am only doing it as a test.

Anyone want to go in on it with my and we'll set up a PayPal account or Fundable Account and pitch in for the test? I will be happy to do the work and post all the results.

Let me know and I'll set it up. Maybe we do $1 a person or $5 to get it off the ground quicker.

Here is a screen grab from my test results. As compared to what I know should be on the card, this looks accurate. Now, I just have to pay to recover and see if they are intact. (The files on this card are not from a commercial shoot, but just shot for testing purposes.)

What do you guys say? Shall we set up the test?

971_1203551337.png
 
Casey Green said:
Here are my results from the scan. Virtual Lab wants about $110 for the software plus another $65 for about 8GB. I have not paid yet, since I am not sure it's going to work, and I am only doing it as a test.

Anyone want to go in on it with my and we'll set up a PayPal account or Fundable Account and pitch in for the test? I will be happy to do the work and post all the results.

Let me know and I'll set it up. Maybe we do $1 a person or $5 to get it off the ground quicker.
Click to expand...

Casey, let's not get carried away.. Contact the vendors, point them at this thread, and let them know what you're trying to do and see if they'll let you do the testing free. It should be worth it to them if they can be seen to be a good option for fixing this problem for people.

For this test, do you have copies of the original 'lost' files? If so, it would be good to do an MD5 checksum of the resulting recovered files against the originals. Will let us know if they've made it through okay. If not, then a diff might be in order to see how damaged the recovered files are.

At that point is when it would be really handy to have an app from RED to split or fix broken R3D files, so you can at least recovered the no damaged footage.
 
Sycophant said:
Casey, let's not get carried away.. Contact the vendors, point them at this thread, and let them know what you're trying to do and see if they'll let you do the testing free. It should be worth it to them if they can be seen to be a good option for fixing this problem for people..
Click to expand...

Thanks for the thoughts, but I have already gone this route. It's unfortunate, but the company is not interested in doing any tests with .R3D files. They are focused on their main business of .jpg and .mp3 files that their average customers have. Sad, but they are not interested.

The great thing about their software is that it analyses file formats by asking for 5 or more similar files, and builds a signature. It found the files and looks promising.

I'm not sure what you mean by "let's not get carried away". This could be a very good chance for us to get a solution. RED has replied to me that they are interested in working on this but they won't start on it until the file type solidifies more, and that is still months away. Plus they have said they have a lot of other issues to work on that are higher priority, so I don't see a utility from them any time soon.

If you are not interested in participating, that is fine, but I am going to set up an account to collect contributions and do a test.

We need some hope here until RED addresses this issue.

Those in favor of contributing to a test of Virtual Lab, please let me know, or PM me. I know we will all benefit from the results.
 
You must log in or register to reply here.
Back
Top