Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Data verification software for copying off CF/REDDrive/etc.

angeleye said:
Thanks for the offer, it's good to know we have so many resources here. I already have file sync utilities and verification software. My question is does the camera format the CF card differently than just a straight format from the PC. ie right click the mounted drive and choose format. What is the difference, other than a process to minimize mistakes?

Chris
Click to expand...

Most verification software checks whether two or more existing files are identical. If your copying big files (as in Red footage), it would be BETTER TO PERFORM CHECK DURING THE COPY PROCESS, which would save you reading the input file twice, as well as having to read in the new copy, again, for verification. This would save quite a bit of time, especially if you have limited CFs. I already have my own command line tool for this, however I could easily create a windowed GUI, which WOULD run under Linux using WINE also (24 hrs). The copy process would go like this:

1) Buffer some data from file being copied
2) Write buffered data to new file (on hard disk)
3) Close new file, then immediately open for reading
4) Read newly written data from new file, compare to data form original file (still in buffer)(display message if data differs)
5) If all goes well, close new file, then imediately open again (in file append mode) or exit (if error)
6) Repeat process till file copied

done, new file is identical (assuming you don`t have faulty memory!). Using a good size buffer really speeds up disk copies too.
 
I have working, on the fly, bit for bit file copy utility with a simple GUI for windows (but will work under Linux through WINE I suspect). I wrote it last night. You can select individual files and copy them where you wish. If no errors are reported during copy, it means YOU DO HAVE AN IDENTICAL COPY OF SOURCE FILE. I shall triple test it tonight, and add the ability to copy multiple files and possibly folders in the next couple of days. It does work now however.
I had a command line version written for my own use, which I use for transfering files (video mainly) to my backup drives, it could be a stop gap solution for anyone worried about data corruption, until something better turns up (which it will). It copies pretty fast. If anyone is interested post a message on thread or send a PM. best regards.
 
Sycophant said:
An earlier post by one of the RED tech team (I forget who, or where) stated that the risk with formatting 'off camera' was that the camera has a few extra steps it takes - things that ensure it increments the reel count and possibly other things.

Although Offhollywood certainly makes it sound like he knows a lot more about the process than the rest of us.
Click to expand...

Formatting the card or drive FAT32 is obviously easy.
Writing the magazine files onto the card with the proper information is easy too - but requires a bit of careful thought. And naming the volume etc. So an application or utility will need to keep track of reel #'s etc.

But yeah, the basics are not rocket science and I expect lots of folks to make simple utilities - what we are trying to do is a more comprehensive application that works from injest through delivery, managing .r3d media and meta data - importing and exporting XML from FINAL DRAFT, FCP, REDCINE, etc.

One thing to remember - Jim and Rob have stated several times here on REDUSER that specs of the files are changing and that could mean that we will see minor changes to volume/file/folder structure. I have requested many times a root folder on each card/drive and I expect that to happen eventually. So .. if you are making a utility or anything else (like we are) - expect stuff to change. And don't be upset when it does.

I think RED would really prefer that you use the camera to format unless you make a utility that writes the correct info into the magazine files and you have tested it extensively.

Hope that helps.
 
angeleye said:
What is special about the camera formatting? If you just format using FAT 32 on a PC, does the camera still want to reformat? Is there an off the shelf utility that will handle the Red CF format specs?

Chris
Click to expand...

Would it affect the Red camera at all if you formatted the CF card off camera (i.e. after backing up just drag the folder contents to the trash and empty it?) and then format it again on the camera itself. This way the camera warns you if there's anything left on the card (which there shouldn't be!) and also you have the added benefit that the camera is doing its own re-formatting (whatever it does that's special/unique compared to formatting off-camera).
 
angeleye said:
What is special about the camera formatting? If you just format using FAT 32 on a PC, does the camera still want to reformat? Is there an off the shelf utility that will handle the Red CF format specs?

Chris
Click to expand...

In a nutshell, you want to format it in the camera because the camera formats the card exactly how it needs to be formatted for acceptable performance. When you get right down to it, you can always format it in a PC but the cluster size is not going to be the same (at least not unless it just happens to be such) as it would be if you formatted it in the RED camera. If the RED CF card writes faster with the card formatted to a precise specification (Sandisk Extreme cards have that quirk) than you want to use that spec every time to avoid dropped frames. RED or someone else can easily write a utility that formats the cards per the exact spec for maximum performance but unless RED shares that information (which they might not even have until they do a lot of testing and tweaking of the card sectors) or someone here discovers it, you should format it in the camera for the sake of playing it safe.
 
Regarding checksum utilities, you can write a checksum to the CF card before copying it to hard drive. Of course, then you have doubled your CF read time - once for the checksum and again for the copy. Might be a bit tight on a busy set.

For the PC, Advanced Checksum Verifier is an elegant shareware tool that can create a single checksum file for entire subdirectory structures. I'll be using it for HD/HD copies.

I wonder if I can use it on the RED Drive before copying it as well...

Michael
 
I wonder if theres a solution from the stills world, these days they are constantly shooting big files with big cards, i know theres plenty of capture 1 style arrangements but Im sure theres DIT's for the stills world who would perhaps have some insight?
 
I believe it can serve to bridge between the CF card and a USB hard drive.

Depending on your setup, it's still not ideal as it's not necessarily going to offer any form of redundancy.
 
Sycophant said:
I believe it can serve to bridge between the CF card and a USB hard drive.
Depending on your setup, it's still not ideal as it's not necessarily going to offer any form of redundancy.
Click to expand...
Acknowledged. I think their main comment was "damn sight easier than juggling a macbook and card-reader" while flying. Of course all data was duplicated from the Hyperdrive after landing to USB/FW drive via Macbook. The Hyperdrive remaining as one of the copies.
 
Michael Morlan said:
Regarding checksum utilities, you can write a checksum to the CF card before copying it to hard drive. Of course, then you have doubled your CF read time - once for the checksum and again for the copy. Might be a bit tight on a busy set.

For the PC, Advanced Checksum Verifier is an elegant shareware tool that can create a single checksum file for entire subdirectory structures. I'll be using it for DH/HD copies.

I wonder if I can use it on the RED Drive before copying it as well...

Michael
Click to expand...

I`ve got a working prototype, and you DONT have to read anything off the CF card twice, only from the destination drive (ie, Raid, to verify copy with read data). My utility copies quicker than Windows Explorer can (in vista anyway), because of a usefull buffer size.
 
Ronx, interesting. But don't you have to read every byte of a source file in order to generate a checksum before the copy? How does yours differ?

Jashunter, interesting find. It is RED's commitment to using standard peripherals that allows us to have an offloading solution for $150 instead of $2000+ for the Panasonic P2 Drive. One problem I could imagine with the HyperDrive: 1GB/min is a bit slow for CF transfers on a fast-moving set.

Michael
 
Implementing your own copy routines make it a little easier.

You read a section of the source file into a buffer, calculate checksum on that block, write that to the target and then read the block back from the target file to verify checksum.

There are existing file copy tools that implement on-the-fly checksum verification, it's just a matter of finding a suitable one I guess.
 
You must log in or register to reply here.
Back
Top