Data Management - Rookie

[lasave]

Hello everyone, as you can see this is my first post-thread, and in 4 days i´ll be real working with a MX RedOne as Data Manager.

Ir´s a short film and i´ll be in charge of the media, back up´s and this kind of stuff, and also they want me to get the Raw clips and downconvert them into AppleProRes 422 so the editor can have a media offline to start editing from the first day.

Here it´s my problem, which is the best application for getting AppleProRes? i can still use RedRushes even though it´s Mysterium X? this task can be done with the Redcine X?

I´m going to use a MacBook Pro there´s any risk my laptop dies trying to process all the data?

Thank you very much and sorry for any mistakes on my english.

 

[Martin Weiss]

RedRushes is no longer supported. Download redcine-x from http://red.com/support and/or give Storm a try. You might also want to have a look at Clipfinder which lets you set a watch folder, from which it will transcode to your heart's delight.

For editing, there is no real need anymore to transcode the footage, as most editing systems can now ingest R3Ds directly - or use the quicktime proxies when using FCP.

 

[Von Thomas]

Get a copy of R3D Data Manager for your back up, and make sure the production provides you with two shuttle drives (PREFERABLY G-TECH OR MAXX DIGITAL that go to editorial each night). Maxx digital has great storage devices such as their SAS RAIDS (EVO 2K AND EVO 4K) that will hold your entire project.

You would be better served using a MacPro with a Rocket card (MacBook Pro will not die, but will not process the files in your timeframe, unless you are taking them home and working over night). You'll need a APC (or something like it) for power battery backup (someone WILL pull your power at some point, and this will protect you). Extra FW cables, 2 to 3 CF card readers (they get hot, and malfunction),

Clipfinder software, QT 7, REDAlert, the lastest version of REDCine X, a good monitor (I use the Eizo 232W, it's amazing, but pricey). You'll need a cart (whether you go MacBook or MacPro), and I recommend The Big Foot Cart.

 

[joe hedge]

Probably the most important advice for a first-time data wrangler is: do not drag-and-drop raw camera files from the Red Drive/Cards to the external back-up drives. Purchase either R3D Data Manager or ShotPut Pro (both are available as downloads), and use it instead of drag-and-dropping. Raw camera files are particularly susceptible to being corrupted by drag-and-dropping, always use dedicated file transfer software like R3D DM or ShotPut Pro.

Start off with either brand new drives or drives that have been freshly reformatted with nothing on them. G-Drives are the industry standard, no cheapo LaCie or Western Digital MyBooks.

If your MacBook Pro has an express34 slot, buy an express34 card with 2 eSata ports, and use the eSata ports on the G-Drives for transfers, which will speed up back-ups quite a bit.

Be very careful what you plug into on-set, the drives are susceptible to getting fried by electrical spikes/surges/blown fuses etc. Try to get the production to rent/buy an APC UPS, or just buy one ourself, or don't plug the drives in at all, download someplace else that the production isn't using.

No other programs running while raw camera files are being backed up (no web-surfing etc)

Check the proxies (by playing them back and watching them, called "QC'ing") after they've been copied to your external drives, but BEFORE you give the camera drives/cards back to the AC, to make sure nothing got corrupted. If you see any dropped frames/digital noise, make sure the camera isn't doing it by checking the camera drives/cards, if the problem is on the camera media, notify the AC immediately.


Those are just the bare basic of on-set data wrangling and should keep you out of trouble...

 

[Cail Young]

Check the proxies (by playing them back and watching them, called "QC'ing") after they've been copied to your external drives

Drive around a circuit and don't hit the wall, this is called "racing".

There's a lot more to QC than just watching the clips on a laptop. I totally agree that visual checking of media is necessary but let's not let producers think that QC can be done on set on a laptop. Call it a "visual confirmation".

 

[Randy Lee]

Sorry to add questions into the great advice here, but I've never seen this one addressed - When back at the office and copying media off of the shuttle drives onto a better working drive such as a Raid, does that person typically use r3d Data Manager or Shotput Pro as well? Is that also used when copying to LTO or other storage for long-term backup? Or at what point in the game do the pros stop using the dedicated media copying applications and turn to the Finder or other tools for their additional copies?

 

[Jeff Kilgroe]

Drive around a circuit and don't hit the wall, this is called "racing".

There's a lot more to QC than just watching the clips on a laptop. I totally agree that visual checking of media is necessary but let's not let producers think that QC can be done on set on a laptop. Call it a "visual confirmation".

+1

Additionally, the above posters are correct to some extent that simply dragging and dropping files can lead to problems. That copy process is as valid as any other, however external media connections, especially FW800 card readers and occasionally finicky RED Drives from someone's rental fleet can have produce data errors. You should always do a verified / read-back copy. Or a separate verification pass after the copy. Utilities like R3DDM can assist you with this process, but are not the only solution. You can also generate checksum values for each R3D file you offload and that checksum can be used to make a quicker job down the line of verifying subsequent copies and data integrity. An awesome tool that is often overlooked for CF card workflows on the Mac is Carbon Copy Cloner. You can do block-level copies, with verify pass, and copy each CF card to your DIT station as a read-only disk image. ...And it's free software, but I highly encourage supporting. CCC is one of the most useful utilities in the Mac world. Honestly, you can do everything it does from the command line, and a lot of what it does from disk utility, but CCC makes it straight-forward and simple.

If you decide to generate checksums, just keep in mind that the checksum is only valid from the point it was created. If the files are corrupt on your camera media or corrupted when transferring from the camera media, even generating a checksum at that time, you can have a checksum value that does nothing more than ensures the integrity of your corrupt data. Hence the visual verification process and other steps you may be able to take. A big part of being a DIT is providing a production with competent hardware adequate skills where such problems can be avoided. It's difficult to verify footage, ensure data integrity, etc.. if you don't have the right hardware and proper time and on-set resources to do so. Trying to do this to a high degree of efficiency and reliability on a laptop would be a real stretch.

Sorry to add questions into the great advice here, but I've never seen this one addressed - When back at the office and copying media off of the shuttle drives onto a better working drive such as a Raid, does that person typically use r3d Data Manager or Shotput Pro as well? Is that also used when copying to LTO or other storage for long-term backup? Or at what point in the game do the pros stop using the dedicated media copying applications and turn to the Finder or other tools for their additional copies?

It's funny, there's a lot of DITs and Data Techs running around that have nothing more than a basic crash course in IT and data management. Some don't even have that. People like to say that R3D files are more prone to corruption and other such nonsense. When, in fact, that's not true at all. They're data, just like any other bag of bits on a hard drive. They are large -- often spanning multiple 2GB chunks. That right there raises the potential for something to go wrong, just by the percentage of data copied in relation to the number of files. But isn't really a reason for concern. Lots of other applications deal in large data sizes too. The shoddy connections often used on productions to DIT systems are mostly to blame, most of the time. And camera media can have faults too, especially RED Drives.

The first step is to get the camera media offloaded or at least backed-up. The best scenario is to back-up, but not format and recycle the media. Which almost never happens. Once a couple verified copies are made, then people do relax about further copies and ingesting it into their systems. After all, if there is a problem, they can re-visit those first backup copies that were made, right?

I make it a rule to never, ever format a piece of camera media until I know I have at least two verified copies of that footage in safe hands. One set of those hands being my own. Three copies is even better if time/resources allow.

Also have to accept that while bad things do happen down-stream in post, there usually should be proper back-ups made at various stages. And properly configured and maintained hardware should encounter problems very rarely. The most important thing is to get that footage offloaded onto multiple sets of media and verify it's all there in proper working order.

 

[Conrad Hunziker]

When back at the office and copying media off of the shuttle drives onto a better working drive such as a Raid, does that person typically use r3d Data Manager or Shotput Pro as well? Is that also used when copying to LTO or other storage for long-term backup? Or at what point in the game do the pros stop using the dedicated media copying applications and turn to the Finder or other tools for their additional copies?

Is one copy any less important than any other copy?

On just about every feature Ive worked on, there has always been a point where s*%t happened and they needed to go to my backups to retrieve the footage. At that point, my backup became the holy grail.

R3D Data Manager was designed to be used at each stage of the copy process. On set, near set, post, archival and restoration. It provides a easy, quick way to ensure that the zeros and ones exactly match from the first point R3D Data Manager was used. Some of the very first Red footage ever shot I know is still exactly as it was written 3 years ago, because of this technique & workflow, even though the original media has been written over thousands of times. CCC & ShotPut dont provide that level of verification.

Yes, you can do some of what R3D Data Manager does with command line scripts, but there are parts that cannot be done, and command line scripting is subject to humans falling asleep/forgetting after 14 hour days. With zero clicks (i.e. fully automated) you can get all your shots copied to 4 destinations and checksummed twice - usually faster than finder or the command line can make that same copy. And if you are not familiar with how the command line works, you shouldnt play around with it on important footage. Bash is not for the faint at heart. (BTW, the first version of R3D Data Manager - v0.8 - was a bash script)

Even after that, you should be watching your footage from a destination in real time. Scrubbing is the next best, but may not catch everything. If you use R3D Data Manager for your copies, then you only need to visually verify one copy - as the other copies have been verified to have the same zeros and ones in the exact same order. If you copy with finder, you really should visually verify each copy. For the copy to post, if they use finder at that point, they should either visually verify again or run a "verify checksums" pass from R3D Data Manager (assuming the first copy was done in R3D Data Manager).

One other tidbit that I dont see mentioned often is to just have tons of Red Media. Just by having more than 2 or 3 pieces of media, then you are not always chasing your tail to try and get media offloaded before it can be reused. That can lead to rushed decisions, which can lead to other bad stuff.... Do yourself a favor and get more Red Media than you think you will need. As a general rule, Ill never go out with less than 8 Red Media per camera, but that is also optimized for my workflow, computer speeds, HD/RAID speeds, and the productions I usually go out on (features). Then I know I have the ability to backup and visually verify in real time before the cards are even close to being needed back on set.

 

[Randy Lee]

Thank you Jeff and Conrad - I've been trying to say the same, that it if the media is important, it needs to be treated properly every step of the way. I've run into situations where it isn't, and while I've yet to run into anyone needing to come back to me for a copy that isn't corrupted, it's nice to know that I'm not the only person who has run into this sort of situation.

*Edit*

On another related tangent, Jeff, you mentioned "It's funny, there's a lot of DITs and Data Techs running around that have nothing more than a basic crash course in IT and data management." Where can a person go to get more than just that? I read forums constantly - here, Creative Cow, Cinematography, DVXUser, and so on, but feel like I'm lacking in formal education that the the pros have. Is there a good textbook or course set somewhere that would give a very in-depth, detailed rundown? None of this "oh, read this set of posts and that quick article" stuff - I would love to find the appropriate classes or sources that take the proper dedication, and have way more detail than I would I think I could ever use. Any suggestions? Mods, if this would be better as a separate topic, make it so, or let me know and I will.

 

[Andy Best]

First thank you for all this great advice. This is very helpful.

I am also new to the RED world and now I know that I have been playing with fire. I have been dragging and dropping raw files to two separate WD drives.

So from now on my flow will be from RED through R3D Data or Shotput to G-Tech drives on set. Then at the office from the drives to my backup database to store at the same time Cooking through REDcine to have apple pro res 422 HQ files ready to edit on an EditShare.

Is this process safe and accurate. Thanks again, I really missed out not getting to REDUCATION this time around, but I am planning to get to the next one.

 

[Von Thomas]

Mini RED DIT Cart

Mini RED DIT Cart

This is my mini cart.

 

[Jeff Kilgroe]

On another related tangent, Jeff, you mentioned "It's funny, there's a lot of DITs and Data Techs running around that have nothing more than a basic crash course in IT and data management." Where can a person go to get more than just that?

I don't have any specific recommendations off the top of my head. It's knowledge that can be gained a variety of ways, often through experience. Partially I'm speaking out of frustration... Over the past year, I've encountered too many so-called "DIT's" or "data technicians" that can't perform basic data tasks on request or they give me a blank deer-in-headlights gaze when I ask them about their data verification process or tell them I want MD5 checksums made along with a visual verification of each clip. Had one shoot where I missed lunch so I could show the "DIT" how to replace the crashed HDD in his Macbook Pro. ....I mean, come on.. really? Glad I didn't hire the guy, the producer did.

I don't think just reading books or articles really prepares anyone to deal with the unforeseen. That comes with experience and having a broad knowledge base or enough experience to draw on. But one will never gain those without a basic understanding of what tools are needed to do their job. If you don't know the ins and outs of your hardware, software, OS you run your software on, etc... You will eventually run into a problem somewhere.

Not all DIT's are data technicians. Larger productions will have data managers or data techs in addition to DITs. DIT has become a blanket description for Digital Imaging Tech, Data Tech and Digital Loader.

I could probably write a book on what I think a "DIT" should be capable of doing.

But I think any technical person on a professional set or in a professional environment should be responsible for their equipment. They should know enough about their equipment and the job at hand where they can manage hardware and software failures, do repairs on the fly if need be and not completely shut down in the face of a hard drive crash or blown PSU. If I roll a DIT cart onto a set and I have a HDD crash or system failure, that is my problem, not the production's problem. I have to keep moving forward. I keep two systems on my cart in addition to a Macbook Pro in my bag... I have spare HDDs and components to do repairs and drive swaps.

 

[Von Thomas]

Jeff is correct, if pressed most so called DIT's would fall apart if they encounter computer system failure. Personally, I did not go to school to gain my skill set, it's been my long slow road in the Mac world starting with my first Apple IIe in 1984. Actually that's not quite true, I was a data processing major in college, and my mom ran a keypunch service. I routinely would deliver cases of processed IBM punch cards to big main frame computer installations (like TRW), and marvel at the giant computer rooms filled with cool gadgets. You really have to be a tinkerer to know your way around a computer.

I ran a digital capture boutique for over 10 years in New York, DigitalTech NYC, renting high end digital capture backs (Phase One), medium format cameras (Hasselblad), DSLR's (Canon 1DS MK II's, and MK III's), monitors (Eizo), and advising my clients (Spiegel, Saks, Bloomingdale, Gray Advertising, Deutsch, and many others) on everything from best archival solutions, color management, and file processing.

Prior to that I'm a professional advertising photographer for the past 37 years, Von Thomas Photography. Having shot everything from sports to product, architectural, advertising, and finally specializing in fashion in Manhattan.

I believe if you peel back the layers of a good DIT, he/she will have a lot of experience in many areas. Being a still shooter, and working in digital capture since the early days, allows me to evaluate exposure using the histogram, the tool most accurate for exposure on RED (or any digital camera for that matter). A lot of DP's turn to me to call out stops, I've been doing this so long, I can tell if the shot needs as little as a 1/4 stop bump in either direction. Working in color management, and using the proper tools (Eizo monitors, Color Munki calibration device, 18% gray cars (AKA The VonCard)), helps me work with neutral monitoring device on set, and the ability to do one light color correction with confidence. Backing up clients files has been my business for the past 13 years, and making sure they reach their destination as intended. To date, I've never lost a file (KNOCK on WOOD).

Outside of this is everything else, RED setup, RED trouble shooting, computer setup, and troubleshooting, it's a lot to deal with, and not to be taken lightly. I don't think you need to be a computer mechanic, but you do need a dozen or more trouble shooting skills in your bag of tricks.

If anyone wants a crash course in what I do, PM me, but I'm not cheap.

 

[Jay Gannon]

Jeff is correct, if pressed most so called DIT's would fall apart if they encounter computer system failure. Personally, I did not go to school to gain my skill set, it's been my long slow road in the Mac world starting with my first Apple IIe in 1984. Actually that's not quite true, I was a data processing major in college, and my mom ran a keypunch service. I routinely would deliver cases of processed IBM punch cards to big main frame computer installations (like TRW), and marvel at the giant computer rooms filled with cool gadgets. You really have to be a tinkerer to know your way around a computer.

I ran a digital capture boutique for over 10 years in New York, DigitalTech NYC, renting high end digital capture backs (Phase One), medium format cameras (Hasselblad), DSLR's (Canon 1DS MK II's, and MK III's), monitors (Eizo), and advising my clients (Spiegel, Saks, Bloomingdale, Gray Advertising, Deutsch, and many others) on everything from best archival solutions, color management, and file processing.

Prior to that I'm a professional advertising photographer for the past 37 years, Von Thomas Photography. Having shot everything from sports to product, architectural, advertising, and finally specializing in fashion in Manhattan.

I believe if you peel back the layers of a good DIT, he/she will have a lot of experience in many areas. Being a still shooter, and working in digital capture since the early days, allows me to evaluate exposure using the histogram, the tool most accurate for exposure on RED. Working in color management, and using the proper tools (Eizo monitors, Color Munki calibration device, 18% gray cars (AKA The VonCard)), helps me work with neutral monitoring device on set, and the ability to do one light color correction with confidence. Backing up clients files has been my business for the past 13 years, and making sure they reach their destination as intended. To date, I've never lost a file (KNOCK on WOOD).

Outside of this is everything else, RED setup, RED trouble shooting, computer setup, and troubleshooting, it's a lot to deal with, and not to be taken lightly. I don't think you need to be a computer mechanic, but you do need a dozen or more trouble shooting skills in your bag of tricks.

If anyone wants a crash course in what I do, PM me, but I'm not cheap.

Heh we have very similar backgrounds and have both ended up as DITs.

As a DIT you should be essentially the onset computer genius.

I have qualifications in Storage Tech, Networking, Wireless, Windows, Mac and anything else I can lay my hands on. I've ran everything from small data centers to college wifi installations. I've also worked as a photographer for many years both on set and commercially.

As a DIT you should aspire to know everything about your systems from the ground up, if necessary you need to be able to run to a shop, grab a component strip your MacPro and rebuild it in less than an hour, while diagnosing a problem with the Wifi and checking the grade on the last setup.

Its a hard but worthwhile position and its great fun and on a lot of sets people hang around your cart and you can have some great interactions.

 

[Von Thomas]

Heh we have very similar backgrounds and have both ended up as DITs.

As a DIT you should be essentially the onset computer genius.

I have qualifications in Storage Tech, Networking, Wireless, Windows, Mac and anything else I can lay my hands on. I've ran everything from small data centers to college wifi installations. I've also worked as a photographer for many years both on set and commercially.

As a DIT you should aspire to know everything about your systems from the ground up, if necessary you need to be able to run to a shop, grab a component strip your MacPro and rebuild it in less than an hour, while diagnosing a problem with the Wifi and checking the grade on the last setup.

Its a hard but worthwhile position and its great fun and on a lot of sets people hang around your cart and you can have some great interactions.

True. It's normal for everyone on set to come to my cart with their own computer problems, looking for help, if time permits, I help them.

Since my early days, I try to stay ahead of the curve with technology (turns me on). So you'll most likely see something new on my cart from time to time. It's not just there as another gadget, it's a function part of my gear, and it always turns heads.

 

[Dustin Cross]

Similar background to Jeff, Jay and Von. Years of computer experience plus a masters in computers and 10 years experience in post and camera. Once Red came out DIT was the logical place for me, when I am not shooting or working on my skills as a Colorist. Learning all the skills I use onset just takes a lot of time and then you have to keep up with the changes. Of all the people doing DIT work in my area, I only trust one of them enough to recommend to jobs when I am busy.

I really wish more shows had a DIT and Data Manager team. That way the DITs can train and mentor Data Managers into good Data Managers and eventually into good DITs. Every other department has entry level people working with experienced people so they can learn.

I have thought about teaching classes in proper data management, but I think someone needs more than a few hours of training incase they run into trouble.

I have described an inexperienced data manager as a PA reloading film mags inside a black plastic trash bag. That is the best analogy I have come up with to get productions to understand why they need somone who knows what they are doing and has the proper tools.


Dustin Cross
Local 600 DIT