Best Tools for Data Wrangling?

[Brandon Kraemer]

I am trying to get some advice that I can pass along on creating a kit that would be sent out on shoots for the purpose of data wrangling, and some advice on the art of data wrangling itself, including what to expect from the crew and rental houses in terms of equipment and expertise provided.

First, in terms of providing hard drives to the data wrangler and to protect the footage, the idea is to buy between 8 and 16 FW 800 drives, and divide them up into two kits, an A kit and a B kit, packed in two Pelican cases. These kits would become mirror images of each other as media gets off loaded from Red capture media (CF cards, RED Drive, RED RAM) by the data wrangler. At the end of the shoot, one kit would be shipped separately from the other for redundancy.

I'd like any feedback on the following...

1. Assuming all media transfered to these drives is checksummed, is this redundant enough? Does this kind of system make sense?

2. Should these drives be FW 800? Should they be powered via the FW 800 bus, or via AC.

3. We are considering 500 GB G-Tech drives. Any opinions as to if that is a good or bad purchase decision.

Second I'd like to hear thoughts on what kind of gear or compliance we should expect from a rental house. We recently had an issue with some corrupted media (dropped frames) that didn't copy cleanly of a CF reader. This wasn't caught in time and the media was recycled. After doing a little research we found some practices that would have caught the issue, mainly using R3D Data Manager to checksum the media as part of the data wrangling, and using Lexar FW-800 card readers. Considering the CF reader and laptop were rented from the camera rental company, we would like to think it would have been outfitted with the aforementioned software and CF reader. Looking forward, what specific hardware and software should we expect in this situation?

Lastly, where should responsibilities lie with the data wrangler? Should the production company have to mandate the wrangler work with certain practices to insure there is no data loss? Do data wranglers typically provide any gear or software?

Looking forward, we want to prevent any situation where we end up with data loss again. We are not interested in pointing any fingers, but advice on what we should expect going forward is most appreciated.

Thanks in advance.

bk

 

[Brandon Kraemer]

just a bump... I'd love some thoughts... drives?

 

[Peter Matthes]

I am trying to get some advice that I can pass along on creating a kit that would be sent out on shoots for the purpose of data wrangling, and some advice on the art of data wrangling itself, including what to expect from the crew and rental houses in terms of equipment and expertise provided.

First, in terms of providing hard drives to the data wrangler and to protect the footage, the idea is to buy between 8 and 16 FW 800 drives, and divide them up into two kits, an A kit and a B kit, packed in two Pelican cases. These kits would become mirror images of each other as media gets off loaded from Red capture media (CF cards, RED Drive, RED RAM) by the data wrangler. At the end of the shoot, one kit would be shipped separately from the other for redundancy.

I'd like any feedback on the following...

1. Assuming all media transfered to these drives is checksummed, is this redundant enough? Does this kind of system make sense?

2. Should these drives be FW 800? Should they be powered via the FW 800 bus, or via AC.

3. We are considering 500 GB G-Tech drives. Any opinions as to if that is a good or bad purchase decision.

Second I'd like to hear thoughts on what kind of gear or compliance we should expect from a rental house. We recently had an issue with some corrupted media (dropped frames) that didn't copy cleanly of a CF reader. This wasn't caught in time and the media was recycled. After doing a little research we found some practices that would have caught the issue, mainly using R3D Data Manager to checksum the media as part of the data wrangling, and using Lexar FW-800 card readers. Considering the CF reader and laptop were rented from the camera rental company, we would like to think it would have been outfitted with the aforementioned software and CF reader. Looking forward, what specific hardware and software should we expect in this situation?

Lastly, where should responsibilities lie with the data wrangler? Should the production company have to mandate the wrangler work with certain practices to insure there is no data loss? Do data wranglers typically provide any gear or software?

Looking forward, we want to prevent any situation where we end up with data loss again. We are not interested in pointing any fingers, but advice on what we should expect going forward is most appreciated.

Thanks in advance.

bk

There are certainly better people to answer some of these questions than me, but in terms of hard drives you might want to look into the use of small RAID systems like Caldigit or WiebeTech. They offer an easy way to split the drives at the end of a shoot day for the redundancy and safety that you are looking for.

At RAID level 1, theses units write to two drives simultaneously creating two mirrored copies. At the end of the shoot day, you can pull the drives. Send one to the editor and keep the other safe. The Caldigit drawer drives can be purchased separately, and the WiebeTech RAID enclosures use bare drives like the WD Caviar Black drives. In both cases, you can purchase them in all different sizes as the need arises. (500GB, 1TB, 1.5TB, etc.)

I have some limited experience with R3D Manager.

The use of R3D Data Manager provides a great deal of security with its checksum features, but when those are employed the download time can be prohibitive when using more than one camera. Hopefully as time and technology progress, those times will improve significantly.

Those are just a couple thoughts.

 

[Morning Glory]

Data Wrangling - Just Say No

Data Wrangling - Just Say No

First, I would suggest dropping the title "Data Wrangler". Doesn't seem to fit in most cases. There are many other titles to choose from that are more accurate and respectful, the old school Digital Imaging Technician, Data Aquisition Supervisor, Data Capture Technician and my personal favourite: Data Management Technician. Second, you have to decide what you want that position to entail. Potentially there are 4 major responsibilities:

1. Camera Tech: Supporting the Camera, knowing it inside and out and being able to troubleshoot problems.
2. Managing the Data: Offloading the Digimag to Multiple Destinations (the Diginegs) Doing a checksum Verification. Doing a Visual Verification (you HAVE to watch these takes from head to tail!!! At least the circled takes). THEN recycling the Digimag.
3. Timing the Dailies: Working with the DP on a look for the dailies. Whether there is a lab or not, I feel the best place to capture the DP's intent is on set, usually between set ups and at wrap.
4. Production - Post-Production Bridge: A good DMT will be involved at prep to help determine the workflow and is in communication with the lab, post supervisor and editorial to ensure the dailies process is running smoothly and whatever is captured and processed on set is optimum all the way through the chain to the final delivery.

As for drives, I would recommend against renting drives as part of a kit. Drives are expendibles just like film and tape and production should bear the cost. You COULD tape over an HD SR tape, but would you? Consider the nightmare of determining liability if/when the drives fail. I have worked with Lacie drives and have had good experience for over 2 years now. You can find someone who has had a failure with any brand of drive. I would choose a drive with an eSata connection. Depending on the length of the show and who is doing the dailies, I often recommend a Lacie Big4 RAID in a RAID 5 configuration as the Digineg Master, it stays on set and contains the whole show for easy reference and will live with the production as the master copy. The second copy is a set of 3*1TB Lacie d2 Quadras. They serve as the Digineg Runners. They travel between set and the lab. At the lab an archival copy on LTO is also made. This can be done on set as well. Of course, different shows have different requirements, and so each show should have a custom workflow designed in prep.

 

[Harry Clark]

Heed Morning Glory's advice. He is spot on with all comments.
Cheers,
Harry

 

[Morning Glory]

Thanks, Harry. That's kind of you.

As you mentioned in your 1st post Brandon, Pelican cases, or Storm or Nanuk, are essential for all hard drives/Diginegs. In the case I laid out in my last post each Digineg Runner would have it's own case so they could travel individually as needed and the Digineg Master has it's own as well. I've almost walked off shows as the producers were resisting buying proper cases for their Digineg.

 

[Brandon Kraemer]

First, I would suggest dropping the title "Data Wrangler". Doesn't seem to fit in most cases. There are many other titles to choose from that are more accurate and respectful, the old school Digital Imaging Technician, Data Aquisition Supervisor, Data Capture Technician and my personal favourite: Data Management Technician.

Certainly wasn't ment to be disparaging, it was the term I was introduced to. Since there is no "imaging" really going on, I think Data Management Technician (DMT) works best. Wasn't aware of all these official titles. Noted.

Second, you have to decide what you want that position to entail. Potentially there are 4 major responsibilities:
1. Camera Tech: Supporting the Camera, knowing it inside and out and being able to troubleshoot problems.
2. Managing the Data: Offloading the Digimag to Multiple Destinations (the Diginegs) Doing a checksum Verification. Doing a Visual Verification (you HAVE to watch these takes from head to tail!!! At least the circled takes). THEN recycling the Digimag.
3. Timing the Dailies: Working with the DP on a look for the dailies. Whether there is a lab or not, I feel the best place to capture the DP's intent is on set, usually between set ups and at wrap.
4. Production - Post-Production Bridge: A good DMT will be involved at prep to help determine the workflow and is in communication with the lab, post supervisor and editorial to ensure the dailies process is running smoothly and whatever is captured and processed on set is optimum all the way through the chain to the final delivery.

As of now, most of our productions (TV commercials) have had a really good AC that handled all the camera and formatting responsibilities. The Data Management was taken care of by assistant editors who cloned the media and handed back the mags to the AC. Since our offline workflow is most often with proxies, there is no timing of dailies. We found it to be a benifit to have the Assistant Editors onset, doing the work of a DMT, as they would later be the ones logging the footage. The last job we shot was overseas, an we had a DIT that copied and reviewed media. Just trying to get a handle on what that position should entail.

As for drives, I would recommend against renting drives as part of a kit.

Absolutely! Thanks for your thoughts on hardware as well. I will take this into consideration and pass the comments along.

 

[Brandon Kraemer]

Thanks, Harry. That's kind of you.

As you mentioned in your 1st post Brandon, Pelican cases, or Storm or Nanuk, are essential for all hard drives/Diginegs. In the case I laid out in my last post each Digineg Runner would have it's own case so they could travel individually as needed and the Digineg Master has it's own as well. I've almost walked off shows as the producers were resisting buying proper cases for their Digineg.

Thanks for that... it was something I strongly suggested after some of the "packing" jobs I had witnessed. Thanks for all your thoughts.

 

[Justin McAleece]

My DMT route

My DMT route

Here is a plan I have devised for a DMT job where we will only be able to use laptops on set and the kit must be pretty small and fast.

As for the plan, I am trying to stay very portable and as such I want to be based off of a laptop for the majority of the shoot. My plan is to use a Macbook Pro to transfer footage to while on location every day to a 2 hard drive system that will be in a hot swap RAID case through an esata express card adapter. One HDD will be a 1TB or possibly 2TB drive that will contain one copy of the footage up to the amount that fills the drive and then we will move on through as many drives as we need in sequence. The other drive will be a transfer drive that we will have 4 of and we will bring back and forth with me every day to the hotel or base to be downloaded to a 6 - 2TB drive RAID5 (10TB total after RAID5 loss of one drive). This way there will always be two drives with the data on them at any one time until the data gets to the RAID5 that I would like to set up in the office/base. That transfer drive will go back to the field the next day or day after that after transferring over night with checksum sequences in use.

To simplify:

Data from RED Drive, CF Card, RED RAM gets transferred simultaneously to a large, single backup drive and a smaller faster transfer drive on set by MBP.

The transfer drive gets carried to the base where it is downloaded with checksum to a Mac Pro with 10TB RAID5. Here the rushes and syncing will be done.

When full the individual 2TB drives from set will be stored elsewhere as a backup.

In this way we will have 1 RAID5 master drive that editing can be done off of and if one drive fails we just pop in another one and rebuild it at any time. The backup will consist of the non-raided individual 2TB drives.

What do you guys think about this approach?

 

[Morning Glory]

I would add in Visual Verification, watching clips from head to tail in real time, not scrubbing, and you have what I would say is a solid bare bones Data Management Strategy. I definitely would do this at the very least. A third copy at the hotel wouldn't hurt either, maybe residing in another room? And if you want to do any timing for dailies, either on set or in the hotel, a calibrated monitor, at least an eizo or an hp dreamcast, is a MUST.

 

[Zakaree Sandberg]

Morning glory,
Real names board... Thanks

 

[Morning Glory]

Certainly wasn't ment to be disparaging, it was the term I was introduced to. Since there is no "imaging" really going on, I think Data Management Technician (DMT) works best. Wasn't aware of all these official titles. Noted.

No worries, Brandon. I wasn't offended, just trying to clarify in these still early days of Digital Cinema.

As of now, most of our productions (TV commercials) have had a really good AC that handled all the camera and formatting responsibilities. The Data Management was taken care of by assistant editors who cloned the media and handed back the mags to the AC. Since our offline workflow is most often with proxies, there is no timing of dailies. We found it to be a benifit to have the Assistant Editors onset, doing the work of a DMT, as they would later be the ones logging the footage. The last job we shot was overseas, an we had a DIT that copied and reviewed media. Just trying to get a handle on what that position should entail.

Even proxies can be timed with Red Alert if so desired. The great thing about the Redcode workflow is it's versitility. It can be modeled to custom fit each show on an individual basis. The ideas I was presenting are based around the needs of cine style production. Even then I tailor the workflow for every new show. Good luck with all your Red endeavors!

 

[Morning Glory]

Morning glory,
Real names board... Thanks

No Worries, Zakaree. I refer you to this thread:

http://reduser.net/forum/showthread.php?t=34801&page=2

 

[John Howarth]

Brandon,

Some advice:

Unless you’ve got a client who specifically requests it, I would recommend staying well away from using portable RAID drives on set. Although the redundancy features of the various RAID levels are well documented, you should also consider the implications of the RAID controller failing – whilst it doesn’t happen so often, when it does…

We generally recommend making identical copies across multiple single drives (generally 3 drives per set of data), with the added option of LTO if additional data security/archive is required. Certainly not as fast as RAID, but it reduces some of the risk.

 

[Brandon Kraemer]

Brandon,

Some advice:

Unless you’ve got a client who specifically requests it, I would recommend staying well away from using portable RAID drives on set. Although the redundancy features of the various RAID levels are well documented, you should also consider the implications of the RAID controller failing – whilst it doesn’t happen so often, when it does…

We generally recommend making identical copies across multiple single drives (generally 3 drives per set of data), with the added option of LTO if additional data security/archive is required. Certainly not as fast as RAID, but it reduces some of the risk.

Solid advice.. I was concerned about the possible heft, fragility and power requirements of a RAID set. Might be well suited for a studio environment, but possibly less so for location. Thanks.

 

[Frank Martin]

We have been using 2-stand alone G tech
1TB drives since day one. Our third backup
goes on MacBook pro desktop. Add the varirlety
of software (r3d for shecksums,
Monkey Extract to viw thunbnails before
transcoding , changing DPX headers eta.
and evev clipfinder to quickly review a
long series if takes).
I noticed the new 4 TB drives are 550.00$
the same price we paid for the 500g
2-years ago! I would think these drives
would be more suited for the riggers of
location work. The G- Techs have been
bulletproof , two cameras two features and
countless commercials/ music videos.

Then just get your client to buyva small
500g esata shuttlecdrive and you will
have a nice simple DIT station. If they
needcan LTO backup take it to a post house
or wait for Redray

 

[Dane Brehm]

Make sure to get a headset to copy back to the 1st when your good.

Communication is key. Know Set etiquette. Alot of Red Techs piss DPs and ACs off and never get hired again

I use 3CP to back-up and generate LUTs for both Dailes and Post. It is the only professional tool for DITs. If you going with the DMT or Digital Loader route a laptop is sufficient but I would not go on-set without at least 2.

I recommend these monitors for on-set.

Panasonic 1760
FSI 1770 ( the best scopes in my opinion)
TVLogic LVM-171


Goodluck,

 

[Morning Glory]

Just for clarity's sake, at least in IATSE 667, a DMT is a DIT for Digital Cinema jobs. The DIT position is reservred for HD shows. A DMT is not a Digital Loader.

Dane raises an very important point. One can have all the knowledge and skills to make a good crew member, but if they lack the social skills they won't find much employment. The film industry is a intimate one. When you are on a show you spend most of your waking hours with the crew. If you are working, you are likely to be working 6 or more months of the year to be able to survive and thrive. That means while you are active in the industry you are spending the majority of your time with your crew. So a key is going to choose people they vibe with. I would say it's just as important as skill and knowledge. And different crews have different vibes. Compounded to this is the fact that this is a new position, and some DITs are taking it on in stride, but there are also some people new to film production who can be very knowledgable on the tech side may come off as arrogant or behave inappropriately. Cinema production has been refined for over 100 years now. It's good to be humble and find one's place within it. The structure exists for very good reasons. And some people can feel anxious about change. This new position befuddles some people and they want to discount it. Don't take it personally. Someone can be a perfectly good assistant and not be tech savvy. There are many amazing DPs out there that are not tech savvy. It can seem threatening to some. I see the DIT/DMT position partly as a assistant in this transformation from film to digital, easing the pain and fear and demonstrating the possibilities and joy.

 

[Harry Clark]

Once again, sage advice from Morning Glory.
Set ettiquette and social skills are at least as important as competency, and certainly more important than being a know-it-all!
Regarding 1TB plus sized drives,
Am I the only one who finds them finicky to mount on a Macbook Pro via eSATA? For that reason, and because I think more smaller drives are more flexible and efficient than fewer larger drives, I usually reccommend the 500 gig variety.
Cheers,
Harry

 

[Brandon Kraemer]

Am I the only one who finds them finicky to mount on a Macbook Pro via eSATA? For that reason, and because I think more smaller drives are more flexible and efficient than fewer larger drives, I usually reccommend the 500 gig variety.
Cheers,
Harry

Agreed about the 500 GB drive size. Seems like larger is also overkill considering that 2 or 3 redundant copies will be made. A lot of commercial productions would not fill a TB.

As for Morning's further comments about set etiquette, etc... thanks for sharing your thoughts. When it comes to protecting the footage, it seems like competency in that role is vital. Of course no one wants to work with a "jerk" on a crew, but hiring someone just because they are cool to hang with on a 12 hour day but might lack technical knowledge seems to be counter productive. Surely a balance can be found, and from my long time experience in both production and post, I know it can.

I think my original query was trying to get a handle on where the job description lines are drawn, so that a producer (I am not one) would know what they should and shouldn't expect from the skill set of a DIT or DMT and how much they are culpable for issues that are within their control. I can see that it's not so cut and dried. I respect that. I think the onus might ultimately on the producers/managers to get the knowledge they need to set protocols they expect to be followed? Like I wrote in the OP... trying to find some info to pass along so expectations on all sides can be understood better.

Thanks again for all the feedback.