Who wrangles, and how?

[Robert Atkins]

[I'm new 'round here, if this is the wrong forum feel free to move this post]

I've done a weekend helping some friends out data wrangling on a Red One shoot. It was fun and I'm curious to see if there's enough work around to make a business out of it.

I'm a "computer guy" rather than a "film guy"; so I know all about RAID levels and storage equipment and how to do off-site backups properly and such.

Aside from getting the data off the CF cards/Red drive and safely into the hands of the editor, what other duties are typically performed by a data wrangler? Is "data wrangler" even usually a separate job to "camera tech" on a professional shoot?

Finally, who's doing this today for all you people shooting on Red? Someone who's just fallen into the job and learning as they go or are there already lots of experienced professional data wranglers with well-tested workflows and bullet-proof gear out there? What rates do people charge? Is there much work about?

I'm in Sydney, Australia.

Cheers, Robert.

 

[Zakaree Sandberg]

its not a real position.
DIT is the position.
you must know all the camera functions, menus, settings, be able to troubleshoot codec errors, camera failure,
exposures, histograms so on..
on set CC (depending on delivery form)

backing up data is 10% of the real job

 

[Christopher Grant Harvey]

its not a real position.
DIT is the position.
you must know all the camera functions, menus, settings, be able to troubleshoot codec errors, camera failure,
exposures, histograms so on..
on set CC (depending on delivery form)

backing up data is 10% of the real job

True.

I just finished DIT'ing (a real word? :huh on a feature and the tasks listed above pretty much cover the duties I had to perform. Luckily we never had a major problem. I also updated the firmware of the camera to B18 and did a black-shading twice. I mostly formatted and handled the drives and cards coming from & to the camera.

As for data backups, I had 3 backups at any given time. I also kept a copy of the sound files from the sound recordist, who used a Fostex FR2 recorder. Just copied the files from the CF card.

I also provided dailies for the DOP and Director to view from the R3D's which was blazingly fast. During lunch I could have a few clips from the mornings shoot ready to view.

I used a very fast dual core PC laptop armed with Build 17 & 18, the users manual, 12 hard drives, some external housings, a few card readers, and some software.

 

[Zakaree Sandberg]

Robert,
since you said your more of a computer guy..
I encourage you to get onto as many red sets as possible,
or rent a camera and learn it.
Spend a few months reading this entire site, learning all the possibilities.
It is a Fun and rewarding job
so keep up the learning you came to the right place

 

[Robert Atkins]

I didn't think it'd be that simple .

A quick google comes up with http://www.4klondon.com/dit.html which looks like more of a complete job description, yeah?

Cheers, Robert.

 

[Nicholas Shields]

The most effective and sought-after DIT's are the ones that understand the filmmaking process. It's the ability to anticipate problems that arise in the normal course of production and prevent loss of time/data that make a DIT valuable. The only way to be this responsive is to know as much about the filmmaking process as possible. For example, as the crew is setting up an action sequence and you overhear someone mentioning "120fps", you need to anticipate the need for 4 times the amount of data per second being recorded and coming your way. It's the little things that will make you a wanted man. I don't believe that either background (IT or production) is in a better position than the other to succeed- but either way, be prepared to take on a new expertise. Best of luck with your endeavors.

Nick.

 

[Conrad Hunziker]

You asked about software, and Id like to point out that R3D Data Manager helps to take care of the actual copying process, in addition to being able to do quicktime renders. For more info, see the link in my signature.

That helps to take care of the 10% of your job that data management comprises. You'll have to figure out the rest on set.

 

[Zakaree Sandberg]

You asked about software, and Id like to point out that R3D Data Manager helps to take care of the actual copying process, in addition to being able to do quicktime renders.

havent used this particular software yet, so I cant comment on the quicktime rendering.. but I will never trust sofware to do all the copying and backup.
it may be a good starting point.. but always double check.

 

[Christopher Grant Harvey]

Oh yes for PC guys check this out: http://www.codesector.com/teracopy.php

I used it to handle all my copying and it works amazing.

Copy files faster - TeraCopy uses dynamically adjusted buffers to reduce seek times. Asynchronous copy speeds up file transfer between two physical hard drives.

Pause and resume file transfers - Pause copy process at any time to free up system resources and continue with a single click.

Error recovery - In case of copy error, TeraCopy will try several times and in the worse case just skips the file, not terminating the entire transfer.
Interactive file list. TeraCopy shows failed file transfers and lets you fix the problem and recopy only problem files.

Shell integration - TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual.
Full Unicode support.

 

[Conrad Hunziker]

havent used this particular software yet, so I cant comment on the quicktime rendering.. but I will never trust sofware to do all the copying and backup.
it may be a good starting point.. but always double check.

Well, Id invite you to use the software.

This software picks up where finder/explorer leaves off. Neither finder nor explorer verify that files actually made it across, nor do they do any sort of file verification, and many times they ignore critical errors. R3D Data Manager checks for all of this. And its one-click copying, which helps to remove the human element.

You make a point about never trusting software, but this software will show you whats gone wrong when finder doesnt, and provides ways to ensure the copied data is an exact duplicate of the original, even when the original is long gone years from now.

 

[Christopher Grant Harvey]

Neither finder nor explorer verify that files actually made it across, nor do they do any sort of file verification, and many times they ignore critical errors. R3D Data Manager checks for all of this.

Teracopy does all this for Free for PC guys who would be interested...

 

[Conrad Hunziker]

Teracopy does all this for Free for PC guys who would be interested...

I dont see where teracopy does checksums, verifies file data integrity or provides for copying redundancy. R3D Data Manager also allows you to check to ensure that files you copied years ago are still valid, even if the source doesnt exist. Even the pro version, which does cost money, doesnt do any of that.

 

[Christopher Grant Harvey]

It comes pretty close considering it is free. Perhaps my choice of words should have been, "it comes quite close and offers similar functions that work in the exact same way to ensure you never lose or miscopy data."

It has it's own system of checking that the files are copied correctly without problems which works perfectly fine. It verifies after each file has been copied and then after the whole process has finished it verfies again letting you know of any problems.

Can't see why you would need more than that...

If the files you copied don't open from years ago then they aren't gonna open so how would R3D data manager solve that? Just wandering...

 

[Uli Plank]

I can only second this– R3D Data Manager is a very helpful piece of software.

 

[Von Thomas]

Data Wrangler or DIT

Data Wrangler or DIT

I have 8 years experience doing digital tech work for the photo industry, and willing to work for free to learn Data Wrangling on RED systems.

I owned a digital capture service in NYC which handled digital still photo shoots (advertising, fashion, editorial). I've recently moved to L.A., and I am transitioning over to digital video capture.

The still cameras I've been working with are the latest 30 and 39 mega pixel rigs. The computers I use are Mac Pro's and MacBook Pro's. I typically had these computers outfitted with the fastest everything (video cards, drives), and minimum 8GB ram to 16GB, to handle the large amount of data.

As I understand from reading this thread, there are many similarities to the still world. I have to know all digital cameras and backs I work with, the operation, the settings, set-up, trouble shooting both cameras and computers if necessary, the software, back-up procedures. This was a learning process that took years. Well I'm ready for a new challenge.

I'm looking to getting hands-on experience with RED cameras to learn the workflow. To give you an idea of what I do, here is a link to my site. http://www.digitaltechnyc.com.

Von
347.463.3731

 

[Conrad Hunziker]

It comes pretty close considering it is free. Perhaps my choice of words should have been, "it comes quite close and offers similar functions that work in the exact same way to ensure you never lose or miscopy data."

But it doesnt do it the exact same way, thats the point.

It has it's own system of checking that the files are copied correctly without problems which works perfectly fine. It verifies after each file has been copied and then after the whole process has finished it verfies again letting you know of any problems.

It does? I dont see that mentioned anywhere in the software. If it does, they dont list it as a feature, which would be pretty big to miss. In fact, unless they are doing checksums there is no way to future-proof that the data you have is correct.

Can't see why you would need more than that...

Ok - heres a real-world instance, that has happened more than once in the past year.

You have made a copy of your data, and made an exact verified backup. Now you have your 2 perfect copies. You put them on the shelf. You come back a month later and look at both copies. Something happened to the media and the file sizes differ! Which one is right?

Your program wouldnt have the foggiest idea. In fact your program would be useless. R3D Data Manager would be able to tell you.

If the files you copied don't open from years ago then they aren't gonna open so how would R3D data manager solve that? Just wandering...

Though the checksum is one-way, if its just a couple of bits in the wrong place you can figure it out by figuring out what you would need to alter to get your checksum to match. Its not that hard really, but would take time.

The bigger point is that R3D Data Manager provides a verified way to make up to 4 copies at once, and while doing so will satisfy studio and bond completion requirements. If you are going to shoot anything with money behind it, the money people want an audit trail - which R3D Data Manager provides.

I dont mean to steal this thread on the merits of one software vs another. I do wish to point out that as a DIT, the data that you are moving around costs tens to hundreds of thousands of dollars to produce. Its worth more than the sum of the data. You really need to consider your workflows and options you provide for the productions you work for. The more reliable you make the stuff around you, the more it will pay off, both in terms of jobs, but also in terms of data security.

Having worked as a DIT and/or Operator and/or DP on nearly 2 dozen RED shows at this point, including a feature currently playing at sundance, theres a level of reliability and safety that producers will require - especially since for a lot of them this is their first venture into anything Red. There simply is no software on the market designed to make multiple copies with a single click that will verify your data independently of the source being present. Thats why I wrote the program within the first week of the first Red being released and had a 1.0 version by January 2008.

I have yet to loose one byte of data for any Red show. Thats not only because of the software, but also because of the management practices on and off set. Even if Im the DP or Operator Ill pay close attention to what the DIT is doing and help them accomplish their mission - not to be overbearing, but because thats gold in them-there drives!

 

[Christopher Grant Harvey]

I have yet to loose one byte of data for any Red show.

Same here, just using Teracopy.

I see where this discusion can go... I don't want it to go there...

 

[Robert Atkins]

Photocon: nice thread hijack, well executed ;-)

I did actually find R3D Data Manager when doing some research before the shoot I was on, but even $70 wasn't in the budget so I did md5 sums from the command line to make sure the data transferred onto the backup drives ok. I know R3DM does more than this and I would definitely purchase it if I was doing a paid gig.

Nicholas: Yes, of course, to be in demand you would need to, in the eyes of the director/2nd AD, "skate to where the puck is going to be".

I think my approach will be to help out on a few more low/no-budget projects and get more familiar with the equipment and environment and if I still enjoy it and have something to offer, try and scare up some paid gigs.

So if anyone in Sydney needs a hand with DIT/wrangler-type duties, send me a PM.

 

[Nelson Goforth]

Using R3D Manager

Using R3D Manager

I just did a job as "Data Wrangler" using the R3D Manager software.

I'd looked into running checksums myself, and thought perhaps I could put together a perl script to do the repetitive work, but when I read about R3D Mgr it seemed to do the job with a level of security that I could accept. The md5 checksums and the reports are critical, and the R3D Mgr makes it pretty effortless. I was copying from a Red Drive onto two 500GB backups and it worked like a charm.

My only comments would be (1) make a demo version available - I was hesitant to purchase without really knowing the software. Only recommendations by others settled me. And (2) we need better documentation - and not online (didn't have connectivity on the set). For some reason the initial checksum wasn't running automatically for me and there was no documentation to help me discover why. And while I understand what md5 is, I'm not sure most people would. There are a number of obscure choices in the software and the 'tooltip' instructional method is okay as quick reminder, but it doesn't replace a manual - even an electronic one.

But on the whole I was very well pleased with the software and it quietly sat there and did it's job.

Nelson

 

[Nelson Goforth]

.. but I will never trust sofware to do all the copying and backup.
it may be a good starting point.. but always double check.

How do you double check?

I'd thought early on about running md5 checksums from the command line, but for a lot of files that could be tedious (and therefore prone to operator error), and literally rewatching each shot (twice for two backups) seems counter-productive.

I have used R3D Manager and it does what I'd planned on (md5 checksum) and quite a bit more (reports with any errors, etc), but without the headache.

Of course I don't completely trust software either, so I watched samples of a number of shots from each backup drive. What's more, the md5 checksums (and R3D Mgr can do even more secure checksums) WILL tell you if you have digitally identical files... but they won't tell you if the original file was any good to begin with :biggrin:

Thanks,
Nelson