Click here to go to the first RED TEAM post in this thread.   Thread: Ballooning Data Rates and Crippling Workflow

Reply to Thread
Page 1 of 3 123 LastLast
Results 1 to 10 of 29
  1. #1 Ballooning Data Rates and Crippling Workflow 
    Junior Member dev.doyle's Avatar
    Join Date
    Nov 2008
    Posts
    16
    I was just on a job with two MX cameras shooting 4KHD RC42. The nature of the show required long takes, which naturally inflated file size. Running on a 2.8 GHZ MacBook Pro w/ a FW800 express card adapter I had three destination drives. Two were connected directly to the FW800 adapter, and the third was daisy chained behind one of those direct drives. The RED RAIDs were being connected directly to the MacBook's FW800 port.

    Being a diligent data manager I began using R3D Data Manager to ingest the footage, but the combination of daisy chained drives and steep shooting ratio I began to get backed up (on a side note - R3D used to be so much faster, two upgrades ago it suddenly took 30% longer to dump - what happened?). Anyways, when pushed to the point of running out of media to hand the ACs I switched over to ShotPut Pro which allowed me to make up some time (I'm well aware of the pitfalls of ShotPut, and whenever using it I'm sure to watch all the footage to verify integrity).

    All day it was a struggle to catch up, and at end of day required an additional 3 hours to transfer the rest of the footage. The question I pose is this: with data rates growing every build (which is great for image quality) what's a data manager to do? I'd use eSata for the destination drives but if you're limited to FW800 out of the RED RAID isn't it a moot point? Daisy chaining with eSata makes sense I suppose to boost speed between the two...or does it? Can anyone shed any light on that?

    What are people doing to cope with these ballooning data rates? I don't know about you guys, but after a 15 hour day on location after 4 company moves the last thing I wanted to do was hang around for an additional 3 hours. Transferring the footage safely and verifying is priority one, but surely something can be done to make this faster. Any gear or suggestions anyone has would be greatly appreciated!
    Cinematographer/AC - devindoyle.net
    Reply With Quote  
     

  2.   Click here to go to the next RED TEAM post in this thread.
  #2  
    Red Leader Jannard's Avatar
    Join Date
    Dec 2006
    Posts
    8,151
    Quote Originally Posted by dev.doyle View Post
    I was just on a job with two MX cameras shooting 4KHD RC42. The nature of the show required long takes, which naturally inflated file size. Running on a 2.8 GHZ MacBook Pro w/ a FW800 express card adapter I had three destination drives. Two were connected directly to the FW800 adapter, and the third was daisy chained behind one of those direct drives. The RED RAIDs were being connected directly to the MacBook's FW800 port.

    Being a diligent data manager I began using R3D Data Manager to ingest the footage, but the combination of daisy chained drives and steep shooting ratio I began to get backed up (on a side note - R3D used to be so much faster, two upgrades ago it suddenly took 30% longer to dump - what happened?). Anyways, when pushed to the point of running out of media to hand the ACs I switched over to ShotPut Pro which allowed me to make up some time (I'm well aware of the pitfalls of ShotPut, and whenever using it I'm sure to watch all the footage to verify integrity).

    All day it was a struggle to catch up, and at end of day required an additional 3 hours to transfer the rest of the footage. The question I pose is this: with data rates growing every build (which is great for image quality) what's a data manager to do? I'd use eSata for the destination drives but if you're limited to FW800 out of the RED RAID isn't it a moot point? Daisy chaining with eSata makes sense I suppose to boost speed between the two...or does it? Can anyone shed any light on that?

    What are people doing to cope with these ballooning data rates? I don't know about you guys, but after a 15 hour day on location after 4 company moves the last thing I wanted to do was hang around for an additional 3 hours. Transferring the footage safely and verifying is priority one, but surely something can be done to make this faster. Any gear or suggestions anyone has would be greatly appreciated!
    Be happy you aren't shooting uncompressed?

    Sorry... couldn't help myself. Someone qualified needs to help with the real answer.

    Jim
    "The camera is arguably one of the most important of all inventions… it is the single tool that has the ability to stop time, record history, generate art, tell stories, and communicate messages that transcend language like nothing else ever conceived."

    "Everything in life changes... including our camera specs and delivery dates..."

    We reserve the right to refuse service to anyone with a bad attitude.
    Reply With Quote  
     

  3.   This is the last RED TEAM post in this thread.   #3  
    Red Team Deanan's Avatar
    Join Date
    Jan 2007
    Posts
    3,846
    Epic/Scarlet will have built in checksums for each frame so it'll be easier to verify after a copy rather than during the copy.

    We to often see productions go out with much less media than they really need. Sometimes it helps if you can get safety media as part of the rental that you open up only if you are in an emergency.

    The other recommendation is to double up the backups. Get an older laptop and do the backups in parallel so you can turn around faster.
    Reply With Quote  
     

  4. #4  
    Junior Member
    Join Date
    Feb 2010
    Posts
    11
    One of the reasons I dislike this job - 19+ hour days on two camera shoots.

    On several occasions we've shot in excess of 400GB per day on commercials. On a two day shoot, we once did 830GB in just two days! Ok, now, if this was common, the solution would obviously be higher end hardware, such as a Mac Pro / RAID setup. But, when clients normally only shoot 60-150GB per day, it's difficult / nigh impossible to expect them to pay for a "real" downloading station 100% of the time. It always ends up being a laptop for budget reasons. I wish I had the answer, the OT is sometimes nice. But, what would be even nicer is to have a normal work day, go to bed at a reasonable time, and earn just a little less money. Or, own your own drives, and get a second day to actually replicate them, instead of doing it all night long. Haven't yet decided to go all out on a downloading system yet, but I probably will to save my sanity on the next 2 camera job. Again, if you're doing episodic television or features, you gotta have that stuff. But for many of us, it's only an occasional need.
    Last edited by Brian Wells; 04-11-2010 at 08:56 PM.
    Reply With Quote  
     

  5. #5  
    For professional level 2+ camera shoots a tower with at least two available busses (1FW800 and 2+ esata) is imperative. You can handle a one camera shoot that is primarily CF card based with a laptop but beyond that you seriously need to build yourself a proper data management cart, ideally with some sort of fast RAID. It's amazing how fast FW800 can be when you free up the bus to just read. That said we're really looking forward to trying the new Red Station card reader.
    "All art is deception."

    My DP reel...
    http://www.evingrantdp.com
    http://www.YouTube.com/evingrant
    360º Cinematography and camera rigs...
    http://www.360dop.com
    Reply With Quote  
     

  6. #6  
    Digital FX Greg M's Avatar
    Join Date
    Dec 2006
    Location
    Louisiana
    Posts
    4,835
    As Evin said you need a faster rig. MacBooks are not appropriate for that much data, nor are they safe. Build a professional rig or don't take on the larger jobs, it's as simple as that.
    Attached Images

    www.digitalfx.tv


    Twitter
    Instagram

    RED One #83
    RED CF Weapon 8K S35 Helium
    RED CF Weapon 8K S35 Helium
    RED CF Weapon 8K VV Dragon coming soon
    Reply With Quote  
     

  7. #7  
    REDuser Sponsor Brook Willard's Avatar
    Join Date
    Dec 2006
    Location
    Burbank, CA
    Posts
    5,318
    Quote Originally Posted by Deanan View Post
    Epic/Scarlet will have built in checksums for each frame so it'll be easier to verify after a copy rather than during the copy.
    That's fantastic news.

    As for dev, the MacBook Pro is not a tool that will have an extended future for data handling on higher-end production. It just sounds like you had the wrong tools for the job.

    When we're shooting 3-7+ cameras on a movie or TV show, it requires a certain amount of equipment that most people do not own. There are carts out there that you can rent [generally $750-1,250 per day] that have the necessary data throughput to enable data transfers at the speed necessary for larger productions. You can build one yourself, but the cost of entry is very high.

    Just as an example, here's a picture of the old cart I built two years ago for Southland:



    My current cart is roughly twice as fast with many times more storage. Keep in mind that this sort of tool is designed as a DIT cart - not as a loader's cart - so its primary function is accurate color correction with data handling falling into the background. Needless to say, one has to have some very powerful equipment to turn a task that would take hours on a laptop into a task that takes minutes in the background of a tower.

    Get the right tools for the job! If it's taking more than 15 minutes after wrap [or so, depending on certain situations], somebody messed up.
    Last edited by Brook Willard; 04-11-2010 at 09:30 PM.
    Reply With Quote  
     

  8. #8  
    what cart is that digitalfx?
    Reply With Quote  
     

  9. #9  
    Digital FX Greg M's Avatar
    Join Date
    Dec 2006
    Location
    Louisiana
    Posts
    4,835
    They are both carts we built for our DIT's

    www.digitalfx.tv


    Twitter
    Instagram

    RED One #83
    RED CF Weapon 8K S35 Helium
    RED CF Weapon 8K S35 Helium
    RED CF Weapon 8K VV Dragon coming soon
    Reply With Quote  
     

  10. #10  
    Senior Member
    Join Date
    Mar 2009
    Location
    Burbank, Ca
    Posts
    605
    Just got off a 14 hour data-tech day here in beautiful downtown Burbank...

    Basically I try to never let the RedRaid drives get over 20% full, so the readout on back of the camera never goes under 80%. A lot of times I end up taking the drives home with me and either taking them back to the set the next day, or if it's the last day of the shoot, taking them back to the rental house the next day myself. Couldn't do that tonight though, as the owners are on dawn patrol for Vegas first light tomorrow...
    Reply With Quote  
     

Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts