Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

BUILD IT, AND THEY WILL STORE AS MUCH FOOTAGE AS THEY CAN!

Von Thomas

Von Thomas

Well-known member
Joined
Jun 9, 2008
Messages
529
Reaction score
0
Points
0
Location
Los Angeles
Website
www.digitaltechnyc.com
As a DIT I battle to get a smooth and consistant flow of backing of RED data. It's a challenge because there is the option to shoot more. If you are working a show, and don't want to be OT more than 30 min after wrap, keep your 128GB SSD's down to 10 to 12%. This will allow your offloads to be quicker, and you well keep up with the flow of data. Now while this is my rule, I cannot always enforce it.

Why storage gets larger is beyond me. I can see if you're shooting solo, and you wait to download once to finish for the day, but on heavy production schedules with one or even two computers, keeping it light will work best.

I pity the fool that shoots to 80%.

Von
 
Von Thomas said:
As a DIT I battle to get a smooth and consistant flow of backing of RED data. It's a challenge because there is the option to shoot more. If you are working a show, and don't want to be OT more than 30 min after wrap, keep your 128GB SSD's down to 10 to 12%. This will allow your offloads to be quicker, and you well keep up with the flow of data. Now while this is my rule, I cannot always enforce it.

Why storage gets larger is beyond me. I can see if you're shooting solo, and you wait to download once to finish for the day, but on heavy production schedules with one or even two computers, keeping it light will work best.

I pity the fool that shoots to 80%.

Von
Click to expand...

I agree in principle.

I think that Epic's data rates will make shooting to 80% more common though ... so the new struggle is to just find some way to solve the problem of off loading faster. From experience the SSD's can, over eSATA, sustain VERY fast rates. (IIRC I saw over 170MBps sustained) I only pulled about 40GB, but it took less than 5 minutes to a SAS RAID 0 array. Extrapolating that would let you offload a full 128GB mag in less than 15 minutes.

On my system at home I saw a smidge under 80MBps over firewire 800. (Basically it maxed the FW bus) That adds up to a bit less than 26 minutes per 128GB Mag. That's more painful ... but its still faster than real time for Red One and the more compressed rates for Epic.

So ... I think 10-12% is too low. I'd say 25%. I think you could work with up to 60% on Red One, though that would be unwise its technically feasible.

Am I missing something? Have I slightly lost my mind about data rates off the 128 SSD?
 
My apologies, I should have qualified this by adding two things, I'm not doing finder copies, I exclusively use R3D Data Manager (checksum takes longer), plus I'm not on my usual Light Iron Outpost cart, I'm on an independent low budget feature, and working off MacBook Pro. I've found as much I need a heavier cart solution, some clients will not pay for it, so with that, I still stand on my original, but offer. My experience with RED MX and 16GB cards using eSata as about 10 to 12 minutes per card, on a tricked out MacPro to SAS RAID 5.

Von
 
Last edited:
So. your solution to long offload times is to shoot less footage? Brilliant. Not sure I quite follow here.

It isn't difficult to achieve consistant offload times that are measurable. Test, and plan/equip accordingly. . .
 
His solution is to change out media more often, they way people should. It's smarter in the long run, and can save production time and money in the back end, especially with RED raid's, and multi cam. Changing out every hour on the hour is the the best thing you can do for the production, unfortunately Von Thomas is right when he says sometimes nobody listens to you. Priority for some producers/directors/clients etc don't have the foresight to see to the end of the day and it makes the loader and or DIT look bad at the end of the day with final backups, when you have a near full drive of 100+ gigs to back up redundantly. Longer if you do checksums.

And of course sometimes it can't be avoided at all. not enough media is also a problem. I've had to stay after wrap 3+ hours because they only had 1 red drive, and 2 16 gb CF cards. They also had to do a 1 hour timelapse, twice. Filled up most of the RED drive. Location was kicking as out and I was still in the middle of a backup and fought to stay where I was. You win/lose your battles, I explained s we needed more media and it's gonna be awhile after wrap to finish the backups. But they don't quite get it sometimes until it's actually happening.

Best way to help yourself is to change out media often, and try to make sure you have ample media to boot.
 
andrewhake said:
So. your solution to long offload times is to shoot less footage? Brilliant. Not sure I quite follow here.

It isn't difficult to achieve consistant offload times that are measurable. Test, and plan/equip accordingly. . .
Click to expand...

Not only having a smaller file sizes of course helps, but drives loose efficiency at some point when they get filled up. I have 30TB of online storage running on my home system or should i say i have 30TB available with 5 hot swappable bays each containing a 3TB SATA hard drive. I NEVER fill those drives beyond 80% each drive.....When i have filled them to 90% or more, the speed at which they transfer or data is accessed falls way off.

I have also experienced RED raid not able to keep up with recording and having numerous errors when beyond 80%. So, i think this is a no brainer when you understand the limiations of hard drives. Not sure with SSD's---but once you have experienced it, you'll NEVER forget it. So that's the OTHER SIDE of filling a drive to near capacity....not only have you lowered your transfer rates, but you've now exposed yourself to possibly not being able to record data.
 
Using a laptop (alone) to data manage any feature is like shooting yourself in the foot.

You need a Mac pro, a Customac pro, or even (gasp!) a PC with three esata ports minimum in order to move data fast enough.

Call me, I can build you a good tower for half the cost of a MacBook pro.

128gb SSD murders the red raid drives.

Dan Kanes - IATSE Local 600 DIT
 
I agree 100 percent. I like to change cards at least once an hour on a DIT supported set. Much safer for sure.
 
Dan,
with what's available now....can you give an idea of HOW you would NOW - TODAY configure a mid range system. What would interest me: NOT something i take on features or anywhere, but ONLY work on my stock and in house documentaries.....What would you build and what kind of price? Features; Cards, Ram etc....

thanks
 
Dan Kanes said:
Using a laptop (alone) to data manage any feature is like shooting yourself in the foot.

You need a Mac pro, a Customac pro, or even (gasp!) a PC with three esata ports minimum in order to move data fast enough.

Call me, I can build you a good tower for half the cost of a MacBook pro.

128gb SSD murders the red raid drives.

Dan Kanes - IATSE Local 600 DIT
Click to expand...

You don't need a MacPro, but it is preferred. I use a laptop for about 30% of the jobs I do, as clients does not want to pay to bring out the big guns. Also, because of my workflow, I normally walk off set within 30 mins of wrap, with all footage backed up at least twice.

Von
 
It depends on what kind of project you're working on. I did a show that routinely shot more than 1TB of footage in less than 12 hours, with 3 cameras. Each take of the scene they were doing was 20-30GB in size per camera. There's no way to keep up at that point. That day we shot 13 hours of footage in a 10.5 hour day.

I would have been sunk with a macbook pro. No matter what my workflow was.

Tim
 
UPDATE - Tuesday night I had a lot of work to back up at wrap, as camera was 200 yards and uphill from my workstation. Rather than hold up crew for last backup (via R3D Data Manager), I did a quick non-checksum to have files in two place, and will do a proper BU on Friday. 50GB took 10 minutes from SSD to MacBook Pro.

Von
 
Single camera indie flick can easily be handled on a macbook pro. It is nicer to have a good tower setup though!
 
Von Thomas said:
UPDATE - Tuesday night I had a lot of work to back up at wrap, as camera was 200 yards and uphill from my workstation. Rather than hold up crew for last backup (via R3D Data Manager), I did a quick non-checksum to have files in two place, and will do a proper BU on Friday. 50GB took 10 minutes from SSD to MacBook Pro.

Von
Click to expand...

What drive is the Macbook dumping to?
 
thunderbolt will solve a lot of bandwidth issues. 1-2 macbook pro's will be able to start handling more workload again, even an imac with dual TB ports won't be a bad idea. on set transcoding though, I'd still bring out the big tower, but things could change if these TB pci expansion boxes work well.
 
This is why I think 64Gb SSD are perfect for Epic. 256GB SSD if you want to overcrank or something special, but 64GB for normal shooting.

I have done lots of commercials that wouldn't pay for anything more than a laptop and shot 400-500GB of footage per day. With the correct tools a laptop can handle backing that up. The key is to get mags from camera as soon as you finish downloading. No down time between downloads.

The thought of someone shooting t0 80% on a Red Drive scares me. 10% used for a 320GB and 5% for a 640% is all I like to see.


Mahalo,
Dusty
 
Dustin Cross said:
This is why I think 64Gb SSD are perfect for Epic. 256GB SSD if you want to overcrank or something special, but 64GB for normal shooting.

I have done lots of commercials that wouldn't pay for anything more than a laptop and shot 400-500GB of footage per day. With the correct tools a laptop can handle backing that up. The key is to get mags from camera as soon as you finish downloading. No down time between downloads.

The thought of someone shooting t0 80% on a Red Drive scares me. 10% used for a 320GB and 5% for a 640% is all I like to see.


Mahalo,
Dusty
Click to expand...

We had a shoot where they had to do a really long timelapsed shot. corporate commercial, filled up 250 gb on a red drive. pre SSD days. and there was more shooting to do during the day, and they wanted 3 backups not two, and they only wanted to pay for the laptop combo.

3+ hours after wrap wasn't fun.

also I've been on sets where they absolutely refused to change out mags as much as I asked them too. it was always "there isn't enough time" slow formats on the raids, the time to eject and mount the new one, was all too slow for them. it got to the point where I was forced to take the drive at the end of the day and do backups at the hotel cause there was not staying at location. it was hard enough trying to get them to change out mags at lunch... It wasn't a fun shoot...
 
You must log in or register to reply here.
Back
Top