Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Copy Entire Contents Of Mag To Hd

I was thinking about starting a thread entitled "Digital Magazine, Oh How I Hate Thee, Let Me Count The Ways" but I'll just put it in here.

Red has gone to the trouble of creating a file naming system that tries not to create duplicate file names. This is good.

BUT....you CAN do it (get repeat file names that match exactly, a venal sin in a file based workflow) by pre-formatting your cards, or by switching from 4K to 2K for high speed work (been bit by that TWICE in the heat of production, and Force Reel scares me since I don't necessarily know what exactly is going on or where we are, so I've been switching to B then C camera designation when switching formats mid-job, but that's another post/thread).

ANYWAY....they've attempted to make a system that won't create dupe file names and are human readable. GOOD. BUT....then they create a file that is Somehow Important (somebody PLEASE explain to me what's in those two digital magazine profile files, and WHY WE NEED THEM!)....and every F(*(*&^*&^ing last one of them has the EXACT same file name.

And it gives you no clue what files are associated with it. Therefore you have to keep'em in same folder with the .R3D folder stuff all through the process, and if you have an "orphan" you are fuxored, since there is no way to tell where it went.

If you go by numbered Mags in folders, makes it tougher to traverse, find, verify you have two copies, etc.

What I'd like to see:

those files should be renamed to have a unique name. If they are created when card is formatted, OK - give it a name based on camera, date/time, etc. that is UNIQUE. Digimag_071129_(serial#/PIN)_4K_2398 or somesuch.

Within the file, right at the top of the header, in plain Engrish, should be the shots that it is associated with.

I hate to sound so ignorant on here, but REALLY - I don't understand what these files are doing for me. Somebody clue me in, PLEASE, because they are making my on-set data management life significantly more difficult.

And kee-riste, if it is causing issues for propellerhead me, woe be unto any Joe who rents and "just wants to shoot, ya know?"

Rant off.

-mike

PS - EDIT - I really LIKE the idea of putting all shots in one Big Bit Bucket on a drive - I can see what shots I do and don't have, and when working as Red Tech and digital data backup on some jobs (anybody else get clients bitching about two people on jobs for this "low cost" camera?), the ability to queue up a double copy (from CF to two separate drives, no seeks to shred transfer performance) and walk away (insert new workflow with AAsync or metric equivalent) is NICE - so I'd like to maintain that if possible.

When there's 8 cards on set and some are hot 2K and hot 4K and has that one been backed up, wait help reconfigure this, did that copy finish, is it all there, etc., is made NOTABLY more difficult if we MUST do Mag#(x) setups - which mag# was that? Oh wait, let me copy/find/paste, etc....bag that - I should just be able to LOOK and see if A009_C012_071124 is there or not.

OK. NOW I'm done.
 
offhollywood said:
Not sure why you say AAsync is "an impossibly slow level" - we are shooting a feature right now and making TWO copies of 130G per day off of 8G CF cards with AAsync - and it's going like butter -
Click to expand...


Thanks for that. I may well be doing something wrong, which is making AAsync struggle. How long does it take you to transfer 130GB with AAsync?
 
MikeCurtis said:
When there's 8 cards on set and some are hot 2K and hot 4K and has that one been backed up, wait help reconfigure this, did that copy finish, is it all there, etc., is made NOTABLY more difficult if we MUST do Mag#(x) setups - which mag# was that? Oh wait, let me copy/find/paste, etc....bag that - I should just be able to LOOK and see if A009_C012_071124 is there or not.
.
Click to expand...

That is so funny. I laughed for far too long, because I wake at night thinking about this.

The best solution I've come up with for this is a physical one. Have way more CF cards than we need (about 30), and have a storage case for blank cards. We take cards from there, and only from there, to the camera. As soon as the card is full it's copied to the MacBook Pro, and then goes into the Halfway House case. At night, the Halfway House case is completely copied to a couple of RAID 1s. After all shots are checked, the CF cards are wiped and returned to the first case.

Not simple, nor foolproof, but if there's no wiping on any given day, and we just shoot-copy-store one card at a time, there's hope.

Better ideas are always welcome.
 
For me as an "on set" guy, I like the idea of things separated into "mags". Maybe it's having stuff beaten into me when I was a film loader, but my mind just works in the world of mags. And when I get into the edit, I have my head organized in the chronologies of Mags.
 
Christopher - that was originally my plan, when the client said we'd probably shoot maybe 20 minutes of footage. I had 8 CF cards, so wasn't too concerned. But we ended up shooting 83 minutes of footage (I pointed out we could shoot 72fps at 2K, and oh boy, does that zip out the frames!), so I ended up cycling through all the cards more than twice. So it wasn't an option on this shoot.
 
ChristopherKenworthy said:
Could you elaborate on the Finder issues? You've started to me me paranoid, even though I've used the Finder to copy countless TBs of data without a single problem. Is the Finder notoriously rubbish at this? I've tried AAsync, but found that it slows the workflow to an impossibly slow level (even with a top of the range Mac.)

If I offload the CF, and watch the rushes every night and they play back OK, do I really need to checksum? What's the worst that can happen if I don't?
Click to expand...

I think the biggest danger with Finder is just slipping with the touchpad. But in addition, I have seen ghostly folders that appear and disappear on external drives. After that I plugged one of those little lights into my USB port, so that I could sleep.

All that said, I have never lost a file because of Finder.

I like terminal because its a little more deliberate and cp -v gives you a report as it's copying, when I'm copying in finder, I don't want to walk away while the blue bar is moving.

I don't think you need checksums really, after all, file size is a kind of checksum, just much less accurate. But if you are on a set spending possibly thousands on each setup, how can you not use every means at your disposal to insure that the data is copied securely.

IBloom
 
ChristopherKenworthy said:
Thanks for that. I may well be doing something wrong, which is making AAsync struggle. How long does it take you to transfer 130GB with AAsync?
Click to expand...

It depends on what mode you're running it in, disk performance, etc. Read through the documentation - AASync goes through a number of "phases", which include reading the source (CF card) in order to generate "database information" (which I'm assuming is basically a checksum), copying the files, then reading both the source and destination *again* to see if they match.

I ran some tests this morning, and saw times ranging from about 45s/Gb to 90s/Gb, depending on type of drive, which FW800 port, daisy-chain, etc. I've also seen times in excess of 100s/Gb with a drive that's normally fast - no idea what caused it. But in general, I find that AASync is just as fast as doing the reads/writes/compares from the command line.

It can be a bit challenging to keep up with the camera dept., when they're shooting a million takes ("Hey - it's digital!"), 'cause I'm essentially reading the card four times. But it's worth it, to not have to tell production that we need to re-set the lights/camera for a lost scene.

As for the Finder issues: me, I don't trust it for heavy file-slinging. And I have a vague memory of getting wildly faster copy times from the command line - but that may have been a couple OS versions ago. One trick: if the file size/count that Finder shows is stale, you can kick-start it by creating and deleting a folder underneath the tree you're looking at, which will force Finder to recalculate its totals.

mitch
 
UPDATE - talked to Stuart English - there is NO NEED to keep the digital magazine files. On P2, you need the other stuff, but NOT for Red.

Throwing it all the .RDC folders into a common folder is JUST FINE.
 
Please do NOT follow that advice. There's future software that might use the two .profile files in the root of the magazine. Just copy the complete magazine.
 
Magazine Profiles

Magazine Profiles

Oops. I stand corrected! (One of the great things at RED is we constantly have new ideas bubbling up.. )

At this time there is no NEED to carry those files over, but its sounds like it could be good practice to do so....for the sake of future post software.
 
I'm a Unix Programmer i do this for living. Let me know if you guys need some help in automating this process.
 
offhollywood said:
I can't say for certain that mac osx finder will ever fail you - but I have personally seen this ... and it hurts -

http://tomkarpik.com/articles/massive-data-loss-bug-in-leopard/

Not sure why you say AAsync is "an impossibly slow level" - we are shooting a feature right now and making TWO copies of 130G per day off of 8G CF cards with AAsync - and it's going like butter -

Do you really need checksum?

Well ... do you REALLY need insurance?

Not unless you have problems. Then it's pretty handy.
Click to expand...

I've had finder bomb on bulk copying of large files- not a fun time.
 
ChristopherKenworthy said:
Not simple, nor foolproof, but if there's no wiping on any given day, and we just shoot-copy-store one card at a time, there's hope.

Better ideas are always welcome.
Click to expand...

Nothing is foolproof to a sufficiently talented fool... not deleting amongst the chaos in the middle of a shooting day, no matter how smart you are, is a great idea christopher.
 
ibloom said:
Update: My currrent workflow is a program called AAsync that I discovered in the series of tubes. This new tool is really going to put me ahead of all my competition. On the west coast they haven't even heard of this shit.
Click to expand...

LoL. Ah yes the good old series of Tubes. "Its not a dump truck you can't just pile things in it. Its more like a series of tubes!"

Jarred Land said:
Nothing is foolproof to a sufficiently talented fool... not deleting amongst the chaos in the middle of a shooting day, no matter how smart you are, is a great idea christopher.
Click to expand...

Its what we all should look out for, talented fools truly are the bane of all existence.

I'm glad this thread started up, everything other than what I am saying now has been actually useful, but I will continue to rant uselessly anyway, enjoy.

It sounds like it might be a good idea to take 50 - 100 CF cards with you into battle, and fist full of red drives. Also I heard rumor that a Spanish film was recently shot on red, or at least they shot the film in Spain and they kept every CF card they used. It doesn't sound like such a bad idea really. The CF cards are 1/3 the cost of 35mm so you still come out ahead by a long way not to mention the tech is cheaper.
Mike you should demand all your clients do that from now on. Tells me and every one how it goes, you know, weather its a sound business stratagem.

Thought I should shift some paradigms... Don't you worry about work flow, let me worry about blank.

Lachlan
 
Hey, the Build_14_Ops_Guide_v2.0.3.pdf (latest as far as I can tell) says this:

"Note: You don’t need to copy the PIN.log, digital.magazine.profile and digi-
tal.magdynamic.profile. files, but will also do no harm if you do."

There's no clue there that we should actually be copying those for future features. So I haven't been, which makes me feel, um, silly is one way to put it. Should the Ops_Guide be updated?

Are there any current workflows/software/hacks that are using those at this point?
 
You must log in or register to reply here.
Back
Top