Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Brook's fast CF card reader

Andrae Palmer

Andrae Palmer

Well-known member
Joined
Sep 1, 2007
Messages
2,480
Reaction score
1
Points
38
Age
52
Location
Fort Lauderdale, FL
Website
www.facetmedia.com
On episode 044 of RED Centre, Brook Willard was speaking of his custom built CF card reader. I was intrigued by the report that it can read a 16GB card with checksums in 2 minutes. So I came on here and searched for a thread on this to see if it was brought up... I must be using the wrong search terms. Anyone have a link or info on this super fast CF card reader? Since R3D Data Manager and the Lexar FW800 is taking 12 minutes for a full 8GB card... I'm very interested in making the transfer faster.
 
I'll post details soon, I'm listening to the interview. :)

[edit: oh man, I was vague about the gear in the interview. I'll be more specific soon enough]
 
Brook Willard said:
I'll post details soon, I'm listening to the interview. :)

[edit: oh man, I was vague about the gear in the interview. I'll be more specific soon enough]
Click to expand...

Thanks Brook. Any info on how we can do Checksum with Finder will be appreciated - Lim
 
I can't really discuss how I was doing that... it was a bit of trickery that would be better left out.

These days I use R3D Data Manager. It's a bit slower than I'd wish for, but it's a great app that I definitely trust and can't recommend enough.

More details once I get a chance to write 'em up.
 
Ok great... i'm wiling to buy or solder... 12 minutes on a 8GB card is ridiculous.
 
It's sort of bittersweet to post some of the "tricks" of rapid data offloading [although there isn't much in this post]. Ultimately it's nothing too revolutionary or surprising, but here is a combination of parts I discovered after a lot of research, failed attempts and replacement parts. It's not the reader I described in the interview, but it's the system I'm currently using.

It's a bummer to post only because this will suddenly show up in everybody's DIT cart... with everybody claiming it as their own. The tricks some of us have been using for years will start to become the norm. It's like the first time some director tried to tell you about how awesome the RED was... after you'd been following it for 3 years. I should be charging for this info... :)

Here's a solution that's technically equivalent to what I have:

Atto H380 8-port external SAS/SATA II card

HighPoint Ext-MS-1M4S External SAS [SFF-8088] to SATA [L] fan cable

Addonics SATA CF Adapter

If you take one H380, one fan cable and 4 CF readers and build a clean power system for the CF readers, you'll end up with 4 full-bandwidth SATA II CF readers. If you're offloading to a fast enough RAID [My primary RAID 6 reads/writes about about 1GB per second], you'll be looking at 2-3 minutes to offload 4 CF cards simultaneously [via the Finder]. If you add in an application like R3D Data Manager, the speed will be dependent on the speed of your processors. It adds about 3-4 minutes to my 4-card simultaneous transfer time. ShotPut RED will be faster, but I've had enough bad transfers using that application [even with the "better/slower" check] that I no longer use it.

As I stated in the interview, moving data quickly is all about pipes. If you have one pipe to move data through in your system, you will be limited. If you have two [say, one pipe in and one pipe out], your speed just doubled. If you have one pipe in but five pipes out... you haven't improved anything. The trick is to create a "series of tubes" that gets as much data in and out of your system as possible. Never let a input or output idle unless there's no work to be done. If you've got one card reader but five cards, you're not getting the best out of your system. If you're hooking five card readers up to one pipe, you're just gonna clog it. If you've got four pipes in and one big pipe out... but your backup software only handles one card at a time... you're getting nothing out of the faster system. Figure out your workflow, think about where data needs to go and when... and pick the parts that will let you accomplish your goals for that job.

I have a drawer full of about a dozen PCI cards, a dozen CF readers, you name it. It's not just for backup... it's for making the best choices. What good is a SATA card if you need a FireWire card for the day? What good is the LTO-4 drive if you're not using it? There's always an ideal way to configure your gear to get it ready for a job and its specific requirements. Figure out your personal system for a job and configure your equipment to make the best of it during the prep. Learn what technologies speak to each other and which don't... figure out the fastest chipsets... buy the parts and test them. If they don't work, well, now you know. That four-figure-a-day equipment rental is there for a reason.

And when you think you've got something good... just remember, there's always a better/faster/safer way... :)

As more people start to build and use higher-end/more expensive DIT systems, I can only ask one favor: don't give it away. Just like you don't want to be that guy with the $150/day RED on craigslist... don't be the guy with the $80,000 DIT cart who charges $250 a day on equipment/kit/box rental just to get hired. It's not "just a computer"... it's your livelihood and mine. :)

I'm working with a few companies to build something of a "standard" DIT and digital loader cart for the future. As bigger shows now use the RED and RED's new cameras will have higher data rates and will require more equipment than before, I think that the industry will benefit from this sort of thing. People who are willing to build their own carts will always build their own... and rent them like an operator rents his Steadicam rig. But that'll only be about 20 people in the city. If you park my cart next to Kevin Stanley's or Michele deLorimier's or Chris Bruket's or Dane Brehm's... the differences will be astonishing. As more and more companies are building [or thinking of building] DIT/loader carts for the RED, I think it'll be a huge benefit to get everybody on the same page.

I don't expect to make a penny from it... but if union loaders can be trained on the same equipment across the city or country and every company has something relatively similar [with their own specific hardware or software onboard], everybody will benefit. You can rent a 435 from any rental house in the city and know what you'll get. But the RED loader and DIT game is all over the map right now. How's a loader supposed to learn a system if there are 20 different varieties out there? How's a DIT who's just starting out supposed to get a job that needs tens and tens of thousands of dollars worth of equipment if he has no access?

Just a thought. I think the market is going to stabilize in the near future... and everybody will benefit. And in the end, it'll be that much easier for a UPM to call around town for quotes and see that you really aren't trying to screw them on your "computer".

The new cameras are coming. People have accepted the RED. The game has changed. Let's take it seriously. :)
 
Last edited:
Hey Brook,

I just built a E-sata Multiple Card reader (utilizing a port multiplier instead of a x4 Multilane Ext mini-sas cable) for laptop use...

I used 3 Addonics Card readers and left the other two sata ports free for LEMO to E-sata Cables for Red Drives / Red Ram.
 
Brook,

Well said, I could not agree more about dropping the price. I helped setup a production who used a junior (no pay) to back up material instead of paying me to save money. The production is now facing corrupt files, dropped black frames (overheated readers) etc... Guess what that is going to cost them in post not only that but it also taints RED's name in such a small film community as ours in Sweden... But enough bitching.

In your interview you mention that you only copy files, do you not check there integrity? Although a checksum checks that the data is the same does this mean that the image is the same. This question might be a bit trivial, but I & the guys I work with always really on the fact that all the material has been watched checking for corruption (nearly non-existant), focus (more usual :-)) etc... etc...

regards

Marcus
 
More interesting is that Brook use on set 3cP - Cinematographer's Color Correction Program that I also use.

No mention that he would use any of "throttle type of profiles" on set that were "recommended" here at the forum from...:conehead:
 
Brook Willard said:
Here's a solution that's technically equivalent to what I have:

Atto H380 8-port external SAS/SATA II card

HighPoint Ext-MS-1M4S External SAS [SFF-8088] to SATA [L] fan cable

Addonics SATA CF Adapter

If you take one H380, one fan cable and 4 CF readers and build a clean power system for the CF readers, you'll end up with 4 full-bandwidth SATA II CF readers.
Click to expand...

Brook - been a while since I tried a config like this - and I haven't played with the Atto H380 yet - but correct me if I am wrong - as I remember it, you will have to manually EJECT the CF cards, not hot swap ? Not a big deal, just want to know if this is the case with this set up and/or the one you are using.
 
Brook Willard said:
As more people start to build and use higher-end/more expensive DIT systems, I can only ask one favor: don't give it away. Just like you don't want to be that guy with the $150/day RED on craigslist... don't be the guy with the $80,000 DIT cart who charges $250 a day on equipment/kit/box rental just to get hired. It's not "just a computer"... it's your livelihood and mine. :)
Click to expand...

Thanks Brook... anyone that sees your workflow will know you are a true professional. I compare it to doing title credits for movies/shows... you can hire a guy like Andrew Kramer from videocopilot and he'll do an amazing job or you can get a 12 year old to type up some fonts in FCP. The value is plainly evident.
 
Marcus Vasques Osorio said:
In your interview you mention that you only copy files, do you not check there integrity? Although a checksum checks that the data is the same does this mean that the image is the same.
Click to expand...

Yep, I check the integrity of every clip in at least 3 ways... usually 5. The most intensive check happens during color correction.

Sanjin Jukic said:
More interesting is that Brook use on set 3cP - Cinematographer's Color Correction Program that I also use.

No mention that he would use any of "throttle type of profiles" on set that were "recommended" here at the forum from...:conehead:
Click to expand...

I'm generating RSX files for camera settings [if I change the ASA, for example] and TrueLight Cubes for color correction.

Mark L. Pederson said:
Brook - been a while since I tried a config like this - and I haven't played with the Atto H380 yet - but correct me if I am wrong - as I remember it, you will have to manually EJECT the CF cards, not hot swap ? Not a big deal, just want to know if this is the case with this set up and/or the one you are using.
Click to expand...

Define hot swap for me in this particular case? I'm able to treat the CF cards in the same manner as I would in a normal FireWire CF reader. As of R3DDM version 5, it can auto-eject cards when configured as listed.
 
Brook Willard said:
If you take one H380, one fan cable and 4 CF readers and build a clean power system for the CF readers...
Click to expand...

Interesting point. I could never get the Addonics SATA CF readers to work reliably in my system and suspected for various reasons that it was a powwer issue.

Did you just give them their own power supply isolated from the system?
 
Is there anything similar, set up wise for Laptops? Or should i need to construct it myself from those MAGMA PCIe mobile slots?
 
It's sort of a DIY thing. The laptop only has one lane of PCI available to it. That means that even if your input [say, from multiple CF readers] is fast, the only output you will have available to you is over FW800. Your overall transfer speed will always be severely limited.
 
You must log in or register to reply here.
Back
Top