Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Cards n Slots

Christopher Barrett

Christopher Barrett

Well-known member
Joined
Sep 30, 2010
Messages
1,518
Reaction score
0
Points
0
Location
Chicago
Website
christopherbarrett.net
In originally organizing my Mac Pro for Resolve I've setup my PCI cards as follows...

Slot 1: ATI 5870 that came with my tower drivin the GUI
Slot 2: Nvidia Quadro 4000 for Resolve
Slot 3: Red Rocket
Slot4: Decklink HD 3 Extreme 3d

Now, my Rocket works ok in the 4x #3 slot but playback is a little jittery in RCX where it used to be silky smooth. Can I swap it with the Nvidia or does that card require a 16x slot? Or any advice on a better way to configure this? My next step would be an expander chassis... and it WOULD be nice to be able to do eSATA.... yada yada.

Thanks,
CB
 
Last edited:
video card def needs a 16x slot. your ONLY choice is the pci e expansion chasis. or wait and hope new mac pros have more slots/lanes. and with a gpu and a rocket sharing a 16x slot. perfect rcx playback is still questionable
 
I would actually go with the following order, unless you get a PCIe expander.

1> ATI card - it's dual width, so you're screwed here anyway
2> RED Rocket
3> Decklink card
4> Quadro 4000

The Decklink only needs a 4X slot. The Rocket and Quadro card can both benefit by being in the X16 slot-2, however I have found there to be a larger bottleneck with the Rocket in in one of the 4X slots. Look at it this way, the Quadro 4000 is a PCIe 2.0 card, so if you put it in Slot-4, it will still be capable of 2GB/s peak bandwidth. The RED Rocket is a X8 PCIe 1.1 card, so its bandwidth tops out at 2GB/s in the X16 slot. However, if you put the Rocket in one of the x4 slots, you cut its peak bandwidth in half to 1GB/s. Best to keep the Rocket in slot-2 so it doesn't become the bottleneck. Bandwidth on that X4 2.0 slot isn't going to be a significant bottleneck for your Quadro 4000 doing CUDA calcs. The bottleneck there is the the calculation pipeline itself.

My current config is the same as what I just listed above, however I have a Quadro 4000 in slot-1 instead of the ATI card (two Quadros in the system). This way, I get full acceleration in my other OpenGL and CUDA apps -- Adobe CS5.5, etc.. I have a PCIe expansion chassis, but have it disconnected as it was giving me some fits with a third GPU installed there. The expander took up slot-2 and I put the Rocket inside it along with a third Quadro 4000 and a SAS/SATA card. I was getting crashes / lockups and the OSX crash log was showing the nVidia drivers as the main culprit. So much more stable after pulling the expander and just dropping the Rocket in slot-2. Works for now, but I have to power down and swap my second Quadro card with my ATTO SAS card once every 7 to 10 days or so, then I can connect my LTO drive and run some backups... Not ideal, but that's faster than trying to back up to another system over the network.

I'm also eagerly awaiting the new Mac Pros, but I'm convinced we'll see FEWER slots this time around. I'm expecting a massive re-design with a rack-mountable enclosure that is more sleek and compact. I'm expecting Apple to switch to 2.5" internal bays or bays that can hold 1 3.5" drive or 2 2.5" drives. Expecting it to have a slimline optical drive or even no internal optical drive at all. For various reasons, I'm expecting the new Mac Pro systems to only have 2 or 3 slots. We're starting to see that with the latest PC workstations as well as we shift to PCIe 3.0. Motherboard manufacturers, except for a couple who build the larger server boards, are taking all our slots away.
 
Jeff, which expander do you have. I have the Cubix and RR and 285 in it along with my decklink. I'm going to swap my decklink into the tower and put my esata card in the Cubix to see if that gives me any better performance in Resolve.
 
Jeff Kilgroe said:
I have a PCIe expansion chassis, but have it disconnected as it was giving me some fits with a third GPU installed there. The expander took up slot-2 and I put the Rocket inside it along with a third Quadro 4000 and a SAS/SATA card. I was getting crashes / lockups and the OSX crash log was showing the nVidia drivers as the main culprit. So much more stable after pulling the expander and just dropping the Rocket in slot-2.
Click to expand...

Thats a bummer about your expansion chassis. I have the Cubix Xpander Pro 4, with 2x GTX286s, RR, and eSata. Tower has Quadro 4000 GUI, Cubix host adapter, R680, and BM card. I've never had any issues with crashes and performance with Resolve is stellar. All drivers are current, OSX is current. 2009 MacPro w/24gb.

Are you on a newer MacPro?

I'm always tempted by shiny new stuff. But my system is running so well right now, I am afraid of touching it. And if there are issues with the Cubix box on newer MacPros, I'll probably keep this config until it dies. I love having all those slots.
 
No performance hit with the Quadro in Slot-4. Or if there is, it's negligible. Much preferred over the Rocket running at half-speed in an X4 slot.

I should have clarified, my cubix xpander (the Desktop Pro 2 w/ 4 single-width slots) works great. Or at least it's great, other than the stupidly annoying fan. WTF were they thinking? The issue is when using a third Quadro card in the system. I can duplicate the same instability and crashes on this system by pulling the Decklink card and installing the third quadro in its place inside the tower. Issue is with the nVidia drivers and/or CUDA extensions on the Mac. It all runs great if I boot into Windows.

System is a 2009 Mac Pro, 2.93GHz 8-core w/ 24GB RAM.
 
Jeff Kilgroe said:
No performance hit with the Quadro in Slot-4. Or if there is, it's negligible. Much preferred over the Rocket running at half-speed in an X4 slot.

I should have clarified, my cubix xpander (the Desktop Pro 2 w/ 4 single-width slots) works great. Or at least it's great, other than the stupidly annoying fan. WTF were they thinking? The issue is when using a third Quadro card in the system. I can duplicate the same instability and crashes on this system by pulling the Decklink card and installing the third quadro in its place inside the tower. Issue is with the nVidia drivers and/or CUDA extensions on the Mac. It all runs great if I boot into Windows.

System is a 2009 Mac Pro, 2.93GHz 8-core w/ 24GB RAM.
Click to expand...

have you tried your gui card being something else and running 2-3 quadros in the expander? hoping the conflict is the quadro card being the gui too, that can help root out a lot of issues. though kind of sucks if u wanna multi task stuff like adobe for cuda acceleration...
 
Hi Jeff,

There were some improvements for running with 3 Q4000s in V8. If you saw those with V7.x, you should try again with V8.

Regards,
Rohit
 
Jeff Kilgroe said:
I would actually go with the following order, unless you get a PCIe expander.

1> ATI card - it's dual width, so you're screwed here anyway
2> RED Rocket
3> Decklink card
4> Quadro 4000

The Decklink only needs a 4X slot. The Rocket and Quadro card can both benefit by being in the X16 slot-2, however I have found there to be a larger bottleneck with the Rocket in in one of the 4X slots. Look at it this way, the Quadro 4000 is a PCIe 2.0 card, so if you put it in Slot-4, it will still be capable of 2GB/s peak bandwidth. The RED Rocket is a X8 PCIe 1.1 card, so its bandwidth tops out at 2GB/s in the X16 slot. However, if you put the Rocket in one of the x4 slots, you cut its peak bandwidth in half to 1GB/s. Best to keep the Rocket in slot-2 so it doesn't become the bottleneck. Bandwidth on that X4 2.0 slot isn't going to be a significant bottleneck for your Quadro 4000 doing CUDA calcs. The bottleneck there is the the calculation pipeline itself.

My current config is the same as what I just listed above, however I have a Quadro 4000 in slot-1 instead of the ATI card (two Quadros in the system). This way, I get full acceleration in my other OpenGL and CUDA apps -- Adobe CS5.5, etc.. I have a PCIe expansion chassis, but have it disconnected as it was giving me some fits with a third GPU installed there. The expander took up slot-2 and I put the Rocket inside it along with a third Quadro 4000 and a SAS/SATA card. I was getting crashes / lockups and the OSX crash log was showing the nVidia drivers as the main culprit. So much more stable after pulling the expander and just dropping the Rocket in slot-2. Works for now, but I have to power down and swap my second Quadro card with my ATTO SAS card once every 7 to 10 days or so, then I can connect my LTO drive and run some backups... Not ideal, but that's faster than trying to back up to another system over the network.

I'm also eagerly awaiting the new Mac Pros, but I'm convinced we'll see FEWER slots this time around. I'm expecting a massive re-design with a rack-mountable enclosure that is more sleek and compact. I'm expecting Apple to switch to 2.5" internal bays or bays that can hold 1 3.5" drive or 2 2.5" drives. Expecting it to have a slimline optical drive or even no internal optical drive at all. For various reasons, I'm expecting the new Mac Pro systems to only have 2 or 3 slots. We're starting to see that with the latest PC workstations as well as we shift to PCIe 3.0. Motherboard manufacturers, except for a couple who build the larger server boards, are taking all our slots away.
Click to expand...

Jeff, i love it that you know this stuff.

I love it even more that you're willing to take the time to share the knowledge. Thanks.
 
Tom.Wong said:
have you tried your gui card being something else and running 2-3 quadros in the expander? hoping the conflict is the quadro card being the gui too, that can help root out a lot of issues. though kind of sucks if u wanna multi task stuff like adobe for cuda acceleration...
Click to expand...

I haven't really given it the time... I'm using one Quadro 4000 for the primary so I have CUDA in CS5 And other apps. It actually replaced a GTX285. The system started out with the GTX card, then I bought a Quadro 4000 for the GPU to use in Resolve. Had to pull dodgy power from the lower optical bay... I pulled the GTX and ordered the Cubix with 3 Quadro 4000 cards. Some of you might recall the picture I posted of the 3 Quadros. ;) One of those was to replace the ATI card in another Mac, but two (and the cubix) were for this Resolve system.

Peter or Rohit warned me about using 3 or more GPUs with Resolve at the time. But it turns out I was having troubles out of Resolve too. I did minimal troubleshooting and had the system running fine with 2 Quadros. I was sacrificing the 3rd Quadro, the expander and my ATTO card... But figured I'd try it all again when Resolve 8 and OSX 10.6.8 and updated CUDA all showed up. Those things are all here now, but I haven't worked with it. I'm not even in town at the moment, but I'll work with it again next week when I'm back and settled.
 
Jeff Kilgroe said:
I should have clarified, my cubix xpander (the Desktop Pro 2 w/ 4 single-width slots) works great. Or at least it's great, other than the stupidly annoying fan. WTF were they thinking? .
Click to expand...

Hey Jeff, when did you get your Cubix? I purchased mine in January and had the same problem of a jet engine sound under my desk every time I turned on my system. At NAB I spoke to them about it, and they said they had a quieter fan, and they agreed to swapped mine out. Now its about as loud as the H680 card. Not quie, but a LOT more livable. :thumbup1:
 
I just loaded this configuration last week and it's been a dream so far running Resolve, FCP and CS5.5.
2009 2.93 Nehalem 16GIG (could use a bit more)
Slot 1: GT120
Slot 2: Cubix Desktop 4
Slot 3: eSata card
Slot 4: RAID

CUBIX
Slot 1: Quadro 4000
Slot 2: Quadro 4000
Slot 3: Red Rocket
Slot 4: BM Extreme 3D

Really stable. And yes, the fan is really loud, but it's all tucked away in a machine room. I still hear it though!

I'm pretty happy Apple forced me away to better platforms. I'm discovering a whole new world out there!
 
I'm now running with this config as of the last 2 hours:

Slot 1: Quadro 4000
Slot 2: Cubix Xpander
Slot 3: BMD Decklink Extreme 3D+
Slot 4: ATTO H680

Cubix 1: RED Rocket
Cubix 2: Quadro 4000
Cubix 3: Quadro 4000
Cubix 4: Empty -- and this thing has to be maxed out!

So far, so good.

Now I need to figure out the fan issue. Freaking loud. Waiting for someone at Cubix to call me back... But I've found a few fans that should be pretty quiet and still move enough air, might order one or two and try them out.
 
Jeff Kilgroe said:
I'm now running with this config as of the last 2 hours:

Slot 1: Quadro 4000
Slot 2: Cubix Xpander
Slot 3: BMD Decklink Extreme 3D+
Slot 4: ATTO H680

Cubix 1: RED Rocket
Cubix 2: Quadro 4000
Cubix 3: Quadro 4000
Cubix 4: Empty -- and this thing has to be maxed out!

So far, so good.

Now I need to figure out the fan issue. Freaking loud. Waiting for someone at Cubix to call me back... But I've found a few fans that should be pretty quiet and still move enough air, might order one or two and try them out.
Click to expand...
Jeff is this config working for you?
We just ordered the Cubix 4 and would like to use your config as first try.
Only thing diff is that were putting an Avid Nitris DX host card in the mix.
Just not sure where.
Also an ATTO R380 instead of H680 and maybe an Apple Fibre 4GB card.

Your thoughts?
 
This config is working great. If trying to use the Quadro cards to their fullest as well as the Rocket card, there would be a significant bottleneck. However, it seems to work just fine with Resolve, performance is good. I'm actually going to pull one of the Quadros out of the Cubix though and put it into another Mac Pro that needs CUDA acceleration, but that's another story.

Still haven't found a good replacement fan for my Cubix... It's freaking loud. Nowhere near as loud as my HP LTO tape units, but still way louder than any "desktop" product should ever be.

When considering your slot placements, just think of the Cubix as having 4 x PCIe 2.0 X4 slots, but it can throttle up the bandwidth to one or more cards when others are not in use. So when I'm in REDCINE-X, I can get the full bandwidth of the Rocket since the Quadros are not being used. In Resolve, I was worried there would be a bottleneck, and I'm sure there is, but it's still faster with the second Quadro in the Cubix than without as I start adding more nodes.

The 4G FC card is your lowest-bandwidth card -- only takes up two lanes of PCIe bandwidth. Actually, if you have the latest revision of those cards, the single 4G port model is a PCIe 2.0 X1 card, so it's a good candidate for placement in the Cubix, other cards won't have to worry about it for bandwidth competition. As for the Nitris, not sure where it would work best. You may ultimately have to juggle some cards around. A Quadro 4000 also works great in Slot-4 within a Mac Pro. You may need to keep the R380 in the tower just to ensure immediate connectivity of any RAID you have attached.
 
Jeff Kilgroe said:
This config is working great. If trying to use the Quadro cards to their fullest as well as the Rocket card, there would be a significant bottleneck. However, it seems to work just fine with Resolve, performance is good. I'm actually going to pull one of the Quadros out of the Cubix though and put it into another Mac Pro that needs CUDA acceleration, but that's another story.
Click to expand...
Its looking more like a single Quadro 4000 for us here. And yes same thoughts of putting the other Quadros in other systems.

Still haven't found a good replacement fan for my Cubix... It's freaking loud. Nowhere near as loud as my HP LTO tape units, but still way louder than any "desktop" product should ever be.
Click to expand...
Where this unit is going (Production Studio) is not exactly the best place for audio either. Its freaking loud in there with the over-sized air unit and other machines from Protools, XDCAM decks, BetaSP decks etc...

When considering your slot placements, just think of the Cubix as having 4 x PCIe 2.0 X4 slots, but it can throttle up the bandwidth to one or more cards when others are not in use. So when I'm in REDCINE-X, I can get the full bandwidth of the Rocket since the Quadros are not being used. In Resolve, I was worried there would be a bottleneck, and I'm sure there is, but it's still faster with the second Quadro in the Cubix than without as I start adding more nodes.
Click to expand...
So its not 4 16x PCI slots? I guess I didnt read that right. Ultimately we need the Rocket to be at its potential and the growth on the CUDA part. And of course the Avid host card to work, but what about the DeckLink HD? This can be placed in a lower slot right?

The 4G FC card is your lowest-bandwidth card -- only takes up two lanes of PCIe bandwidth. Actually, if you have the latest revision of those cards, the single 4G port model is a PCIe 2.0 X1 card, so it's a good candidate for placement in the Cubix, other cards won't have to worry about it for bandwidth competition. As for the Nitris, not sure where it would work best. You may ultimately have to juggle some cards around. A Quadro 4000 also works great in Slot-4 within a Mac Pro. You may need to keep the R380 in the tower just to ensure immediate connectivity of any RAID you have attached.
Click to expand...
We might not have the FC due to the ATTO R380. Its getting to the point that the more HD space we have the more clutter occurs. Were going to be more efficient with this by not dumping all MXF into a huge pool of G-Tech FC RAID.
As far as the Avid, its just a host card and recently Avid runs fine as software only...well runs fine with the added crashes here and there but thats another story :P

Thanks Jeff for the tips. Im hoping I can get this going in one attempt :)
 
You must log in or register to reply here.
Back
Top