Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Mac Pro PCI card configuration / PCI expander / GPUs / Resolve

Jarek Zabczynski

Jarek Zabczynski

Well-known member
Joined
Apr 3, 2007
Messages
2,410
Reaction score
4
Points
38
Location
Los Angeles
Website
www.jarek.com
So I just picked up the latest 12 core MP, not sold on the new thing coming out later.

Trying to figure out my best plan of attack on loading it up. Already picked up SSDs and 64GB of ram.

Initially my plan is to just use the MP alone with Nvidia GPU (580, 680, 780)
BM Decklink 4k extreme
Atto r680 with sas tower
and maybe a rocket.

However I've been considering a Cyclone chassis.
Would two less powerful NVIDIA GPUs in the chassis plus my stock ATI in the tower be a better idea?
This would also allow me to put in some esata/usb3 cards and maybe a LTO drive in the Cyclone.

Thoughts?


EDIT
------------------------------------------------

So here's my setup, or at least as soon as everything arrives. Coming from a 2006MP I'm pretty excited! Still debating my additional GFX cards.


2012 Mac Pro - Dual Six Core 2.4 (will upgrade processors when prices drop)
64GB Ram from OWC
Dual 256GB Samsung 840Pro SSD for Boot - Raid 0
1TB Velociraptor for Application cache/scratch and sound and image libraries
Dual 2TB Ultrastars in Raid 1 for project files
Stock 1TB drive for personal stuff, music, pictures
External Bluray drive on USB2.0

Slot 1 - Nvidia GXT680
Slot 2 - Cyclone 2707 Card
Slot 3 - Atto r680 Raid Card
Slot 4 - BM Decklink 4k Extreme

Promax 24TB, 8 Bay Raid Tower


In Cyclone

Slot 1 - GFX Card 2???
Slot 2 - GFX Card 3???
Slot 3 - Red Rocket
Slot 4 - Atto h644
Slot 5 - Caldigit esata/usb3.0 card

4 Bay trayless drive cage with fanout from h644
External SAS port for LTO drive in the future

Turning my 2006 into my secondary machine and dual boot win7 machine so I can dump my aging HP pavilion.
 
The Cyclone is a nice unit, you should hit up fellow Red User Dustin Cross with questions. I think Dusty has more real-world experience with the Cyclone than anyone I know. At least doing the sort of stuff those of us here tend to do... You will at least want to replace the fans in the Cyclone, possibly put it in an entirely different chassis to accommodate your needs.

That said, you would do best with a good primary GPU. The GTX680 Mac edition is my top recommendation for now. Barefeats recently did some Mac GPU benchmarking and the official Mac edition 680 held pace with an EFI modded 780 they picked up from a reputable supplier.

I can't comment on DaVinci Resolve 10 and how it will use hardware, other than it has been "improved". With Resolve 9, having 2 GPUs is a waste. You're better off to go with one good one or to really go for it and use 3 or more! The reason why is that when you have one GPU, it uses it for acceleration and GUI and the penalty for the dual use is only around 5%. If you add a second GPU, the primary is used for GUI, the second is used for acceleration. You effectively have wasted a slot to gain a 3~5% bump in GPU performance. When you have 3 or more, the primary is still GUI, but all subsequent cards can be pooled for acceleration. After 3 or 4 CUDA cards in the accelerator pool, you have passed the threshold of diminished returns -- and will have totally over-saturated your PCIe slot if the cards are in an expander.

For slot configuration in the Mac, the bottom slot is obviously the GPU as it's #1 and the double-width slot. Next up is slot-2, the other X16 slot and that is where the expander would connect. I would put the ATTO R680 in slot-3 and the Decklink in Slot 4. You will have the issue of where to run out the additional cables for the Decklink...

Now, this is where it gets fun and creative and there are really no right answers. You could put a Rocket as well as the Decklink in the expander. Really, you could also put the SAS host in there as well and use a one like the R608 or even the R60F and have a large internal RAID inside your expander unit. If you want to add an LTO drive, it's a bit tricky on the RAID host. I would actually add a second SAS card -- the H680 -- that you can use for connecting the LTO and any eSATA devices you have -- REDMAG Readers, etc..

So maybe what you could do is the GTX680, Cyclone PCIe expander card, a good USB3 card and an ATTO H680 could be in the Mac. In the Expander, you could put the Rocket, Decklink, and a RAID card with drives. You could also add another GPU or two in there, but you're really starting to overload what one PCIe X16 slot can handle. All fine and good, provided you're not trying to use too much of it all at once.
 
Jarek Zabczynski said:
Do all my GPUs need to be nvidia? Could I do the stock ATI in the tower and say a pair of 680s in the expander?
Click to expand...

Yes, you can do that. I have encountered some glitching with mixed ATI/Nvidia configurations. But if the two Nvidia cards are just there for CUDA acceleration, it should be just fine. At least for Resolve. I'm not sure how Premiere CC would respond, but multi-GPU support there is still very much a work in progress.

Most everything else just uses single GPU, so would operate off the ATI card in your tower.
 
Jeff Kilgroe said:
If you add a second GPU, the primary is used for GUI, the second is used for acceleration. You effectively have wasted a slot to gain a 3~5% bump in GPU performance.
Click to expand...
That issue had been addressed, if I remember it correctly, starting with version 9.1.3. Now you can use the GUI card to process both by ticking the "Use Display GPU for Compute". Unfortunately, in that case I doubt you can use mixed GPUs types.
 
I'm putting together another single GPU system next week. I'm going with nVidia Titan for this box, should be here on Tuesday (it's my first Titan build). The Cuda cores and RAM are the main factors I'm paying attention to these days.

This will be the system that eventually houses the new Red Rocket-X when it arrives and I'm sort of building it with Resolve 10 in mind and whatever Dragon 6K does do those minimum requirements. Might add an additional GPU. Have to see what I have laying around here.
 
I would bet anything, that single GPU, even the Titan, will not cut it with Resolve 10. Actually, I doubt, that two will cut it as well. Other than noise reduction by Dark Energy with it's massive hardware requirements, I hadn't seen any other software, that is capable of doing a real time temporal noise reduction and motion compensated speed ramps. And don't forget OFX. That's another huge GPU drain. And, if you want to do 4k in 3D... well good luck with that.
I truly hope, that BM doesn't continue to try to reinvent the wheel by trying to render everything on the fly, instead of doing it the way it's done by pretty much every other manufacturer- by efficiently using the background render cache.
 
As always Jake. You speak the truth.

My only real goals with this box are grade at 1/2 res and export at full. My fingers are crossed.
 
jake blackstone said:
That issue had been addressed, if I remember it correctly, starting with version 9.1.3. Now you can use the GUI card to process both by ticking the "Use Display GPU for Compute". Unfortunately, in that case I doubt you can use mixed GPUs types.
Click to expand...

Cool. I think I'm at 9.1.3 or whatever the current release is, I missed that little gem of a feature.

Yes, mixed GPU types are not a good idea for compute. IIRC, Resolve only uses one form of acceleration at a time anyway and it tries to go to CUDA if available then OpenCL.
 
Great input guys. I'm thinking for now I'll just try with a single GPU setup and see how it goes. NVIDIA 680 still the best choice? Was also looking as modded options from macvidcards on ebay.

Really wish there was another slot on the MacPro. I'm getting a Rocket and TB enclosure for DIT work that I've been getting a lot of calls for in my area. Though it would be great to throw it in my tower when I'm home. The downside is that I'm left without any room for esata or usb3 cards.

Jeff, are there any SAS cards with more ports that would run an 8 drive tower AND eSATA fanout? I know there are issues running a raid and single drive at the same time.
 
You may want to consider going Hackintosh instead of actual Mac Pro if you want that extra expandability. The Mac Pro lacks expansion slots and enough lanes for all those cards to run optimally. A PCI expander may get you more PCI slots and cards, but you're just diminishing the performance of all the cards in it.

I looked down on Hackintoshes myself for a long time, but really started looking into it more once the new Mac Pro was announced with no internal expansion. I don't want to be hooking up a bunch of external equipment on set and want it to be as easy as possible to transport a full system. So I did some research and built my first Hackintosh and I wish I had done it earlier.

I now have a tower with a GTX Titan (which the current Mac Pro can't handle without special power additions), Red Rocket, Atto SAS card, Decklink card, and room for another PCI card. And the Titan is running at x16 and all the other cards are at x8, plus I do still have one Thunderbolt port. There's also 6 USB 3.0 ports, 9 USB 2.0 ports, 1 eSATA port, 6 SATA 6G ports, and 2 SATA 3G ports all on the motherboard. I already had all the PCI cards and monitors (except the Titan) and the total cost of the whole system was under $3K with the Titan, 32GB 2400MHz RAM, motherboard, CPU, cooling, case, PSU, optical drive, SSD drive, memory card reader bay, much less than an actual Mac Pro. It may not be quite as fast as a Mac Pro being that it's a single quad core CPU configuration, but you could build a dual Xeon Hackintosh for less than you spent and get more capability. Check out tonymacx86.com.

Regarding running a RAID and eSATA fanout, I'm looking for the same thing. I found the Areca ARC-1882-ix-16. That particular card (there's other models) has one external SAS port and 3 internal SAS ports for 16 channels total, and it lets you run multiple RAIDs from one card. What Areca told me is that you could set up 8 channels for a RAID and then set up the other 8 as pass-through, but they weren't sure they'd work with an eSATA fanout cable. My case has room for 8 hard drives that I could run for a RAID, plus more for additional drives and my SSD. Personally I want to run 8 of those channels (2 internal SAS ports) to an internal RAID inside the case, and then the 3rd internal SAS port to a plate that would give me a 2nd external SAS port, and then run those 2 external SAS ports with eSATA fan-out cables. Areca said it may be possible but they couldn't guarantee the fan-out cables would work. So it would need to be tested and it's not a cheap card. I do have room for another card in my Hackintosh, so I could just keep the H680 for eSATA and add a separate RAID card like the R608 (internal) or R680 (external).
 
Hackintosh systems are always an option, but not an option I can ever recommend professionally. There are PC systems out there with more slots, but you have to understand the platforms they're built on, to really know what you're getting. Sometimes more slots is not more better. The i55xx chipsets, namely the i5520 that drives the current Mac Pro systems is only capable of 40 lanes maximum of PCIe v2.0. Apple has allocated all 40 of those lanes into 4 slots via 2 @ X16 and 2 @ X4. Some PC makers offer more slots on the same platform by splitting up the lanes more -- like 1 @ X16, 2 @ X8, 1 @ X4, 4 @ X1...

Some larger server motherboards can actually incorporate two i55xx series chipsets in a PCIe cascade arrangement that increases the number of lanes from 40 to 64. However this comes with other compromises in terms of additional PCIe cycles and latency as well as system size. PCIe lane allocation always leads to compromises. I'm puzzled every time Intel releases a new platform and they don't address this, they continuously miss the mark. Desktop class PC's are terrible in this regard, most of the i7 based systems out there can't actually run all of their slots at full bandwidth or even manage to run all of them at any capacity. Intel has made some efforts with the E5-2600 series Xeons and they have moved the PCIe host controller off of the chipset and onto the CPUs themselves. So with dual CPU systems we get more lanes, and more again with quad-CPU arrangements.

IMO, the biggest problem with PCIe expanders is the price. I've used them, but ultimately moved away from them myself. For most of the work I do and for the price of it all, it made more sense for me to have more systems with no expanders than fewer systems with a PCIe expander or two.

In the end, there is no one right or wrong answer to any of this. You have to be able to build the system(s) that will work the way you want them to. Assuming that's even possible.

As for SAS cards with more ports... Yes, there are some, but for some reason no one seems to make the card I truly want. ATTO offers the R60F model which offers 4 mini-SAS connectors internally for 16 SAS/SATA channels at 6Gbps each. Unfortunately, they don't offer one that does a split of 8 internal and 8 external or one that offers 16 channels externally. Best you can do is 4 internal with 4 external or a card with 8 external. Areca has a few different options, but they fall short on the ports too. Not sure why this is.... That said, 4 or 8 external channels is usually pretty good for most uses. All 8 channels is 48Gbps total bandwidth or nearly 6GB/s throughput. You can have tons of drives on each channel via port-multipliers.

Given those restrictions and the other little hang-ups with trying to run individual eSATA devices on a card also driving a RAID, it makes more sense to allocate all of your RAID card to RAID activities. And then use another card for other devices. The drawback to that is we obviously need another precious PCIe slot.

Like I said above, I've found more use with multiple systems. I can connect systems via 10G Ethernet and have one system that has a decent GPU, 10G card, Decklink card and a Rocket. I can have another system that has eSATA and SAS connectivity, hosts my LTO system and other things. They each have internal RAID workspace, but also share NAS or SAN storage over the 10G connection(s). That's expensive to set up too, but it seems to get more done than trying to shoehorn everything I want into a single workstation via an external box that costs just as much as another workstation.
 
Blackmagic has suggested that the recommended config for v10 will be a gtx680 in the Mac for GUI and two titans in an expander for gpu.
If this is the case, a lower price 2 double wide slot expander like this would be a good deal
http://www.ebay.com/itm/Cubix-Xpand...243?pt=PCC_Video_TV_Cards&hash=item4611cbe6a3
That will leave you two slots in the mac for hba and video io. No room left for a red rocket. Oh well....
 
The 10GbE solution sounds nice, but I need a 10GbE switch for that to work don't I? Or can I just directly link two systems together? I have two towers now. A 2006 and a 2012.
 
It should work without a switch. I can confirm directly connecting 2 Myricom 10GbE cards (2 x CX4 or 2 x fibre) will work and give you a min 300MB/s (large file transfers) between workstations. I haven't used any of the more expensive brands but AFAIK they should either auto-negotiate a link or be configurable to work without a switch if it's a direct connection between 2 cards. Emailing support before a purchase is a good idea. BTW the latest drivers from Myricom are thunderbolt aware and both types of cards work with my Magma Expressbox 3T. I use the fibre cards for Phantom work and the CX4 cards for NAS applications like Jeff described.

FWIW I use one of these for PCIe expansion when needed albeit it's limited to x8 not x16. I put the board into another tower to accommodate an internal RAID.

http://www.dynapowerusa.com/dyna/ASP/CP_i_NA250A.asp

So far, if a card shows up using the Mac Pro's internal PCIe slots it'll also work in this chassis. Disclaimer - if you're going for multiple GPUs for Resolve then spend the extra cash and get the Cyclone like Jeff suggested as the Netstor I have will bottleneck quickly. This one looks more promising:

http://www.netstor.com.tw/_03/03_02.php?MTEw#

Also for my Atto H608 and R608 I have one of these which comes in handy:

http://www.pc-pitstop.com/sas_cables_adapters/AD8788-2.asp
 
Knowing what I do about expanders and based on my own experience with them, the only two I would recommend for Mac Pro tower expansion is the Cyclone -- I think the 2707 model? And the GPU XPander models from Cubix. I've had terrible luck with Netstor and Magma. The Magma Thunderbolt expanders look nice and I keep hearing good things so I don't want to turn people away from those, but I have not tried their Thunderbolt models myself. At any rate, Thunderbolt PCIe expanders don't help us here with the Mac Pro. At least not until the new Mac Pro comes out and then we'll be holding our breath for Thunderbolt-2 capable expanders...

Two titans in an expander is going to push that expander to its limit if it's PCIe v2 based -- which anything connected to a current Mac Pro tower will be. I have not seen too much in the way of PCIe v3 expanders yet. But I think most systems that can benefit from them are multi-CPU Xeon boxes and with the Sandy Bridge EP Xeons, we have more lanes. So I don't think the market for a PCIe v3 expander is very big as we move forward with this.
 
Well my 24TB raid from Promax and Atto r680 are on their way. Still figuring out the rest. Probably getting a GTX680 GPU, should I go with the Mac version or a flashed one with more memory?
Planning on getting a rocket, though I'm also tempted by the Dragon upgrade and getting a discount on rocket-x. I would want my rocket mobile since I've been getting calls for one in my area recently, but also being able to use it in my workstation when possible, that's why having it in an expander seems so appealing. I might just bite the bullet and go with the cyclone, put two more GPUs in there and a rocket plus usb3/esata. Also room for an LTO in the future.

Decisions, decisions...
 
You must log in or register to reply here.
Back
Top