Can Adobe CS6 Use 2688 CUDA Cores?

[Jeff Kilgroe]

Yes, on Windows. Originally the GTX690 was being reported as only one card in Resolves info panel. Having gone back to check in other configs since posting that, it reports as two.

I had some difficulty getting the Quadro 6000 to play nice with the GTX cards. Had to clean-install the GeForce WHQL drivers first and get the GTX card working, then install the Quadro WHQL drivers. From there, had to update GTX drivers to current beta via the properties panel in the Device Manager.

Working on the write-up now, probably won't get it finished up today as I'm about out of time for this. I'm going to re-run the tests as soon as new TITAN drivers show up that seem to improve consistency. The card shows a lot of promise, but it's all over the place. The 690 is the superior card for some tasks, even according to nvidia's own info.

And the Quadro 6000 is still an awesome card that doesn't get enough street credit. NVIDIA has yet to release the K6000 that will replace it -- which should effectively be a GK110 based card (similar to the TITAN). Not sure why it hasn't appeared just yet. They did ship the K5000 for Mac officially last week. I'm not sure I see the point, but OK, whatever...

Glad to hear that!

Did you use a tool to wipe the drivers?

BTW - do you find better CPU coolers allow you to stay in Xeon's TurboBoost mode longer - and how does one measure when one is in TurboBoost mode?

Just used the nvidia option for the clean install to wipe the drivers.

Good CPU coolers are a must. If properly cooled, the CPUs can run at full speed/ load all day long. When all cores are invoked, the clock speed drops, full turbo speed is only available when using the primary core and each core after that drops the speed a bit. The system may have 16 cores, but when software scales that far, the clock rates drop. So it becomes a balancing act of sorts and when it comes to walking that cores vs. clock tight rope, most software just sucks at it. On the bright side, most software doesn't scale to that many CPUs well, so at least the clock rates hold up with less core usage.

There are a handful of CPU monitoring tools out there. Most the time just watching CPU meters in task manager tell me what I need to know anyway, just knowing how the systems run. Start with tools like CPU-Z and go from there. Or if you have an ASUS, Gigabyte, EVGA, etc.. motherboard, they include their own monitoring tools that give all sorts of info.

 

[Joel Arvidsson]

What cpu coolers do you use in the supermicro?

 

[Jeff Kilgroe]

Mine is a kludge design that I don't want to recommend! It was put together early in the game when these parts just started shipping, but it works so I haven't updated it. I'm using Corsair water blocks that cover LGA-2011 and they just barely cover the CPU surface of the Xeon E5's. That seems to be a problem with most coolers out there, few have cold plates large enough for these Xeons. I'm using the SuperMicro 7047 chassis combo with the X9DAi motherboard and have a ThermalTake reservoir with front-mounted radiator and fan deal in the top two 5.25" drive bays. It sucks. Both figuratively and in actuality when it comes to airflow. Not terribly loud, but not pleasant to listen to...

If I were to change it, I would put everything into a different chassis. I like the SuperMicro one, but it has inadequate airflow for these CPUs, IMO. The fans are too small and there's not enough room to install "standard" 120mm units or dream of getting 120mm or even 140mm fan+radiator combo units. I would go with one of the new Corsair, Thermaltake or Cooler Master cases if I were building a new Xeon rig. And coolers like the Corsair H100i or H80i

I really wish someone else out there besides HP were making all-in-one vapor chamber coolers for these CPUs. HP's do a great job, but they also depend on the ducted airflow design of the Z820 as they don't have fans on the coolers themselves.

 

[Les C.]

I'm having an X9DAi placed in an Antec P193 V3 this week with a pair of E5-2670's and 128GB RAM - along with the Titan that just arrived. I put an X8DAE in the earlier P193, so ... I'm hoping it will go smoothly.

In theory, if they work, the heatsinks will be the Zalman CNPS9900Max - which should easily handle the max wattage the Xeons will produce, and with 130mm fans, should be rather quiet.

Only issue is, if the fans die, they will be a pain to replace, as they are mounted right in the center of the heatsink.

Shame to hear about the TurboBoost being limited to a single core. I guess I misread the docs.

Are Premiere/AME/AE taking advantage of the 12 cores?

Which apps are you noticing are NOT taking advantage?

 

[Jeff Kilgroe]

Should be a nice system. The Titan is going to sing once nvidia gets the drivers under control. Usually takes them a month or two with new GPUs. Even as it is, other than being somewhat inconsistent, it seems to perform adequately as the one and only GPU in the system under Resolve 9.1.1. In the "candle test", the Titan scored the best of the three GPUs I tested. When combined with the other GPUs, CUDA 5.0 didn't want to utilize both the Quadro and GTX cards together, so I left the Quadro 6000 as GUI. Performance on the other two cards with the Quadro as GUI were only about 5% better than when used as GUI+CUDA. Best scores were with the GTX690 as GUI+CUDA and Titan as CUDA. Scores were on par with what I'm seeing from others benchmarking 12-core Mac Pro systems with Cubix expanders holding 3 x GTX580 GTX680 cards.

I think when the driver settle a bit, the Titan is where it's going to be. Especially if you can run two of them....

I couldn't tell the difference between the 3 cards' performance in Premiere. They're all just fine for Premiere and run great. You have to do some serious transitions and GPU-accelerated FX stuff to hammer on the GPUs all the way and you'll wear out the CPUs first. GTX690 is nice, but falls short on CUDA compared to the Titan, or I think it will once Titan matures. What hurts it is the smaller RAM allotment and being divided amongst two separate GPUs. It's an awesome card, but the Titan is just going to kick its ass when it comes to larger data and compute tasks. Besides, the 690 is a frickin' power hog, needing two 8-pin power connectors.

For general work in Resolve, a single Titan will probably fit my needs just fine. Truth be told, I haven't been using Resolve all that much lately, and the little I have been doing has been served just fine on the Z820 with the Quadro 6000 serving as both GUI+CUDA.

As for the turbo boost, I didn't explain it very well look at it above. I'll take another stab at it in a subsequent post. It isn't just limited to 1 core, but the clock rate scales based on the core load on the CPU.

Premiere and Resolve use all the cores, although we seem to cross over that threshold of diminishing returns somewhere between 8 and 12 threads, depending on what's going on. AE doesn't scale to all those cores while working. But it can scale very well for rendering. Same with Premiere/ AME.

REDCINE-X is optimized for the Rocket and does a crap job of using multiple cores. REDLINE rendering is better facilitated for that, but you really need a separate render manager to take full advantage. If you have a Rocket the system is going to scream like crazy anyway, combining the Rocket for decode and the rest of the system for encoding.

...Not familiar with those Zalman coolers.

 

[Sebastian_A]

Jeff,

Not sure if you have the monitor to test this or not...but can the Titan output 10bit over Displayport like the Quadro's do?

I'm also interested in this

 

[Les C.]

...
Premiere and Resolve use all the cores, although we seem to cross over that threshold of diminishing returns somewhere between 8 and 12 threads, depending on what's going on. AE doesn't scale to all those cores while working. But it can scale very well for rendering. Same with Premiere/ AME.

...

Does that indicate that hyper-threading should be turned OFF in the BIOS - or is most sofware now able to detect and deal with hyper-threading?

And does that have anything to do with the amount of RAM divided by the # of cores? If I remember, Adobe wants maybe 4GB per core, but ... possibly that's a minimum and more would be better?

 

[Jeff Kilgroe]

Leave HyperThreading on! HT handles very well these days.

RAM... The 4GB per core amount is pretty common these days. That's the minimum I would consider if you do lots of intense 3D rendering or similar tasks that can utilize all 32 possible threads in these systems. 4GB per core is only 2GB per thread. I started with 64GB in my Z820 and that's what my SuperMicro build has in it still. I found I could run out of RAM in the Z820 if I really pushed it for some things. So I upgraded to 128GB and that has been plenty good.


Have not got to the whole 10bit output thing yet. I actually have to go get some work done now...

 

[Les C.]

Should be a nice system. The Titan is going to sing once nvidia gets the drivers under control. Usually takes them a month or two with new GPUs. Even as it is, other than being somewhat inconsistent, it seems to perform adequately as the one and only GPU in the system under Resolve 9.1.1. In the "candle test", the Titan scored the best of the three GPUs I tested. When combined with the other GPUs, CUDA 5.0 didn't want to utilize both the Quadro and GTX cards together, so I left the Quadro 6000 as GUI. Performance on the other two cards with the Quadro as GUI were only about 5% better than when used as GUI+CUDA. Best scores were with the GTX690 as GUI+CUDA and Titan as CUDA. Scores were on par with what I'm seeing from others benchmarking 12-core Mac Pro systems with Cubix expanders holding 3 x GTX580 GTX680 cards.

I think when the driver settle a bit, the Titan is where it's going to be. Especially if you can run two of them....

I couldn't tell the difference between the 3 cards' performance in Premiere. They're all just fine for Premiere and run great. You have to do some serious transitions and GPU-accelerated FX stuff to hammer on the GPUs all the way and you'll wear out the CPUs first. GTX690 is nice, but falls short on CUDA compared to the Titan, or I think it will once Titan matures. What hurts it is the smaller RAM allotment and being divided amongst two separate GPUs. It's an awesome card, but the Titan is just going to kick its ass when it comes to larger data and compute tasks. Besides, the 690 is a frickin' power hog, needing two 8-pin power connectors.

For general work in Resolve, a single Titan will probably fit my needs just fine. Truth be told, I haven't been using Resolve all that much lately, and the little I have been doing has been served just fine on the Z820 with the Quadro 6000 serving as both GUI+CUDA.

As for the turbo boost, I didn't explain it very well look at it above. I'll take another stab at it in a subsequent post. It isn't just limited to 1 core, but the clock rate scales based on the core load on the CPU.

Premiere and Resolve use all the cores, although we seem to cross over that threshold of diminishing returns somewhere between 8 and 12 threads, depending on what's going on. AE doesn't scale to all those cores while working. But it can scale very well for rendering. Same with Premiere/ AME.

REDCINE-X is optimized for the Rocket and does a crap job of using multiple cores. REDLINE rendering is better facilitated for that, but you really need a separate render manager to take full advantage. If you have a Rocket the system is going to scream like crazy anyway, combining the Rocket for decode and the rest of the system for encoding.

...Not familiar with those Zalman coolers.

Jeff - I've got my X9DAi + Titan + Corsair AX1200i with win7-64 installed.

The SuperMicro BIOS is flashed with the most recent 1.0c and I've got nVidia 314.22 installed.

If I connect a single DELL 2560 x 1440 to the DisplayPort and if I drag a browser around, it's like molasses - and it leaves trails. This also happens when I connect my two DVI 2560 x 1440 monitors (but I wanted to try just the one).

I noted that many of the default optimum settings in the BIOS don't match the PDF manual.

The RAM DIMM settings are a bit confusing how to set it up properly for ECC (demand scrub is disabled, patrol scrub is enabled) and for some reason Data Scrambling gets enabled for the optimal defaults, even though it's shown as Disabled as default in the manual (I disabled it again).

And for the Chipset/North Bridge/IOH configuration/CPU1 PCI-E 3.0 Slot 3 (slot used for my Titan) link speed, it was set to GEN3 even though Gen2 is the default in the manual. Even if set to Gen3, the nVidia control panel system information showed Gen2?

I tried changing it also, but dragging a browser on the screen is still messed up.

Do you have any guidance on how you set yours up? This is crazy that a Titan & X9DAi performs like a dog compared to my 3 year old X8DAE with GTX-580? This is before I've installed any software, just the drivers. Ahhhh!

 

[Brandon J.F.]

REDCINE-X is optimized for the Rocket and does a crap job of using multiple cores.

I noticed that as well. It only uses around 30-40% of my CPU when playing back footage.

 

[Les C.]

Jeff - I've got my X9DAi + Titan + Corsair AX1200i with win7-64 installed. The SuperMicro BIOS is flashed with the most recent 1.0c and I've got nVidia 314.22 installed. If I connect a single DELL 2560 x 1440 to the DisplayPort and if I drag a browser around, it's like molasses - and it leaves trails. This also happens when I connect my two DVI 2560 x 1440 monitors (but I wanted to try just the one). I noted that many of the default optimum settings in the BIOS don't match the PDF manual. The RAM DIMM settings are a bit confusing how to set it up properly for ECC (demand scrub is disabled, patrol scrub is enabled) and for some reason Data Scrambling gets enabled for the optimal defaults, even though it's shown as Disabled as default in the manual (I disabled it again). And for the Chipset/North Bridge/IOH configuration/CPU1 PCI-E 3.0 Slot 3 (slot used for my Titan) link speed, it was set to GEN3 even though Gen2 is the default in the manual. Even if set to Gen3, the nVidia control panel system information showed Gen2? I tried changing it also, but dragging a browser on the screen is still messed up. Do you have any guidance on how you set yours up? This is crazy that a Titan & X9DAi performs like a dog compared to my 3 year old X8DAE with GTX-580? This is before I've installed any software, just the drivers. Ahhhh!

Jeff - nevermind. I went back to the BIOS optimal defaults again, and that fixed things. I must have disabled something that was needed.

 

[Jeff Kilgroe]

Jeff - nevermind. I went back to the BIOS optimal defaults again, and that fixed things. I must have disabled something that was needed.

Glad you got it fixed. Sometimes these things are funny that way. I've found that with a lot of these workstation boards too that their BIOS settings out of the box or when you choose a certain profile, it doesn't match up with the recommended settings. Usually best to not stray from default or optimum settings if you can.

Titan drivers are still a little quirky. 314.22 WHQL drivers are the way to go, they're about 3 weeks old -- I've been hoping for a new beta release, but nothing yet...