Z800 - Optimum card order for BlackMagic, Cuda, Nic and Telsa

[Simon Dunne]

So, i have a fair few Z800's to build. I've got the first one ready with our first Z800 CS6 image and want to ensure it's a fast as possible. Inside the unit, we've got a BlackMagic Decklink HD Extreme 3D and Quadro 4000. Currently, this is the card configuration we have.


Slot 1 - PCIe x8m/x4e : SDI O/P BLACKMAGIC 3D Extreme
Slot 2 - PCIe x16 : nVidia QUADRO 4000
Slot 3 - PCIe x8m/x4e :
Slot 4 - PCIe x16m/x8e :
Slot 5 - PCIe x16 :
Slot 6 - PCI 32BIT :
Slot 7 - PCIe x16m/x8e :

So, we've set up CS6 (build 6.0.0) with mercury transmit and it's pumping everything out of BM card (drivers 9.5) from the time line. At the minute, we're testing it with raw XD-CAM HD422 mxf wrapped footage on the XD-CAM HD422 Premiere sequence. The BM card is set to 1080i50. All the footage is stored on a NAS with a 1Gb connection. I've also got a few .R3D 4K clips that i'm testing on the XD-CAM HD422 sequence too.

When scrubbing the xd-cam and red footage, we get a lot of black. Both the preview and program playback and paused settings are set to full. When we set this to 1/2, obviously, we get better results. So, this seems to rule out the network as the issue as it's still reading the files weather 1/2 res or full. So, questions;

1: Is Premiere, the Quadro or the BM card the cause for all this black video when trying to scrub?
2: We have a tesla to add to the system. What slot should we put it in for best performance. I'm guessing 5?
3: We're going to add an Intel NIC to see if we can boost network speed, again, which slot for this?
4: Do we need the nVidia transmit driver? http://www.nvidia.co.uk/object/cs6-sdi-transmit-plugin-driver-uk.html ... surely that's only for people with the nVidia SDI output card, not blackmagic, right??

We've taken a look at the following;

http://h20331.www2.hp.com/Hpsub/downloads/Optimizing_PCI-Express_on_Z800_3-18-09.pdf

....but it would be nice to hear if Dave from Adobe has any thoughts. Are there any BM peeps on here too??!

TIA, Simon

 

[Eric Z]

Simon,
I'm not an expert in HP Worksation-class machines. I have more experience with Dell-branded machines.
However, I'll try to answer your questions and add some more info below:


1. Premiere CS6 is most likely not the cause. The Quadro 4000 is an OK card, but a Quadro 5000 is much more recommended.
I'm guessing the OS is Windows 7 Pro SP1 x64. Make sure the OS is patched with all the latest stablity and performance updates. Also make sure the Quadro driver is the latest one certified by HP.
I would also test and compare the latest Quadro driver available on NVIDA's website, even if it hasn't been through ISV certification.
Then make sure the NVIDIA driver options are configured properly (there are many sliders for performance/Quality preference).


There is also 30bit color support which you should enable if your monitor(s) support 10bit/channel (1.07billion possible colors).


To eliminate the option that the BlackMagic card is causing a bottleneck, connect your monitor(s) directly to the Quadro card and check the performance you're getting.


2. I'm also guessing slot #5 is the place or the Tesla card. Not sure, as I have never used a Tesla card. Consult with Tesla documentation on what's the best way to setup a Maximus configuration.


3. The Intel NIC is probably going to be a PCIe X1 card, and can be installed in any one of the Z800's PCIe slots.
Leave the Intel NIC card aside until you have all your slots populated with the more important cards.


4. The NVIDIA Mercury Transmit Plugin is only for Quadro cards with SDI-output. Do you have those or do you have the regular Quadro cards (with DVI-DL, DP)?


I'd recommend strong Xeon CPUs with 8GT/s (Xeon E5-2665 is a good example), and at least 32GB of 1600MHz ECC RAM for each CPU (64GB total).
I'd also recommend adding two IcyDock units (around $60 each) that would hold 4x2.5" SATA3 SSD each (8x SSDs total).
All the SSDs are to be connected as one big RAID0 array to a HighPoint RocektRAID card (PCIe X8), for example the RocketRAID 2720SGL. It's small, cheap ($160) and delivers up to 3GByte/s with fast SATA3 SSDs.
Work on your 5K R3Ds on this array. Once you're done, export via network/FireWire/eSATA/USB3.0 to the external storage for backup and delivery.

So if the Quadro occupies slot #2, and Tesla card occupies slot #5, the RAID card should occupy slot #4.

This kind of setup should get you smooth 24fps playback, maybe even at 1:1.
Note: it's important that the path from the storage to the RAM is as fast as possible. Scrubbing means that Premiere CS6 creates a RAM preview.
Only when the video portion you're looking at is in RAM - it will playback smoothly back and forth.

 

[Simon Dunne]

Thanks for the reply Eric. So;

1: Yup, W7 64x Enterprise, SP1, all patched. I think we're running the latest release driver, not a beta.
2: I think 5 is going to be best for this
3: That's the plan....agreed
4: So, the SDI board, i have. But i'm not using it in this system, just the blackmagic, so i'm guessing no transmit driver then

After doing some testing, playback is much smoother from the D: drive, which is 2x600GB 15K SAS drives in raid 1. Funnily though, it's pulling the same amount of data as the network does when running the project of the NAS. It seems though, that scrubbing the timeline mega rapidly off the D:, seems to take advantage of the 24GB of RAM in the system. The NAS is running on SMB1 at the minute, so we're going to switch it to SMB2 to see what speed advantage that may give us. Again, this is XD-CAM HD422 footage, not RED....just yet. We like to bed the systems in first to iron out all of the bugs!

Anyone else with experience or knowledge of systems and networks, feel free to chip in :auto:

 

[Karl Soule]

For troubleshooting out random problems, I do recommend a small utility program called SiSoftware SANDRA. It's mostly known as a benchmarking tool for gamers, but it gives great information about performance of your RAM, drivers installed, CPU, etc. Plus, you can pick a similar system to compare to, and see how your current config is stacking up. Years ago, it solved a bizarre problem I was having with video playback - When I ran SANDRA, I noticed that the access speed of my RAM was about 1/50th of what it should be. Started reinstalling drivers related to the motherboard and RAM, and one of the Northbridge drivers turned out to be the culprit.

 

[Eric Z]

I believe luigivaltulini and Jeff Kilgroe have the most knowledge with regard to building custom workstations.
I highly recommend you consult with them, as they have lots of hands-on experience using state-of-the-art parts.