New Workstation Build - i7 X99, Titan X, Overclocking...

[Phil Holland]

Okay. Playing around a bit more with stripe size is working well I think. 5.8GB/s Read and 1.4GB/s Write on main RAID 0 volume now. Going to investigate my other volumes now.

My work RAIDs mainly hold bulk .R3D data and that's backed up triplicate. RAID 0 stability is a light concern, but since everything is backed up "early and often" and source images are backed up before even hitting this box I'm less concerned. The enterprise and NAS drives seem to be working fine and have some lasting power (knock on wood).

 

[Shiblon Wixom]

All devices accounted for in Device Manager. Not sure what the strays were, but after a full shut-down and restart everything was properly identified. Still some more software to install, but almost done. I have not tested the Thunderbolt card yet, going to do that now, I guess....

You may need to install a Windows OS, in order to install/update the Motherboard chipsets and TB cards. Sometimes it also takes a few restarts for all the firmware updates to kickin.

I have an older internal Areca 1224 HBA card, with x8 SAS drives in a RAID5, my system gets around 1.4 GB/s Read and 1.2 GB/s Write. When I mount the HBA controller in a TB chassis and the drives in a SAS Raid Tower, the system gets about half those speeds over a TB cable, around 700 MB/s Read and 600 MB/s Write.

 

[Jeff Kilgroe]

You may need to install a Windows OS, in order to install/update the Motherboard chipsets and TB cards. Sometimes it also takes a few restarts for all the firmware updates to kickin.

This system is only running Windows. And yes, multiple restarts to get things sorted are normal. This one was a bit weird that it took so long for the system to identify and sort it.

All running well here now except for the 121 and 131 UpdateManager errors that currently plague most X99 and similar systems. Easy to fix (tell Windows not to go searching for hardware driver updates on startup), so I haven't done that yet. This is looking to be a nice build. Hopefully the other Thunderbolt card will work properly when it arrives. But that's the only thing I'm waiting on hardware-wise. I have a RED mini-mag reader all ready to mount internally as well and I'll do that the next time I pop the hood, so it may or may not happen before the TB card arrives.

I have an older internal Areca 1224 HBA card, with x8 SAS drives in a RAID5, my system gets around 1.4 GB/s Read and 1.2 GB/s Write. When I mount the HBA controller in a TB chassis and the drives in a SAS Raid Tower, the system gets about half those speeds over a TB cable, around 700 MB/s Read and 600 MB/s Write.

Sounds about right. Especially if that's TB and not TB2. If you're dropping that much speed over TB2, then you have something crimping your bandwidth.

 

[Jeff Kilgroe]

Okay. Playing around a bit more with stripe size is working well I think. 5.8GB/s Read and 1.4GB/s Write on main RAID 0 volume now. Going to investigate my other volumes now.

Which disk benchmarking tools have you used? Those writes seem to be about what you should be getting, even a bit on the fast side for those drives, assuming you still have the 8 x 4TB drives you started with. That read speed is F'd up or getting seriously skewed by repetitive data being cached... Seems like typical CrystalDiskMark hijinks.

My work RAIDs mainly hold bulk .R3D data and that's backed up triplicate. RAID 0 stability is a light concern, but since everything is backed up "early and often" and source images are backed up before even hitting this box I'm less concerned. The enterprise and NAS drives seem to be working fine and have some lasting power (knock on wood).

Same perspective and approach here. Workspace RAIDs are typically RAID-0. Once the RAIDs are up and running, failures are usually minimal and far between. It's usually much faster to swap a failed drive and restore data from a backup than it is to replace a redundant drive in a parity set and rebuild the array.

Preliminary numbers from a RAID-5 made from these 6 x 6TB drives is giving me 860MB/s writes and 830MB/s reads. It's about what I was expecting, really. RAID-5 on my Pegasus 2 R6 units with 3TB Seagate drives installed is right around 810MB/s read and write. Those 3TB drives that Promise is using in the Pegasus2 RAIDs hold about 200MB/s R/W on their own.

I'll do some more testing to see if those numbers hold on this RAID and will also give RAID-50 a try. Although, it seems that RAID-50 on the Areca controller should be implemented the nearly the same in terms of data and striping allocation, just more to manage on the setup side. RAID-6 would probably make more sense from a redundancy perspective since I would gain a guaranteed 2-drive redundancy. RAID-50 still allows for 2 drive failures, provided they are the right ones, but we get that with the standard RAID-5 setup as well...

Maybe I should just leave it a RAID-0 and be happy with the performance and knowing that I have oodles of immediate storage... It's only 1.5~2X more space than my other DAS RAIDs and will probably feel small in another year or two anyway. The problem is, I could actually fill this up and if I get lazy and fall behind with backups...

I can always partition into a couple more manageable chunks if I feel the need...

 

[DJ Meyer]

All I've used for years now are RAID-0 working space and hourly incremental backup over 10Gb. Worst case scenario I lose an hour's work and don't take a big performance hit for using parity. Yet to have a drive fail.

 

[Jeff Kilgroe]

Going back to other details on the build... The C70 has become a bit cramped for this setup. The SAS interface cables all have additional molex power connectors and there's barely enough room to make all the HDD cables fit nicely on the back-side of the system and get the cover panel on. I spent a ton of time trying to manage the cabling in this system as I finished it up and there's just not enough room to do it well. Cable lengths have some to do with that as most of the PSU cabling turns out to be an awkward length where it needs to be wrangled, but it's not long enough to properly bundle and secure out of the way. So this causes a bit more disruption in airflow than I was hoping for. OTOH, air flow and temps seem to be great. But I have two 140mm SP intake fans on the front of the case, two 120mm fans pulling air the same direction mounted on the drive cages, to pull air over the HDDs. One 120mm at the rear as an exhaust, the two 140mm fans on the top radiator as an exhaust and the PSU exhaust fan which is about 140mm.

I thought I had the wake from sleep issue figured, but apparently not. Put the system to sleep last night when I was done and figured I'd see if it worked properly. It was running this morning and no indication in the event logs anywhere as to why this would be. The Corsair link software seemed to be the culprit before and this still may be related to the H100i cooler, it could be waking the system via its USB connection to operate the pump.

 

[Danko Dolch]

Hi Jeff,

thank you for sharing your setup - great info as always ;-)

- Have the same issues with the sleep mode on the ASUS X99-E WS
- Power off and sleep mode of DVI and Display Port connected monitors started to make trouble after switching a NVIDIA GTX 670 with a EVGA Titan-X SC

-> I'm still unsure what M2 SSD would make sense would like to have 1TB NVM version but there seems to be no offering with that specs...

-> I'm curious about the Areca Raid-0 setup... I use 2 independent software Raid-0 (4x4TB HGST SATA Deskstar each) delivering between 600 and 550MB/s
- I see that the Raid's are getting high IO loads starting with 3 concurrent requests... (I know that scales better with SAS but how much... and is it worth the higher cost?

-> What would be a good config for a 4TB + SSD working volume?

nice to see that the Thunderbolt card is working - has anyone running a BM UltraStudio 4K with such a config?

 

[Scott Pultz]

Where did you get 6Tb SAS drives for only $260/each?

How are you liking the Areca? Trying to decide between that and the Adaptec...

 

[Jeff Kilgroe]

Hi Jeff,

thank you for sharing your setup - great info as always ;-)

- Have the same issues with the sleep mode on the ASUS X99-E WS
- Power off and sleep mode of DVI and Display Port connected monitors started to make trouble after switching a NVIDIA GTX 670 with a EVGA Titan-X SC

It's weird. My other X99 build using the same X99-E WS motherboard has no sleep issues at all. Only difference between the two builds at this point is the memory modules are slightly different, CPU coolers are different and GPUs are different -- dual GTX980 in the other system. Oh, the other has an ATTO RAID card, but the sleep issue here still occurs without the RAID card installed. I'm stumped. Could be some obscure BIOS or Windows setting or glitch that will be near impossible to track down. I did however disconnect the USB comm from the Corsair H100i cooler and sleep seems to be functioning so far <knock wood> But then I shut the system off completely and haven't been back to it since late last night.

-> I'm still unsure what M2 SSD would make sense would like to have 1TB NVM version but there seems to be no offering with that specs...

NVMe is turning into a mess. Storage makers and motherboard makers can't agree on form factor. Intel and Toshiba are the primary NVMe SSD producers at the moment and they've gone with SAS connectors on the SSD units. Most of the motherboard makers are still trying to shoehorn NVMe into the m.2 paradigm and only provide one connector. At least it looks like SATA Express is going to die. Thankfully. There are really no NVMe M.2 units out there to be had and Samsung is the only one offering an m.2 in > 512GB capacity, but it seems to not actually be available if you try to buy one. Intel has 1.2TB NVMe 750 Series SSDs in 2.5" form factor.

-> I'm curious about the Areca Raid-0 setup... I use 2 independent software Raid-0 (4x4TB HGST SATA Deskstar each) delivering between 600 and 550MB/s
- I see that the Raid's are getting high IO loads starting with 3 concurrent requests... (I know that scales better with SAS but how much... and is it worth the higher cost?

Most of your onboard SATA hosts are going to run into bottlenecks when you start adding up the concurrent transactions. They're not built for performance RAID use. Even the ones which have onboard RAID. Software RAIDs incur some overhead and while RAID-0 isn't significantly demanding, the performance penalty increases a bit with each drive and more so when you create multiple RAID volumes this way. Going with a dedicated RAID controller is really the best solution for performance. SAS controllers are more robust and capable, but you don't have to run SAS drives on them, they are downward compatible with SATA drives.

-> What would be a good config for a 4TB + SSD working volume?

As of right now, I would use something like the Areca 1883i controller and the Samsung 850 Pro SSDs. You can fit an 8 x 1TB array into 4 x 3.5" drive bays. Start with 4 x 1TB in RAID-0 and that will give you 2GB/s reads and 1.8GB/s writes. Total cost on that RAID for SSDs with controller should be under $2500. And that leaves 4 SAS/SATA channels open for expansion down the road.

nice to see that the Thunderbolt card is working - has anyone running a BM UltraStudio 4K with such a config?

It's not working. At least the dual port card is not properly compatible with this motherboard. The single port version of the card is supposed to be compatible and I will have one tomorrow.

Where did you get 6Tb SAS drives for only $260/each?

How are you liking the Areca? Trying to decide between that and the Adaptec...

The pricing on the drives was a promotion for enterprise buyers last month and it was on 20-unit boxes through TechData, possibly a couple other distributors. Some vendors like Provantage seemed to have picked up on it and they lowered the prices on individual drives briefly. But this was a pretty good deal. I should have bought a few 20-packs and not just one. These drives are awesome, so far anyway.

The Areca controller is nice. Onboard LAN connectivity is goofy and the BIOS configuration and HTTP based management tool feels dated compared to what ATTO is offering these days. Performance is great, no complaints so far. I don't have any experience with Adaptec's current offerings. They were the go-to standard for years in the PC industry, but fell by the wayside about 10 years ago and seem to have been overtaken by 3Ware and LSI at the time. Currently, ATTO and Areca seem to dominate all the performance tests and the ATTO controllers are bullet proof. I'm being told the same for Areca and so far I'm liking this 1883i controller. I have a couple of the 6Gbps Areca controllers in other systems and they have been flawless.

 

[Jeff Kilgroe]

More notes on RAID performance:

RAID-5 and RAID-50 posting identical numbers around 880MB/s write, 820MB/s read.

RAID-6 730MB/s read and write in preliminary testing. All modes are useable, but I'm going to run with the array as a large RAID-0 and take the > 1GB/s read write speeds. So we'll see how it holds up over time.

System should be properly burned in now and temps are holding quite well. It's time to really work on the overclocking so I'll have more info shortly. It's been running at a conservative 3.8GHz, but the system already ramped my memory to 3000MHz even though it's technically rated at 2800MHz.

I think I'm going to order up another Titan X SC GPU. I'm running the system with a 30" Dell 2560x1600 display. Going to swap around monitors this weekend -- I have a couple new 27" 4K units coming in and it's time to retire a couple of these old 30" ones.

 

[Jeff Kilgroe]

Updated RAID numbers.... I should have given more time to the RAID-5 / 50 testing. Preliminary numbers for RAID-6 were not very accurate as the RAID must not have been fully initialized or the system was still indexing something. It's still in RAID-6 mode and is showing me the same 730MB/s or so read speed, but writes have improved to around 880~890MB/s. Running through 5K and 6K clips in Premiere is working great so far on this box. The RAID is not a bottleneck at all even as a RAID-6 volume. Still not sure what I'll do with it. It's plenty large as RAID-6 at 22TB real-world formatted capacity. But I'm still a performance junky. Anyway, I'm going to leave it as is for the time being and we'll see how it plays out so it's a RAID-6 for now.

Sleep issue seems to be common on these X99 boards with various overclock settings and there doesn't seem to be much success getting it all sorted out. No error codes or anything of that sort. Disconnecting the H100i cooler from the USB header did nothing. So it's back in place so I can monitor temps. I'm now clocked at 4.6GHz and going to hold here for a while and see what happens. I know I can push for more, but the dynamics will change a bit if I add that second Titan X...

 

[Jeff Kilgroe]

Just an update to this. The system is performing great, but I think the overclock limit is 4.6GHz in its current incarnation. Airflow seems good in the system and all the temperatures are are fine, but the CPU just can't be held stable beyond 4.6GHz and it fails in the form of a reboot or shutdown at 4.7GHz if I try to push the system and before temps climb too high. It might go higher if I fiddle with the voltage a bit more, but it looks like I'm right there with most of the faster overclock reports I can find. Some are getting 4.8 to 5.0 GHz with some extreme cooling and/ or what appear to be uncommonly good CPU samples.

At 4.6GHz, this thing is pretty snappy though. I'm at 4.3GHz on my 5960X build and it doesn't want to go any higher than that, but it's also air cooled with a big Cooler Master CPU cooling tower thing that sounds like a jet engine. Hate it. This build is nice and quiet, even those 6 x 6TB HDDs. I will have the second Titan X SC GPU on Tuesday.

Oh, and the single-port Thunderbolt EX II card works just fine in this system. I've questioned ASUS about the dual port, since they told me it would work, and now they're telling me they're still working on it and the plan is to get it working too. It just didn't seem like it was a huge priority, and I suppose it's not. They have some more pressing issues to fix with this motherboard and their UEFI platform, system boot process stupidity and whatnot. All mostly minor issues, but those pesky little details that always nag at you and never seem to get fixed.

I wish I would have built the system in the 750D case instead of the C70. Just for the extra bit of room to wrangle cables...

The read times are proving to be a bit lackluster on the RAID-6 array. Nothing that has caused me issues so far, but I have not tried any complex edit just yet. I'm betting it will choke up a bit. I'm going to run through a real project next week on the system and see how it goes. After that, I may switch back to RAID-0 and just take the performance over redundancy.

 

[Phil Holland]

The 750D certainly has that extra room on the rear for cable runs. That and the neat little 4X SSD trays sold me on the whole thing. Plus I could modify it for my needs without getting a hack saw.

I didn't try running the CPU above 4.6GHz with the air cooled setup either. Running now between 4-4.2GHz. And I'll settle on something around there.

The loudest thing in my particular setup is the bloody radiator. But it's no big deal. Oh. And the RAID card beeps silly loud if you do "the wrong thing".

 

[Ryan Nguyen]

Hi Jeff,

Been lurking your posts for some while now; finally got approved by the forum moderators to post something! :001_smile:
I've built a similar X99 Resolve workstation a couple of months ago, and am currently playing catch up as it relates to inflating my bottlenecks for the color grading work I do.
Having that said, I'm hunting down the Thunderbolt Card but for the life of me I can't seem to find it on the market anywhere. Do you suppose there's a specific reason for this?

 

[Jeff Kilgroe]

The Thunderbolt cards are sometimes a pain to get ahold of. Demand for them is actually quite low and thus most component vendors don't stock them and ASUS does not produce many of them. Same goes for the cards from Gigabyte.

If you're running an ASUS motherboard, then you will want the single-port Thunderbolt EX II card. The dual port card is apparently not compatible [at this time] with any of their X99 platform boards, but may be in the future.

There are a couple vendors who have them in stock right now on Amazon. They come and go on Newegg.com or other. Oddly enough, a good source for them seems to be Office Depot as they sell them online and occasionally have them in stores if you're near one with a larger PC section. Fry's and MicroCenter have them too, but not always in stock.

 

[Ryan Nguyen]

Thanks for the clarification Jeff! I had read that you've been having trouble with the dual-port. Sorry to hear that.
IIRC, did you mention at some point that the dual-port card does not feature true independent TB2 bus lanes, but that they share the bandwidth?

As for the single port card, I just managed to grab one off Amazon. Phew, finally! I had been hunting it for a couple of months now.

 

[Jeff Kilgroe]

The dual-port card only uses a single TB2 header, so they share bandwidth. It can provide discreet video I/O as there are two display port connections for routing, but the card itself operates as a PCIe v3.0 X2 device. So you have PCIe v3 X2 (1970MB/s) bridged to PCIe v2 X4 for the TB2 host. Each discrete TB2 host needs its own PCIe v2 X4 (2000MB/s) link for full bandwidth.

 

[Jeff Kilgroe]

Frustration....

It seems I will have to choose between running multiple GPUs and using the Thunderbolt card in this system. Thanks, ASUS, for being complete dumbasses with your PCIe lane allocations. I saw this coming, but had numerous assurances otherwise. Trying to see if there's a way around it, but it seems that for multiple GPUs to work properly, the primary GPU *MUST* reside in slot-1. Then other GPUs must cascade downward from there in slot 3 next, then 5 for 3 GPUs, then slot 7 if you run quad GPUs. Other PCIe cards be damned. Installing the second Titan X in slot 5 or slot 7 results in the system refusing to post. If I move the RAID card to slot 5 and put a Titan X card in Slot 1, which means the Thunderbolt card is removed, then I'm off to the races.

So I may just have to live without Thunderbolt on this system. No big loss, but would be nice to swap around those Pegasus RAIDs I've been using or connect an mTape unit. I have two free SAS connectors on the Areca card, not that I want to spend more money, but I could drop an LTO-6 unit into my free 5.25" bay or get one of the new mTape Connect units that offer USB3 connectivity in addition to SAS and Thunderbolt.

 

[Eric Bowen]

The Dual GPU setup works fine with the cards in Slots 3 and slots 5. You have to have the latest bios or the bios before although that one was somewhat sketchy resource wise if you dont have above 4G decoding enabled in the bios under boot. If the TB card is already in the you have to remove the 2nd video card long enough to enable the 4G decoding if it's not working.

 

[Jeff Kilgroe]

Heh. Less than 10 minutes later I have it working... <knock wood>

ASUS needs to seriously address their BIOS issues. I want to say it's complete rubbish, but they're still better than Gigabyte and I don't think anyone could pay me to use an ASRock or MSI motherboard right now. Would also help if UEFI in PC land wasn't so schizophrenic when trying to co-exist with legacy BIOS systems.

FWIW, the TB card is back in slot 2 where it should belong. However, I was able to get it to also work in slot 7. But only if slot 6 is unoccupied since the two slots are apparently linked for lane bandwidth.

Not sure why it's deciding to work now. I reset the BIOS to defaults and then made all the appropriate changes again and after a few boots it came up. Int19 has to be postponed and yeah, the above 4G decoding enabled...