Welcome to our community

Be a part of something great, join today!

Register Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

AMD Threadripper builds for compositing?

PatrickFaith

PatrickFaith

Well-known member
Joined
Nov 4, 2011
Messages
2,560
Reaction score
0
Points
36
Location
California
Website
www.pudls.com
I am finding I don't have any problems keeping up with r3d 8k anymore, it's getting so fast with all the RED improvements. After I get to a DI though, rendering and compositing often fall apart at higher res. I have noticed the processing is happening fairly evenly across the cores, and the GPU's are not the bottle neck.

I have been looking at the 32 core 2990wx amd chip, but I have never done a amd build. Also these machines run contiously for months, and I would like to try liquid cooling(the sound of machines running continously is also irritating me, so I am looking for things a bit quieter). This will mainly be running Arnold and Fusion/davinci, probably windows 10 but I'm pondering centos os.

Any suggestions are appreciated, exact build specs especially. I am kind of brain storming on this, so tangentential approaches of dealing with high res compositing are very welcome.

[ I am currently on dual-10core hp workstations, but I am trying also to be a bit budget minded on this build also. I'd really like to keep the budget under 5k per machine, and get two workstations - I have disk farms so also am having no worries on disk speeds and am finding I like a single large ssd per machine for local caches.]

[also I noticed this thread ... http://www.reduser.net/forum/showth...core-cpu-for-Red-files&highlight=Threadripper ... but as I said I am not having a r3d rendering problems so not worried at all about r3d]
 
I just built two new work stations and put lot of effort into it and also got took help from some one that´s possibly the best I ever came across when it comes to set up good work stations for compositing.

I also thought of TR at first but got a quite convincing speech to go for the i9´s instead.

Watercooling is not more quite than fan cooling. If you don't attach pooring water or got a big tank or such. Simply the water coolers are better to bring the heat away from the CPU but they also got fans that actually breaths straight out of the box so more likely they even make more sound than a fan cooler that sits more central in the box. But best possible cooling is very much to strive for so yes go of a water cooler even the not so good ones are better than the good fan coolers.

If going with the i9 buy this one: https://www.caseking.de/en/search?sSearch=der8auer

It´s ready and licensed for overclocking to 4,7 Mhz thats pretty much twice the speed of the stock version and even if you don't overclock it will stay cooler and run much smoother than the stock version.

Best as I have right now , got two of these and two with pretty much the same setup but Asus. I like both but the gigabyte card is on board TB3 and 3x M2 instead of asus, only two M2 slots and Asus has their deluxe card which only has one TB3 and only as a PCIe card. However I would run both with a real Alpine ridge card. As that works way better for hackintosh.

So my recommendation would be, not to go for the thread ripper but intel i9 as follows:

Gigabyte X299 DESIGNARE EX
Intel Core i9 7980XE X-series - 2.6 GHz - 18-core 36 threads
Dual 1080ti GPU
Gigabyte GC Alpine Ridge Rev 2.0 Thunderbolt 3 Certified Add-in Card (not really needed for PC use then the two built in ports will do just fine)
Ballistix Sport or any type fast 8x 16 GB
CORSAIR AX1600i - PS.
Samsung 970 EVO MZ-V7E1T0BW - Solid state drive - 1 TB
Corsair Crystal Series 460X RGB - Miditower


I would go for two or more 1080 cards, instead of the 2080ti´s as you get good bang for buck and the newer cards does not speak hackintosh just yet.
 
water cooling is a lot of fun and really not that hard. There are plenty of youtube channels that will run you thru every step. With enough radiator space you can run the hottest systems with low fan speeds. Hit me up if you need some help sourcing cooling parts.

Are your disk farms in racks. Do you want your computer rack mounted as well?
 
Björn Benckert said:
...

Watercooling is not more quite than fan cooling. If you don't attach pooring water or got a big tank or such. Simply the water coolers are better to bring the heat away from the CPU but they also got fans that actually breaths straight out of the box so more likely they even make more sound than a fan cooler that sits more central in the box. But best possible cooling is very much to strive for so yes go of a water cooler even the not so good ones are better than the good fan coolers.

Best as I have right now , got two of these and two with pretty much the same setup but Asus. I like both but the gigabyte card is on board TB3 and 3x M2 instead of asus, only two M2 slots and Asus has their deluxe card which only has one TB3 and only as a PCIe card. However I would run both with a real Alpine ridge card. As that works way better for hackintosh.

So my recommendation would be, not to go for the thread ripper but intel i9 as follows:

Gigabyte X299 DESIGNARE EX
Intel Core i9 7980XE X-series - 2.6 GHz - 18-core 36 threads
Dual 1080ti GPU
Gigabyte GC Alpine Ridge Rev 2.0 Thunderbolt 3 Certified Add-in Card (not really needed for PC use then the two built in ports will do just fine)
Ballistix Sport or any type fast 8x 16 GB
CORSAIR AX1600i - PS.
Samsung 970 EVO MZ-V7E1T0BW - Solid state drive - 1 TB
Corsair Crystal Series 460X RGB - Miditower


I would go for two or more 1080 cards, instead of the 2080ti´s as you get good bang for buck and the newer cards does not speak hackintosh just yet.
Click to expand...
I have been pondering going thunderbolt 3, thanks for the tip. On the power supply, how loud is that if only running one 1080 (you still like that power supply)?
Editorial compositing is happening on workstation like you describe using intel cpus, I have noticed a lot of single core bottlenecking for that so the boost is important - but this is more for secondary workstation and long running scheduduled jobs (i.e. jobs that might go a month). The problem I'm having is compositing and rendering starts getting a huge backlog of render jobs which is holding up editorial.

Chad Lancaster said:
water cooling is a lot of fun and really not that hard. There are plenty of youtube channels that will run you thru every step. With enough radiator space you can run the hottest systems with low fan speeds. Hit me up if you need some help sourcing cooling parts.

Are your disk farms in racks. Do you want your computer rack mounted as well?
Click to expand...

I have about 400 tb running in a rack, I was pondering putting the compositing workstations into a rack, but they might have to be moved between rooms and run as primary workstations from time to time.

I do think I will go with liquid cooling so I can keep the fan speeds lower. I was thinking of getting one of those three fan radiators and just have the fans barely going. I haven't bought a liquid system in 7 years, but I've done quite a bit with heat pipe and liquid systems like 40 years back. So on the liquid system, I was also thinking of getting a large case so I don't have to smash the pipes, any suggestions on the cooling systems (exact models, etc) would be nice. I think I'd prefer to get a liquid system that has a small tank reservoir with routable pipes, where I can see the fluid flow and levels.
 
PatrickFaith said:
Also these machines run contiously for months, and I would like to try liquid cooling(the sound of machines running continously is also irritating me, so I am looking for things a bit quieter). This will mainly be running Arnold and Fusion/davinci, probably windows 10 but I'm pondering centos os.

Any suggestions are appreciated, exact build specs especially. I am kind of brain storming on this, so tangentential approaches of dealing with high res compositing are very welcome.
Click to expand...

When it comes to CPU rendering nothing beats the 2990wx at the moment (CB R15 MC 5600 point at 3.6 GHz-all-core with an air cooler).
When you run these machines continously for months it might be wise to go for ECC-UDIMM (something like this to start with https://www.newegg.com/Product/Product.aspx?Item=9SIA7S67BJ1651) and yes they are easy to overclock to make them faster (just raise the voltage to 1.35 volt and you can lower the CL and raise the freq.)
For cooling pick an enermax liqtech TR4 https://www.newegg.com/Product/ProductList.aspx?Submit=ENE&IsNodeId=1&N=100008008%2050001400%20601303056

For the GPU it's often a choice of how much memory you need, professional drivers for certain applications, 10 bits overlay color in OpenCL, CUDA-only applications, price, etc..

For the motherboard you have many options, we went for the ASRock Fatal1ty X399 Professional Gaming sTR4 AMD X399 SATA 6Gb/s USB 3.1 ATX AMD Motherboard and it turn out to be the best compatible motherboard for different kind of memory and different OS (MacOS, windows10, CentOS, Ubuntu, FreeBSD and Fedora). At the time of buying last year our main reason was the built-in 10G-ethernet and 8 on-board sata ports.
https://www.newegg.com/Product/Product.aspx?Item=N82E16813157784

When you like long, kind of boring video about Threadripper, have a look at Wendle form Level1techs https://www.youtube.com/results?search_query=level1techs+threadripper

Der8auer also kind of knows how to get the most out of the Threadripper (and he also knows how to overclock EPYC).

 
To me rack mounted boxes is just a hassle Better to put shelfs in the racks and put them 2 and 2 side by side. That way you can easily slide them out if needed. Most rack mounts comes with some over elaborate sliders that needs to be mounted and only fit the actual box they where bought with. I don't know how many machines thats moved in an out of these cabinets but the shelfs have stayed the same. Now it´s all a bit messy as I just changed lot of things, but you get the idea. Not much space lost if having the boxes standing up.

IMG_20181229_211358 by Björn Benckert, on Flickr
 
PatrickFaith said:
I have about 400 tb running in a rack, I was pondering putting the compositing workstations into a rack, but they might have to be moved between rooms and run as primary workstations from time to time.

I do think I will go with liquid cooling so I can keep the fan speeds lower. I was thinking of getting one of those three fan radiators and just have the fans barely going. I haven't bought a liquid system in 7 years, but I've done quite a bit with heat pipe and liquid systems like 40 years back. So on the liquid system, I was also thinking of getting a large case so I don't have to smash the pipes, any suggestions on the cooling systems (exact models, etc) would be nice. I think I'd prefer to get a liquid system that has a small tank reservoir with routable pipes, where I can see the fluid flow and levels.
Click to expand...

a 360 or 420 rad (120mm or 140mm fan x 3) won't be enough for a hot processor with 2 gpu to run with a less aggressive fan curve. Minimum 360 per loop just for basic cooling (if running dual loops).
if you want a flashy case to show off the loops check out singularity computers spectre https://www.singularitycases.com/product/spectre/
Radiators/blocks - EK https://www.ekwb.com/
Reservoirs/pumps - Singularity Computers Protium https://www.singularitycomputers.com/product-category/watercooling/
loop/fan control - Aqua Computer aquaero/accessories https://shop.aquacomputer.de/index.php?cPath=62_64&XTCsid=nu2u9gpek7mgcq09tb9smejj5e3avhlo
connectors - ek and/or bitspower

if you wanted something rackmounted it would take a little modding or designing a custom case with someone like protocase
https://youtu.be/ccp5KtSD_lk
 
Misha Engel said:
...

For the motherboard you have many options, we went for the ASRock Fatal1ty X399 Professional Gaming sTR4 AMD X399 SATA 6Gb/s USB 3.1 ATX AMD Motherboard and it turn out to be the best compatible motherboard for different kind of memory and different OS (MacOS, windows10, CentOS, Ubuntu, FreeBSD and Fedora). At the time of buying last year our main reason was the built-in 10G-ethernet and 8 on-board sata ports.
https://www.newegg.com/Product/Product.aspx?Item=N82E16813157784
Click to expand...
I will look into that motherboard, what is super important to me is 10g ethernet … and having that with the board is a huge improvement to adding cards and dealing with device drivers.

Björn Benckert said:
To me rack mounted boxes is just a hassle Better to put shelfs in the racks and put them 2 and 2 side by side. That way you can easily slide them out if needed. Most rack mounts comes with some over elaborate sliders that needs to be mounted and only fit the actual box they where bought with. I don't know how many machines thats moved in an out of these cabinets but the shelfs have stayed the same. Now it´s all a bit messy as I just changed lot of things, but you get the idea. Not much space lost if having the boxes standing up.
Click to expand...
I think your right, I'm just going to go with workstations. Also liquid cooling on servers can be a total pain.

Chad Lancaster said:
a 360 or 420 rad (120mm or 140mm fan x 3) won't be enough for a hot processor with 2 gpu to run with a less aggressive fan curve. Minimum 360 per loop just for basic cooling (if running dual loops).
if you want a flashy case to show off the loops check out singularity computers spectre https://www.singularitycases.com/product/spectre/
Radiators/blocks - EK https://www.ekwb.com/
Reservoirs/pumps - Singularity Computers Protium https://www.singularitycomputers.com/product-category/watercooling/
loop/fan control - Aqua Computer aquaero/accessories https://shop.aquacomputer.de/index.php?cPath=62_64&XTCsid=nu2u9gpek7mgcq09tb9smejj5e3avhlo
connectors - ek and/or bitspower

if you wanted something rackmounted it would take a little modding or designing a custom case with someone like protocase
https://youtu.be/ccp5KtSD_lk
Click to expand...
Man this is some nice stuff, starts me thinking of doing a wall mounted workstation (I've seen some really nice office setups on wall mounts). Might be a bit pricey, but would be good to get a case where I'm not throwing it out every few years and can just re-use the case for new motherboards.

Misha Engel said:
Something like this: https://pcpartpicker.com/list/8WF3XP

The memory is ECC-UDIMM for 24/7/365
GPU is also ECC for 24/7/365
Click to expand...
I love this site pcpartpicker, haven't heard of it before but will be for sure using it, much thanks!
 
I started building a 2990wx system. I decided to go Linux, but trying to decide centos or Ubuntu. I heard with centos I would need to upgrade the kernel, but that wasn't necessary with Ubuntu. Also I will have a 2080 in the box, but I guess I will have to forgo r3d transcodes in Linux? My entire shop will slowly be flipping Linux, I guess I could keep some old machines for r3d transcodes, but then the r3d boxes will be using the oldest gpu's I have.
 
PatrickFaith said:
I started building a 2990wx system. I decided to go Linux, but trying to decide centos or Ubuntu. I heard with centos I would need to upgrade the kernel, but that wasn't necessary with Ubuntu. Also I will have a 2080 in the box, but I guess I will have to forgo r3d transcodes in Linux? My entire shop will slowly be flipping Linux, I guess I could keep some old machines for r3d transcodes, but then the r3d boxes will be using the oldest gpu's I have.
Click to expand...

Check this website every now and then to stay updated about Linux https://www.phoronix.com/scan.php?page=home

And for the RED stuff https://www.red.com/downloads/options?itemInternalId=16146

For the same price as the 2080 you can have a 16 GByte card Vega 2/Radeon 7 https://www.amd.com/en/products/graphics/radeon-vii some extra memory is always nice to have with your kind of workloads (unless you need CUDA).
 
Misha Engel said:
Check this website every now and then to stay updated about Linux https://www.phoronix.com/scan.php?page=home

And for the RED stuff https://www.red.com/downloads/options?itemInternalId=16146

For the same price as the 2080 you can have a 16 GByte card Vega 2/Radeon 7 https://www.amd.com/en/products/graphics/radeon-vii some extra memory is always nice to have with your kind of workloads (unless you need CUDA).
Click to expand...
Excellent info. I just tried to install fusion and the red libraries on ubuntu, was having a bunch of problems. Researched the sites you referenced, I think it looks like centos 7 is the best bet for me. Much thanks for the leads.

[edit ... so far centos has been a total PAI too ... the nvidia drivers are tricky and there is so much "manual" junk I have to do by just almost randomly trying things from the internet ... the davinci resolve instructions which are suppose to work for centos are also incomplete. I really like ubuntu since I haven't had to deal with driver issues, going to centos I feel I dropped back in time 10 years]
[EDIT 2 - GOT IT WORKING ... was something called a opensource nvidia packaged called nouveau, which I had to very carefully disable, then reinstall the nvidia drivers. Got Fusion running inside davinci with the GPU too]
 
Last edited:
Hey Patrick,

I just built my PC for the exact same reason. With the new updates and BETA builds for RCX, the GPU will be taking majority stakeholder in the rendering department for R3Ds. Unless you are going to do heavy CGI render scenes or complex machine learning, I wouldn't get the top of the end CPU.

MSI X399 CREATIVE
AMD 2950x CPU - 3.5 GHz - 16-core 32 threads
Nvidia Titan RTX GPU
Enermax - LIQTECH TR4 II 360 102.17 CFM Liquid CPU Cooler
G.Skill Trident Z RGB - 4x8GB (32GB) - 14-14-14-34 Samsung B-Die
SeaSonic - PRIME Ultra Titanium 1000 W 80+ Titanium
Samsung 970 EVO - Solid state drive - 1 TB
Thermaltake - View 71 TG RGB ATX Full Tower Case
 
Shaeden Gallegos said:
Hey Patrick,

I just built my PC for the exact same reason. With the new updates and BETA builds for RCX, the GPU will be taking majority stakeholder in the rendering department for R3Ds. Unless you are going to do heavy CGI render scenes or complex machine learning, I wouldn't get the top of the end CPU.

MSI X399 CREATIVE
AMD 2950x CPU - 3.5 GHz - 16-core 32 threads
Nvidia Titan RTX GPU
Enermax - LIQTECH TR4 II 360 102.17 CFM Liquid CPU Cooler
G.Skill Trident Z RGB - 4x8GB (32GB) - 14-14-14-34 Samsung B-Die
SeaSonic - PRIME Ultra Titanium 1000 W 80+ Titanium
Samsung 970 EVO - Solid state drive - 1 TB
Thermaltake - View 71 TG RGB ATX Full Tower Case
Click to expand...
That looks like an awesome setup, also the 2950 is actually better for windows configuration then what I got. I am 90% post production in Linux and am almost exclusively exr for my DI, so the 32 cores makes sense for my workload of mainly compositing & rendering.
 
PatrickFaith said:
That looks like an awesome setup, also the 2950 is actually better for windows configuration then what I got. I am 90% post production in Linux and am almost exclusively exr for my DI, so the 32 cores makes sense for my workload of mainly compositing & rendering.
Click to expand...

Thanks! Oh perfect, that definitely makes sense for you :) Linux looks to be awesome.
 
The build is going fine except my cooling controls are horrible with centos. The fan speeds seem fine under centos, but I have no access to what is going on with the liquid system in centos. [is the pump speed constant and the cooling speeds based on fan speed?] Anyway, none of the interfaces of the cooler are available in centos. Also my current cooler pump is off of usb, and I don't like it's settings and can't modify them inside of bios or centos utilities. I guess I need to go with one of the more complicated "pumps" for centos, are there any recommendations for pumps that work well with centos but are newbie friendly? I've searched around the internet but haven't found a good spot for people doing 2990 with centos that go over liquid fan stuff in detail? I guess I need to go with some reservior ones like Chad was talking about, but I'm having a bit of time figuring out how to do this for centos ...

[i btw got a self contained liquid cooling unit for my rtx 2080, and the gpu cooling seems to be working fine]
[It's btw pretty cool in unix shell having 32 cores/64 process threads available. ]
 
Pump is constant speed and the fans on the radiator are speed controlled.
 
You must log in or register to reply here.
Back
Top