Mac Pro with D500 and D700 overheating on exports

[Will Duncan]

2016 marks the first year that I'm working on larger, feature length 4k projects with both Premiere and Resolve.

For the most part, our infrastructure is Mac, mostly 6-core trash cans with a mix of d500 and D700 GPU's and some iMacs sprinkled in. Playback and editing are fine up to 5k (exclude a climbing count of crashes with Adobe Premiere that editors are reporting), specifically when editing R3D footage. Most of that stability is probably due to 10GbE network with some large Terrablock SAN systems.

But then comes the export. Once the GPU temps climb, we start seeing a lot of artifacts and problems in the output files, horizontal green and red lines (a few pixels thick) are the most common. It wasn't an issue during 1080p exports, at least not regularly. Smaller projects never showed problems. It wasn't until we started working on longer length TV and feature length content (40 minutes +) that we started seeing the issues regularly.

All footage is ProRes besides R3D files. We keep those native now that our archive system supports it.

I've read about similar problems on other forums for over a years, but I churned through a few pages and Mac Pro threads in here without much mention.

Is anybody having similar issues exporting from the nMP? Over on CreativeCow a few people had their GPUs swapped out multiple times before the problem was resolved, but that sounds like a long shot full of hope for a post-house constantly in production. Not to mention that's going to take a ton of coordination pending Apple agrees to replacing GPUs in multiple machines.

 

[Jon Dishler]

curious ambient temp of mac trash can environment? Also, I have regularly taken cover off of mine and vacuum dust which slows the air flow. Longest project though has been <40 minutes.

 

[Will Duncan]

Ambient temp is variable. The facility doesn't have temperature control for edit bays, so it ranges from 65 to 75 on average I'd say, but at night during sustained exports (multiple hours) it's hard to say. When the Mac Pro fans kick in, ambient temperature can rise a great deal. And the air conditioning doesn't run at night in the building.

During initial issues on 1080 and 720 exports for TV content, we used iStat Menu and cranked the fan speed manually, which resolved the artifact and random horizontal line issues. But on a 4k export, it resolves 70%, maybe more. Without the fan speeds pegged manually, there are countless hits.

As a workaround, specifically for Resolve exports, exporting as an image sequence is a solution. But that adds 6-9 hours on a feature length project.

I haven't had the opportunity to place a Mac Pro in our data center, turn the temperature down and see if that helps. Most post-production facilities I've worked in are borderline cold, but this one doesn't have that kind of control. It's a huge office complex and the building is on central system. On or off, with very little temperature control.

 

[Jon Dishler]

I am no expert, but my office building turns off the A/C at night and it gets pretty stuffy, maybe you can get a cheap recording thermometer to put in there at night, at least one that tells min/max temp -- but if computers get too hot, they get wonky for sure. Also, my fan was running constantly until I cleaned out the dust.

 

[Eric Santiago]

Using D500 and D700 nMP for export from Resolve to 4K DCI ProRes4444 and no issues.

These are full feature as long as 1.5 hours.

After Resolve we use Media Encoder sometimes FCPX/Premiere Pro to h264 for simple trailers, samples and dumb down versions.

But the clips are mostly from Resolve exported clips so no crazy GPU action from the NLEs.

I myself do a ton of Maya/Rman to AE compositing but small snipets at 4K.

Mosty CGI support for the full feature.

All of the above are 99.9% R3D workflows.

My studio is a small bedroom and wished it had some HVAC help.

 

[Jeff Kilgroe]

There is a GPU recall on some 2013 Mac Pro trash can systems, so look into that. However, green line and odd pixel artifacting is a known problem with this specific GPU family. It mostly happens within Adobe Premiere and AME, but can sometimes happen in other apps like Resolve, mostly after things heat up. Not just limited to the Mac Pro, this same issue also exists in the same GPU family on PC video cards. AMD seems to have addressed much of the issues on the PC with drivers, but Premiere is still a problem here as they don't do buffer flushing in their rendering code and we see persistent garbage data. The fixes we have seen in the Windows driver set have not made it over to the Mac. I have not worked with Mac OS Sierra (10.12) much on the trashcan Macs and no longer have any trashcan Macs to play with, so can't comment there. But as of the latest release of 10.11 (Yosemite), the AMD GPU drivers are still poorly optimized and lag well behind the Windows drivers in terms of fixes and functionality. Adobe, Apple and AMD are all aware of GPU issues on these Mac Pro systems with Premiere and a few other applications and there has been a lot of finger pointing at one another over the past few years.

That said, to be perfectly blunt about this, I think Adobe is the one that needs to fix these issues with their software. It may be related to sucky AMD drivers, but nonetheless they have the biggest problem with this issue. FCPX does not have such problems and runs great on the cylinder Mac Pro systems. Resolve runs well 99% of the time without the same issues. If you do experience these issues in Resolve, then that may point to the GPU recall issue or a separate overheating issue. -- Make sure your systems are cleaned and you have good airflow in the work environment. The central "chimney" of a Mac Pro can get very dusty and many of the problems happen more toward the base. Make sure it can pull air in freely all around its base.

 

[Will Duncan]

@Eric — I have one machine, the first one we purchased, that doesn't show the same symptoms as the others. With that said, anything that's a fairly clean timeline (one two cameras, a few audio tracks) has been pretty stable across the board. TV projects include 12-24 multicams with a mix of R3D cameras in packages. Those were the first projects we saw issues with.

@Jeff — Thanks for the detail. I wasn't particularly familiar with the GPU family until we purchased the nMP. We went almost 2 years without major problems, but we weren't doing anything incredibly heavy either.

 

[PatrickFaith]

I think on liftgammagain.com Marc recommended using a fan control program. So I am using crystalidea.com fan control on both of my mac pros (12core d700 and 8core d500), I basically put the fan now close to max and haven't had any problems after I've done that. I do think if the part has overheated a lot before you used the fan control, that it's too late to use the fan control and it basically has to be serviced. I have no idea why the fan control/heating design is so crappy on the mac pros without getting these "extra" programs to control it.

 

[Marc Wielage]

That said, to be perfectly blunt about this, I think Adobe is the one that needs to fix these issues with their software. It may be related to sucky AMD drivers, but nonetheless they have the biggest problem with this issue. FCPX does not have such problems and runs great on the cylinder Mac Pro systems. Resolve runs well 99% of the time without the same issues.

That is not true in my thousands of hours of experience over the past 2-1/2 years. There are many, many, many GPU glitch issues with Resolve exports. Heck, the GPU sometimes craps out just in playback. And this is mostly 4K projects being monitored in HD. I've also seen HD projects monitored in HD that fall apart. I blame AMD for making crappy chips, and I blame Apple for building a small cabinet that doesn't provide enough cooling.

I'm using Macs Fan Control (same link as Crystal Idea above), which has been fairly robust and accurate. I also keep an eye out on the overall system temperature as well as the temp of GPU #2, which is the main one that fails. Many renders wind up with the computer running at close to 150° F, and the GPU gets close to 180°F... and that's with the fan cranked to 1800RPM.