Qmaster & Compressor Troubleshooting Tips

[michael zaletel]

Hi Everyone:

After hours of searching forums and several phone calls with top Compressor tech support at Apple, I have compiled a list of "things to try" whenever you are having trouble with Qmaster and Compressor.

Routers, Switches and IP Addresses.
1. DHCP is preferable over manual IP addresses.

2. On all cluster nodes, in System-Preferences --> Apple Qmaster --> Advanced Tab you want to set "Use Network Interface" to Ethernet or Airport depending upon which you are using not to "All Interfaces" which would allow it to switch around.

3. Make sure File Sharing is turned on under Sharing in System Preferences and make sure the Computer Name does not have spaces or strange characters such as apostrophes. Give each computer a UNIQUE name.

4. Make sure all Qmaster node computers are plugged via Ethernet into the same network switch if at all possible and that your Airport Extreme or Cable Modem, etc. is plugged into that switch. If you must daisy chain switches, test first with one switch to see if working, then re-route, restart and if doesn't work, that could be your problem.


Qmaster Preferences Pane Settings
1. On all cluster nodes, in System-Preferences --> Apple Qmaster --> Advanced Tab, under Extras, make sure the name set for "Identify this computer to Qadministrator as:" is all one word with no spaces or strange characters. Dashes probably okay. Do the same under QuickCluster on Setup Tab even if that computer won't be a QuickCluster. Give each computer a UNIQUE name.

2. On Qmaster Advanced Tab, Make sure "Shared Cluster Storage" directory has not been changed from default. It should be "/var/spool/qmaster". If it has been changed, stop sharing, click set, then click Shift-Command-G and type in the default directory name.

3. Under Setup Tab, one computer should be set as QuickCluster with Services and "Share" boxes should be checked for Distributed processing and shake but do not click the "managed" boxes. Other computers should have same boxes checked but "Share this computer as:" should be set to "Services Only"

4. Under options for selected service, you can choose more than one processor if you have a dual-core, quad-core or 8-core machine. Best to leave an 8-core set to 7 instances and a quad to 3 instances but with duo-core laptops and etc. it's fine to set to 2 instances to use both processors.

Reset and Restart *This is VERY IMPORTANT
1. I was having trouble getting all of my nodes to participate in a Compressor Batch Submit job just yesterday (even though all was fine in the past). Could have been a change to my network, a software update from Apple or a variety of other unknown conditions. What I learned from the Apple tech is that there is a very specific method, process and order in which to resolve such issues.

a. On all Qmaster nodes in System Preferences --> Qmaster -->Setup Tab click "Stop Sharing". Once you've done that on every computer (really helps if you are running Apple Remote Desktop btw), then going back to the first computer you stopped sharing on, mouse over the "Start Sharing" button, hold down the option key and the button will change to "Reset Services" and click that. Do the same for every other Qmaster Node computer on the network. Make sure you do not forget about that rogue laptop or desktop in the other room that is on the wireless network and has Qmaster sharing turned on for some strange reason.

b. Then SHUT DOWN (not restart) all computers on your network. Once all are turned off, it's a good idea to unplug (power only) cable modems, network switches, airport extremes and etc. and power those back up. Once network lights are green, then TURN ON all computers on the network.

c. Go into the control panel for the computer that will serve as your QuickCluster and START SHARING on that computer first. Then go to each of your other computers on your network and START SHARING for the nodes defined as Services Only.

d. Finally, open Apple Qadministrator (search for "qadministrator" in finder) and check to make sure all computers are showing up under Qmaster Service Browser and that your QuickCluster is listed under Cluster at top.

Other Troubleshooting Tips
1. Some people have reported serious issues with "Back to my Mac" under Mobile Me in System Preferences. If you use that service, you may have to turn it off when you want to use your Qmaster Cluster. Some people have reported that was the only way they could fix their cluster issues. The Apple tech I spoke to yesterday confirmed that Back To My Mac shares some networking components/technology with Qmaster and it's reasonable that there could be a conflict there.

2. Hard Drives and Computers and File Paths should not have spaces or strange characters like commas or apostrophes. Ie. avoid a drive name like "Bob's Hard Drive". Instead change it to "BobsHardDrive" or something. Same with network names, Qmaster node names and computer names and file directories.

3. Make sure all computers have the same operating system version OS 10.5.6 as of today. Same Compressor Version 3.0.5 as of today, (get info on file to see) and same Qmaster Administrator (also 3.0.5). Having different OS or Compressor or QMA versions on different computers could cause problems.

4. For Qmaster and Compressor to work properly, the source and destination volume must be mounted on the desktop of all computers with all read and write priviliges. (Under Finder --> Preferences, click the checkbox next to "Connected Servers" to make the network mounted volumes show up on your desktop.

5. Sometimes you may have to leave the computer that the source files (and destination target, hopefully the same computer volume) reside on out of the cluster because there could be issues resolving the network path. Remember that a drive on computer number 10 has the same path from computers 1-9 but the path to that drive from computer 10 is different.

In summary, Qmaster can be finicky so it's a good idea to reset all possible points of failure, turn off qmaster sharing, reset services, turn off all computers, switches and modems, then turn everything back on, once cable modem and switch shows live internet, turn all computers back on and let Apple's OS work out the complex network issues, assigning IP addresses through DHCP and etc. then start sharing qmaster services (quickcluster node first). Check your computer names, sharing names, volume names, directory names and make sure no issues there with spaces and strange characters.

Most of all, if you have anything to add to this list, please, please post it here to help make everyone's lives easier.

Take Care,

-michael zaletel
(shooter)

 

[michael zaletel]

It's also a good idea to set Cluster Options in Compressor Preferences to Never Copy to Cluster.

Compressor --> Preferences --> Cluster Options "Never Copy Source to Cluster"

And here's another LONG LIST of Troubleshooting tips I just found that can help with Qmaster, Shake, FCP and Compressor:

http://discussions.apple.com/thread.jspa?threadID=951536

-michael zaletel
(shooter)

 

[michael zaletel]

One last point. If you are using Qmaster with Shake, you may need to make the computer that the source and destination files reside on (and the shake script) your Qmaster QuickCluster and set the "Shared Cluster Storage" directory to be the same as the volume mounted on all your nodes using Shift-Command-G to specify absolute path like /Volumes/Cluster-RAID/Temp. You will also need to set the working folder in the RenderFileOut dialog in Shake to be on that same network accessible volume using /Volumes/Cluster-RAID/shake-temp or something.

Point being that most problems with Qmaster, Compressor and Shake relate to network communication, IP Addresses, paths to the files and etc. In other words, in a render farm, what's most important is that all of the computers sharing the rendering or compression tasks can communicate consistently with one another and that all of them can find the source file, know where to write the destination file(s) and have access to the same temp and working directories.

-michael zaletel
(shooter)

 

[danbrazda]

One last point. If you are using Qmaster with Shake, you may need to make the computer that the source and destination files reside on (and the shake script) your Qmaster QuickCluster and set the "Shared Cluster Storage" directory to be the same as the volume mounted on all your nodes using Shift-Command-G to specify absolute path like /Volumes/Cluster-RAID/Temp. You will also need to set the working folder in the RenderFileOut dialog in Shake to be on that same network accessible volume using /Volumes/Cluster-RAID/shake-temp or something.

Point being that most problems with Qmaster, Compressor and Shake relate to network communication, IP Addresses, paths to the files and etc. In other words, in a render farm, what's most important is that all of the computers sharing the rendering or compression tasks can communicate consistently with one another and that all of them can find the source file, know where to write the destination file(s) and have access to the same temp and working directories.

-michael zaletel
(shooter)

Again, thank you Michael for taking the time to post this. I will be testing all of your points ASAP and post my results.

I can already anticipate a question though: I have 6 Intel Mac Workstations on the same network (in addition to 4 additional G5 Power PC stations dedicated to rendering only) and would love to be able to take advantage of all of the station's processing power FROM any station at any time. Am I right in assuming that I should allocate a central storage area on the network that can always be mounted on all workstations and then creating a cluster that is unique to each workstation that does not include the host workstation that the render originates from?

Wow, that was exhausting even typing but I don't think it would be that hard to do. Just wanted to confirm my logic before proceeding.

 

[michael zaletel]

Am I right in assuming that I should allocate a central storage area on the network that can always be mounted on all workstations and then creating a cluster that is unique to each workstation that does not include the host workstation that the render originates from?

Hi Dan:

That's exactly how I'm solving the same problem. I have a RAID attached to one of my old G5's and I mount that RAID Volume on every computer. That computer is the QuickCluster controller and I set it's temp directory as an absolute path to that RAID drive as in /Volumes/RAID. Then I make sure my shake scripts and Compressor use that same absolute network volume path for source and destination or in the case of Shake filein and fileout nodes. Finally, in the Render FileOut Dialog box in Shake, you have to set the working folder to a folder on that same RAID absolute path. There may be other ways to make this all work but this seems to be the best approach.

For Shake, you also may need to edit or add a few .h startup files which isn't all that complicated. Finally, you want to make sure you set the appropriate number of instances for both compressor and rendering in Qmaster Preferences. Use 6 or 7 max for 8-Core machines but I use 2 for all Dual-Core machines like MacBook Pro.

Next I'll be experimenting with Link Aggregation which allows you to combine two Gigabit Ethernet ports to double the transfer speed assuming both computers have fast enough read/write hard drive speeds and both have two Gigabit Ethernet ports. I bought a dual-GigE card for my G5 but MacPro has two on board. Not sure if there's a PCIexpress card for MacBook Pros so you can have two GigE ports on those.

-michael zaletel
(shooter)

 

[michael zaletel]

Those reading this thread, please note there is another thread with additional discussion points and troubleshooting tips here:

http://reduser.net/forum/showthread.php?t=30192&page=2

-michael zaletel
(shooter)

 

[Kaku Ito]

Michael,

Thanks for this thread.