Qmaster

[Trevor Meier]

I realize Build 16 is coming... but I need something that works today (until B16 and it's tools are ready for primetime).

I'll likely end up using Xgrid or some other command-line dispatcher with Redline, but Qmaster is so tantalizing, I wanted to give it a shot.

I've experimented with the Qmaster support in RedAlert, rendering 853x480 SD Prores files (nice & small for testing). Here's what I've found:
* Setup a named cluster using QAdministrator
* Specify Cluster & Node in RedAlert render window
* writes file named qmaster.mov to same folder as .R3D - ignores filename & folder settings set in the Render window and overwrites any existing file with that name.

I've just been using one machine tonight for my tests. I have some more hardware to work with in the morning.

Anybody else have any notes to add? Does filenaming work? Anyone know how to specify all nodes in a cluster?

 

[Trevor Meier]

So I did some more testing today with Qmaster and a bunch of machines setup as a render farm. I started the long way - using QAdministrator to set up one large cluster for all of the machines and trying to specify nodes in RedAlert. Doesn't work... RedAlert always seems to submit to the same machine, regardless of what you put in the node box. Interestingly it will fill up each node (there is one node for each processor) on that machine, but once it's full it won't move on to another machine.

... but fortunately there's the easy way. Use Qmaster's System Preference pane to create a QuickCluster for each machine, give it a memorable name, and enter that name into the Cluster field RedAlert's render dialog. Voila... instant render farm.

The next trick is making sure each of the machines has access to the same volume as the machine running RedAlert, so the paths match. Try this link on using AutoFS to get started...

 

[Trevor Meier]

For the filenaming issue... as long as you are rendering to a volume other than the boot volume, the filename & path specified in RedAlert's render dialog work as expected.

 

[Trevor Meier]

One more thing to note... Qmaster doesn't seem to queue jobs for each Cluster. Each time you submit a job from RedAlert it starts rendering via RedLine immediately; and each instance of RedLine sucks up 1.5-1.8 GB of RAM.

I wasn't able to hit the node limit (perhaps it'll start queuing if the machine has more jobs than processors) but it's interesting behaviour given that each node launches a multithreaded decode that gobbles up most of all eight processors.

 

[Trevor Meier]

Did a little test for the node limit... so it seems the rule is, using QuickCluster, Qmaster creates one cluster per machine, with the number of nodes matching the number of processor cores (e.g. two nodes for a Core Duo laptop, eight on an Octo Mac Pro). That might be good for non multi-threaded renders, but for RedCode it might be better to throttle the number of node instances being presented.

In the Qmaster system prefs pane, select the rendering service and click "Options for Selected Service". By default there will be a list of "This Computer"s to match the number of cores. It worked here on my laptop to remove one instance of "This Computer" in the list, and Qmaster would only submit one RedLine render at a time. Clicking the plus icon, then selecting local, adds another instance back.

I'm going to experiment tomorrow with some speed tests, to see what the optimal setting is for eight core machines rendering redcode. Since Redcode already nearly maxes all eight cores with just one render, I'm guessing somewhere between two and four node instances will give the best performance without bogging down the system.

 

[Trevor Meier]

Did some testing this morning using different numbers of nodes with some interesting results. For each test I used RedAlert to submit 10 jobs, each of a 4-second 4K clip rending a full debayer to fit-height 1080P ProRes.

8 nodes and 4 nodes per 8-core box produced the same render time, about 9.5x realtime. Using just 2 nodes took longer, about 12x realtime.

8 nodes used about 10GB of RAM; 4 nodes about half that. So it seems the system likes being stuffed full, as long as you have enough RAM to handle the number of concurrent processes.

I'm going to set machines in my farm to 4 nodes for now so that it's kicking out finished files more quickly.

 

[Gunleik Groven]

Trevor

This is sorta the thread of my dream, BUT

I'm a complete idiot, but qMaster always craps on me.

Would it be possible for you to make a step-by step guide on how you're setting it up?

Gunleik

 

[Tai Wah Lim]

Trevor

This is sorta the thread of my dream, BUT

I'm a complete idiot, but qMaster always craps on me.

Would it be possible for you to make a step-by step guide on how you're setting it up?

Gunleik

Anyone can help out there? Lim