Render Farming

[Gordon Breckenridge]

We are using Rush to render R3D files across our farm to great effect. It's not a free solution though. Haven't tried them with qmaster yet although we do have a cluster running now. is qmaster just sending out a terminal command to render the .r3d? One movie per node as it were? Rush works like
that.

 

[danbrazda]

Can't help myself I have to add to this subject despite trying to walk away many times...

I am the most hardcore Apple advocate you will ever find but I also think I'm open-minded enough to say it like it is when necessary, so....

-Qmaster is currently a JOKE when it comes to practical acceleration of render tasks across multiple stations

AND

slightly off subject but not really...

-If Color is going to be part of the Final Cut Suite it has to feel like it belongs on a Mac and integrates smoothly between all Final Cut Suite Apps


I will be the first to post congratulate Apple when Final Cut Suite 3 is released fixing these issues as well as allowing true 4K native support for Red.

Love ya Apple, but... we're waiting .

Rant complete.

 

[Colin Hubick]

Qmaster

Qmaster

We are using Rush to render R3D files across our farm to great effect. It's not a free solution though. Haven't tried them with qmaster yet although we do have a cluster running now. is qmaster just sending out a terminal command to render the .r3d? One movie per node as it were? Rush works like
that.

I've been using the Qmaster approach using Monkey Extract, and once Qmaster is setup correctly it works very well. Yes it appears that Qmaster is sending a terminal command, but it renders one clip per instance. So on our 8 core machines, we are running 8 instances. Qmaster was a b**tch to get working correctly, but it's now been running rock solid for quite a while.

 

[danbrazda]

I've been using the Qmaster approach using Monkey Extract, and once Qmaster is setup correctly it works very well. Yes it appears that Qmaster is sending a terminal command, but it renders one clip per instance. So on our 8 core machines, we are running 8 instances. Qmaster was a b**tch to get working correctly, but it's now been running rock solid for quite a while.

I would LOVE to hear more detail about what you went through to getting it to the point where it runs smoothly. Not sure if you've read this entire thread but if you can sort through the right and wrong (from your experiences) of what everyone has posted I'll bet a lot of people would be greatly appreciative.

Would it help if I said I love Canada and I want to move there?

 

[Colin Hubick]

The first biggest issue I had when setting everything up through Qmaster was the inconsistant version of Qmaster that I was running. The symptoms to this problem showed up with weird messages that Qmaster was not running, or it would just not respond at all (frustrating start)

I didn't want to reinstall the whole FCS2 package on every suite, so I found this handy app called FCS Remover which allowed me to only remove the Qmaster portion of FCS2. After completely removing Qmaster from all the systems, I did a reinstall of Qmaster from the FCS2 install disc, located in the "extras" folder called "AppleQmasterNode.mpkg" after installing the Qmaster off of the disc, I did a software update which brought up a 28mb ProApp update which updated Qmaster to the most current version. (if you don't do this ProApp update nothing works right)

So now that Qmaster is running the same version on all machines I started to setup the shared drives. I created a common account on all machines that would be used to login to mount the shared drive. (this to is very important or you will get a bunch of failed renders)

So now each machine is running the same version of Qmaster, same version of Redline, and the most current version of Monkey Extract (at the time of writing this it is 2.301)

The next part of the process is deciding which machine is going to be setup as "QuickCluster with Services" and which machines are set to "Services only".

Pick one computer, and goto the Apple Qmaster settings in the system prefs, and select the radio button "QuickCluster with services" there should only be one machine set this way. Then give this QuickCluster a name under the "Identify this QuickCluster as:"
Then goto the Advanced Tab and select "Show Qmaster service in menu bar" this allowed me to glance at the screen to see if it was working correctly or not without having to go into the prefs.

best rule of thumb is no spaces or weird characters for the cluster name. ie REDCluster

on all other machines select the "Services Only" and then on the Advanced tab check off "Show Qmaster service in menu bar"

***very important that all machines that are a part of this cluster reside on the same subnet on your network. (we are using GigEthernet) don't use wireless connections as they are not solid enough for Qmaster (turn off Airport altogether)

Once all this is done, click the "Start Sharing" button starting with your "QuickCluster with Services" machine then start your "Services Only" machines.

The best way I found to check that all machines are working correctly is to launch "Apple Qadministrator" on the machine you have chosen as your "QuickCluster with Services" and look for the name you gave your "QuickCluster with Services" in the top left area called "Cluster". The remaining "Services only" machines should be listed on the bottom.

If all this is correct, then launch Monkey Extract, and try submitting a Qmaster job, when you are prompted for the name of your QuickCluster enter it, and continue. You will be able to tell if everything is working by looking at your "Batch Monitor" for a list of jobs, and their status.

(Canada Kicks Ass

cheers
Colin

 

[Colin Hubick]

**UPDATE**

I just tried version 2.303 of Monkey Extract, and everything is working fine with Qmaster

Colin

 

[danbrazda]

Thanks for taking the time Colin. I look forward to testing everything on this end. I'll post results Tuesday (out of town until then).

 

[Colin Hubick]

Thanks for taking the time Colin. I look forward to testing everything on this end. I'll post results Tuesday (out of town until then).

how did this work out for you?

 

[danbrazda]

how did this work out for you?

Hey Colin, sorry I dropped the ball on getting back to you on this. Things have been very busy and I can't bring myself to do more troubleshooting with QMaster at this time. It's been very frustrating up until this point. Bottom line for now is I have a single Nehalem 8-Core machine doing my Compressor tasks because it is faster than all ten of my machines together. Others have suggested that your cluster is only as fast as the slowest node in it (in my case, 4 PPC G5 Quad Cores from 2006). If this is true, I guess it would explain my situation but I have to tell you it defies logic to me whereby a software program that is designed to manage available resources in a Gig-E network yet can't figure out how to properly utilize different "flavors" of machines within that network, well, let's just say I view that as Beta software and doesn't qualify for a release build.

I am holding out until the new Compressor/QMaster is released as part of FCP Suite 3 before I test my setup again. Thank you, nevertheless, for your diligent assistance.