Windows vs OSX Redcine Performance

[Uli Plank]

Are we talking about 2:1 or 16:9 ?

Regards,

Uli

 

[laguun]

1m 4K RED CODE to 1080p = 1hr

1m 4K RED CODE to 4K DPX = about 2.5hr?

1m 4K RED CODE to 4K TIFF = about 3hr?

We use distributed computing (dozens of systems here).

~500€ standard Vista/OSX/XP PC (Quad 2.6/GF 8800GT/4GB/1.5 TB Storage)

Those run each typically 1:3 RT @ 1080P, 1:22 RT @4K, 2:1 Aspect Ratio, DPX.
1m 4K RC to 1080p = ~3min.
1m 4K RC to 4096p = ~22min.

running OSX they are slightly slower - probably also as they dont have the best drivers to access to the very fast SAN/Network/RAID.

 

[Richard Lackey]

We use distributed computing (dozens of systems here).

~500€ standard Vista/OSX/XP PC (Quad 2.6/GF 8800GT/4GB/1.5 TB Storage)

Those run each typically 1:3 RT @ 1080P, 1:22 RT @4K, 2:1 Aspect Ratio, DPX.
1m 4K RC to 1080p = ~3min.
1m 4K RC to 4096p = ~22min.

running OSX they are slightly slower - probably also as they dont have the best drivers to access to the very fast SAN/Network/RAID.

Laguun,

How many systems are tied to your SAN to get those figures? (If you don't mind saying). Is each 'node' tied to it with Fibre Channel?

Surely 1m 4K RC to 1080p can't be 3min on just one machine, I am assuming that is an average when all your systems are running?

I've been considering building a render farm for 3D work for a while, and am also looking at a Bright systems SAN with a business partner. It seems this might also do well for Redcode transcodes.

So, I am assuming that:

1. Redcine is no faster on OSX than Windows on the same hardware platform.
2. 4K RC 16:9 to 1080p 16:9 DPX at a ratio of 1:60 is normal for a single 8-Core Windows or OSX system.
3. A ratio of +/- 1:20 on a Coreduo powered Macbook Pro is not possible.
4. To get Laguun's 1:3 ratio, one would need 20 x 8-core Xeon (in my case at 2.3GHz, 8GB RAM) 'nodes' tied to a SAN backbone?
5. If Redcine only scales well up to 4 processor cores (as someone previously mentioned), then 20 x Intel Q6600 based nodes, each with 4GB RAM should be cheaper and give me similar results.

 

[Richard Lackey]

Are we talking about 2:1 or 16:9 ?

Regards,

Uli

We're talking 16:9, I haven't shot 2:1 yet, so I don't know if the transcoding is any faster.

 

[laguun]

Laguun,
Surely 1m 4K RC to 1080p can't be 3min on just one machine, I am assuming that is an average when all your systems are running?

Single machine.
redcode source on SATA
dpx target on striped SATA

transcode settings:
1080p project size
4K rc 36 source
det med, nr off
display full / high
filter linear
process standard

things that can slow down extremly
- display format different from output
- project size different from output
- process full

1. Redcine is no faster on OSX than Windows on the same hardware platform.

Yes. here even slightly slower on OSX (but that might be slightly different with different drivers)

2. 4K RC 16:9 to 1080p 16:9 DPX at a ratio of 1:60 is normal for a single 8-Core Windows or OSX system.

No. Set project to 1080p. scale all. display full/high. nr off. det medium. filter lin. process standard. set output to raid.

Gives RT 1:3 - 1:4 RT on a ~500€ quad.

4. To get Laguun's 1:3 ratio, one would need 20 x 8-core Xeon (in my case at 2.3GHz, 8GB RAM) 'nodes' tied to a SAN backbone?

Every single quad performs at
~1:3 @ 1080p standard,
~1:20 @ 4K standard.
Target storage is buffered locally on sata raid and pulled by the server.

5. If Redcine only scales well up to 4 processor cores (as someone previously mentioned), then 20 x Intel Q6600 based nodes, each with 4GB RAM should be cheaper and give me similar results.

[/QUOTE]
In 1080p we can decode quite faster than realtime on the *group* of rendering systems.

 

[Richard Lackey]

Laguun,

Thanks, you have helped me out so much with your answers. I am going to give it another go with your settings and see what I get. I should get a much better ratio.

So, my 1hr per min of RC is not an accurate idea of how long this transcode takes, it should be much faster, in the region of 1:3 if my settings are as Laguun suggests.

And, there is no real difference between doing this on Windows vs OSX.

Ok, time to run a DPX output with Laguun's settings. If I get 1:3, I owe someone an apology and it seems my claim of 1:60 did lose them the Red shoot.

Man, I hate being wrong. At least I am right about the OSX vs Windows issue.

I will post my results soon, also with 4K DPX and TIFF just for interest sake.

Thanks for your help.

 

[Vigen Vartanov]

Sow you mean that COU speed 8 COres is ok , and we are only limited in HDD ?

 

[Richard Lackey]

Hi Laguun, tried your settings and I'm still getting +/- 1hr per minute. The Redcode Source is on a separate internal SATA drive, the destination is a 8 drive RAID 5 array. All the settings are the same as you have said.

Redcine is definitely not maxing out all 8 cpu cores, seems to be using only one anywhere near 100%, the other 7 are all 17% or so.

I can't think what is the problem. I'm getting the same results out of Scratch too.

One strange thing is I can't set the filter to linear, it gives me garbled pink mess.

Any ideas?

 

[Lucas Wilson]

Hi Richard,

Are you *sure* you are processing from "Standard" and not "Full?"

Also... Laguun...

You mention you are processing "Standard" but viewing "Full/High." I believe your render would be faster if you were viewing 1/2 rez high, yes?

Lucas

 

[Richard Lackey]

Are we talking about 2:1 or 16:9 ?

Regards,

Uli

Is there a difference in render performance between 16:9 and 2:1?

Thanks everyone, I have to get to the bottom of this.

So far I've been given the following:

2min 45 secs per minute of RC outputting to 1080p DPX with process set to standard and display on half res, high on a 2.8GHz 8-Core Mac Pro with 6GB RAM. I know they were shooting 2:1 ratio.

So why am I still getting 1hr per minute of RC using the same settings?

Only differences I am experiencing are that I am Windows, the Mac is obviously OSX, I am running 2.3GHz not 2.8GHz processors, I have 8GB RAM, the Mac has 6GB, they are processing 2:1 aspect ratio footage, I am processing 16:9.

Laguun? Anyone?

 

[Lucas Wilson]

Richard,

Something is definitely wrong here.The fact that your cores are not multithreading properly is also really concerning. When I process Redcode files on REDCINE, all 8 cores are pegged pretty evenly - they are all at 80% or more.

This is a good discussion, so keep it up here... but can you also post this to the SCRATCH support alias? This sounds very much like a systems issue.

Lucas
------
ASSIMILATE, inc.
LA, CA, USA

 

[Richard Lackey]

Sure Lucas, I will do so.

I've just installed the latest Redcine build, the Linear filter now works, no change in performance. All CPU cores are running 4% only. This should be way faster if it were using all 8 cores at 100%

Very very strange.

Looking at the processes running:

System idle process: 97%
Redcine.exe: 3%

 

[laguun]

Sure Lucas, I will do so.

I've just installed the latest Redcine build, the Linear filter now works, no change in performance. All CPU cores are running 4% only. This should be way faster if it were using all 8 cores at 100%

Very very strange.

Looking at the processes running:

System idle process: 97%
Redcine.exe: 3%

Something is wrong.
Do you render to a mapped network drive?

 

[laguun]

Also... Laguun...

You mention you are processing "Standard" but viewing "Full/High." I believe your render would be faster if you were viewing 1/2 rez high, yes?

Lucas

yes, that shaves off another ~10% of the render time and we do so if the client is in a hurry. However for monitoring purposes we prefer running the full/high while converting, as subtle noise levels etc are easier to judge while the transcoding process on the monitor.

 

[Richard Lackey]

Something is wrong.
Do you render to a mapped network drive?

Nope, the RAID is local. I'm doing the same output now from Scratch on a 1080p construct, fit width, process standard, 1/2res, high and I'll watch the CPU utilization as well.

Lucas, could I have to change something in the system environment variables?

 

[Joel Kaye]

When I process Redcode files on REDCINE, all 8 cores are pegged pretty evenly - they are all at 80% or more.

I can verify this is what happens on my Mac also. (2008 8 Core)

 

[Richard Lackey]

Yep, Scratch output does the same thing, 1-3% CPU utilization. Very strange indeed.

Seems like it might be a disk write speed issue.

 

[Florin Andrei]

I've just installed the latest Redcine build, the Linear filter now works, no change in performance. All CPU cores are running 4% only. This should be way faster if it were using all 8 cores at 100%

Quote from the .txt file that comes with REDCINE:

* Command line parameters:
-------------------------------------------------------------
-d or -dd - One or more increases the 'logging' level.

-p# - Setup multiprocessing for '#; cpu's, overriding the
default which is the actual nr of physical available cpu's.

Try to launch REDCINE from the Command Prompt, something like this:

C:\Progra~1\REDCINE\bin\RedCine.exe -p2

or

C:\Progra~1\REDCINE\bin\RedCine.exe -p8

Tweak that command if something's different on your system.

Tweak the -p parameter, use a few different values. I'm not sure whether -p means the number of cores, or the number of physical CPUs, the document is not clear. So try both. Use the lowest value that seems to load all cores to 100%.

These experiments should be very relevant in your case, since you have 8 cores to experiment with. I only have 2 so I can't experiment much (one core is busy anyway with decoding and stuff like that, so it's probably not worth loading it with the encoding job).

 

[Richard Lackey]

thanks, I will certainly give that a try!

I have done a read/write test to the array. Read is 291m/sec write is 3.149m/sec, it should be in the 200+ range. This looks like the issue. Not sure what's wrong.

 

[Florin Andrei]

I have done a read/write test to the array. Read is 291m/sec write is 3.149m/sec, it should be in the 200+ range. This looks like the issue. Not sure what's wrong.

At 3 MB/s, in 1 hour you could write 10GB of data to that volume. If the output file is smaller than 10GB / hour of encoding, then the disk, even though painfully slow indeed, is still not the bottleneck.

But yeah, 3 MB/s, wow, my USB Flash dongle is faster than that. Something's definitely not right.

By the way, RAID5 is not exactly a performance champ. If all you want is balls-to-the-walls performance, try RAID 0, but a single drive failure will lead to the loss of the entire array, so use the array only as a "work area" and store the important stuff elsewhere.

http://en.wikipedia.org/wiki/Standard_RAID_levels

Or try RAID 10, it should be fast enough and give some protection, but you "lose" half the drives' capacity.
Finally, fine-tuning a RAID array is not trivial. Don't expect to just setup an array and wham! it works at its full potential. Speed ups of 2x or more through fine tuning are not uncommon.