Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Windows vs OSX Redcine Performance

Hi Florin,

Thanks, I'm sticking with RAID5 I think, it offers the level of protection I need. I have lost one drive in the array before and it's saved my skin.

Ok, I'm getting nowhere, so the write speed is waaay too slow but that's not the issue hanging up my transcodes from Redcine and Scratch.

This is driving me nuts. It's got to be something.

Hmm after further investigation, you say I could write 10GB /hr at 3MB/sec? That is just about where I'm stuck. Each frame is 8,108KB, that makes 7.91MB per frame. At 3.149MB/sec max write speed, that's 2.51 secs per frame multiplied by 1500 frames gives me 62mins!

The write speed is the problem!

Now to figure out why I'm not getting 200MB/sec+ writes to the array.
 
Is anyone?

Is anyone?

4K Hub said:
Now to figure out why I'm not getting 200MB/sec+ writes to the array.
Click to expand...

Is anyone getting 200MB/sec write speed?

I tried my "proxy" playback on my brother's new computer with new SATA harddrives to see if it was faster than the EIDE drives I have on my test computer,

On my EIDE computer the 640x350 "proxy" resolution uncompressed images play back more or less without dropped frames in random access.

On his machine the hard drives seem to run out of data every 10 seconds or so and the playback falls behind by maybe 8 or more frames, I guess while the cache refills. So while his system plays 1K out of the disk cache well, after a few seconds of playback the delays become so bad it is useless.

Maybe there are some issues with the cache size, a smaller cache may allow the software to finish small transfers faster.

I have been updating my program to write to the disk faster under Windows (tm), and found that reading and writing the data was most of the time for processing 4K files, the actual CC-resize and such in CPU was just a small part of the total time for processing uncompressed files.

Try using the Windows (tm) copy command and see if you can move 1m of DPX or TIFF files in less than an hour, that would tell you if the way the software writes to the disk is an issue. I found a 20x speed improvement in changing the way I talk to Windows (tm) about moving the same amount of data to a file.

PS. I am looking forward to the RED CODE to TIFF bench mark numbers to see how many hours it takes a/your system to make 1m of 4K uncompressed TIFF CCed files shot with a RED ONE (tm).
 
www.DANCAD3D.com said:
Is anyone getting 200MB/sec write speed?
Click to expand...

With a good, well-configured RAID? Oh yes! :)

Or, you mean 200 MB/s out of the application? Probably not, I'm not sure.
But, well, even if the RAID is an order of magnitude faster than what the application can spew out, that's not a bad thing. Then the system can focus resources on things that really matter.

To 4K Hub: You can probably use a single drive to store the output of the application, and it will definitely be faster than 3 MB/s (on my laptop I get 20, on the desktop about twice faster, and this is pretty mediocre hardware). This way you can try other things while you figure out the RAID bug, which may take a while.
 
I'm not going to do the 4K DPX or 4K TIFF tests until I sort this RAID bug out. I'm running a defrag on the array, although I don't think this is the issue. It'll take a while, so in the meantime I'm going to check though every system and controller card setting I can find. It's a 3ware 9550SXU-8LP, maybe I should also update the driver and firmware while I'm at it.
 
4K Hub said:
I have done a read/write test to the array. Read is 291m/sec write is 3.149m/sec, it should be in the 200+ range. This looks like the issue. Not sure what's wrong.
Click to expand...

thats certainly one, if not the, issue.

The standard sata raids on the onboard controllers off the 500€ systems give easy >250-300 Mbyte/sec read and write.

Our more perfomance orientated workstation (striped raid 5 / 6 areca controllers) easily sustain 500 and up.
 
4K Hub said:
I'm running a defrag on the array, although I don't think this is the issue.
Click to expand...

Thats not the issue. Defrag is wasted time in this case. Even at search only (what worst fragmentation could produce) the array must keep >100. We use areca controllers. Highly recommendable, extremly powerful, reliable, inexpensive to the max and very fast in raid 5/6/50/60 and striped up to SATA 72 ports / terrabytes. they cost ~400-500 per 8 channel system iirc, ~650 per 16.
 
Hi Laguun,

I'll look into the Areca controllers for sure. What do you use for enclosures? You've got the card in the machine obviously, how do you connect ports on the internal card to an external enclosure?

Well, I've got the 3ware, which isn't a bad card, any ideas in your experience what could be giving me such bad write speed?
 
4K Hub said:
thanks, I will certainly give that a try!

I have done a read/write test to the array. Read is 291m/sec write is 3.149m/sec, it should be in the 200+ range. This looks like the issue. Not sure what's wrong.
Click to expand...

Check the status of your array to see if your RAID controller is currently in "rebuilding" or "re-verifying" mode. If it is, this could certainly account for bad write performance (but good read performance) under RAID5. Your RAID management application will tell you...

You will suffer bad performance until the operation is complete. It can day up to a day (or even longer) if you have a large array. You can cancel the operation and re-run your transcode benchmark testing, but you won't have fault tolerance enabled without doing the full build/verify process.

Adam
BOXXlabs
 
Ah, got it. Enabled write cache on the controller card. 289.3MB/sec read, 247.2MB/sec write.

Now for DPX export from Redcine and Scratch.
 
What Adam said... good thing to check.

With the 3ware controllers - pull up the 3ware control panel. I think the administrator password is "3ware" by default. Then see if anything is verifying or rebuilding.

Even if it was, that shouldn't kill your write performance to the level you're at! But it is something to check.

Lucas
 
BOXXLabs said:
Check the status of your array to see if your RAID controller is currently in "rebuilding" or "re-verifying" mode. If it is, this could certainly account for bad write performance (but good read performance) under RAID5. Your RAID management application will tell you...

You will suffer bad performance until the operation is complete. It can day up to a day (or even longer) if you have a large array. You can cancel the operation and re-run your transcode benchmark testing, but you won't have fault tolerance enabled without doing the full build/verify process.

Adam
BOXXlabs
Click to expand...

boxxlabs is correct, the read/write difference can be caused by rebuilding arrays. if you have important data stored on that array, dont experiment with it until rebuild is done.
 
Ok first test:

2 x Intel Xeon E5345 2.3GHz
8GB RAM
QuadroFX 3500
3ware 8-port RAID controller card, 8 x Hitachi 500GB RAID5 array
WinXP x64 (and WinXP Pro) on separate SATA drives, and separate swap file SATA drives

Here's the Scratch results (3.9 build462) on XPx64

Scratch 4K RC to 1080p DPX: 1080 construct, 4K Redcode 16:9, fit width, process 1/2 res, show 1/2 res, high, DPX output - 1500 frames at 2mins 53 secs (8.67fps)

Rebooting into XP Pro now for Redcine test (it crashes on XPx64).
 
Here's the Redcine results (3.08 build 108) on XP Pro

Redcine 4K RC to 1080p DPX: 1080 project, 4K Redcode 16:9, fit width, process standard, show 1/2res, high, DPX output - 1500 frames at 1min 50secs (13.63fps)

That's way faster than I expected, that even beats Laguuns times, and beats the times I have from a colleague run on a 2.8GHz Mac Pro (2 min 45 sec). Redcine wins over Scratch it seems for pure transcoding of RC footage on Windows. Interesting, I will run 4K DPX as well, and TIFF but it's 9:43PM in my part of the world, it might have to wait until tomorrow.

All down to enabling write cache on the controller card.
 
on redBOXX we get about 12-13 fps using your settings on b16 16:9 clips in Scratch beta 481. RAID array is stable at about 90-100MB/s during the processing.

for DANCAD3D's inquiry, Scratch takes slightly less a second per frame when transcoding the same clip to 4K full quality 16bit TIFF. RAID writes sustained at about 75 - 85MB/s.
 
BOXXLabs said:
on redBOXX we get about 12-13 fps using your settings on b16 16:9 clips in Scratch beta 481. RAID array is stable at about 90-100MB/s during the processing.

for DANCAD3D's inquiry, Scratch takes slightly less a second per frame when transcoding the same clip to 4K full quality 16bit TIFF. RAID writes sustained at about 75 - 85MB/s.
Click to expand...

Wow 12-13fps in Scratch. I wonder why I'm getting only 8.67 out of Scratch but 13.63 out of Redcine.
 
Richard,

Not trying to point out the obvious, but unless your test is on the same OS, it doesn't mean a whole lot for comparison's sake.

SCRATCH on XP64 vs. REDCINE on XP32 isn't really telling you much. :biggrin:

Lucas
-----
ASSIMILATE, inc.
LA, CA, USA
 
4K Hub said:
That's way faster than I expected, that even beats Laguuns times, and beats the times I have from a colleague run on a 2.8GHz Mac Pro (2 min 45 sec). Redcine wins over Scratch it seems for pure transcoding of RC footage on Windows.
Click to expand...

it better should beat a single 500€ quad :).

however, if we throw some of our quads at the job, lets say 4, 1500 frames get processed in ~45 seconds. Thats a 2000€ setup and therefore allows pretty good pricing for the customers.
 
You must log in or register to reply here.
Back
Top