Metafuze 2.0 : Slower than before...

[Edouard PERCHET]

I've been testing this new version, expected like the messiah, and all the changes that i can see in terms of transcoding (not functionnalities) is a bigger CPU consumption (yes, all my 4 cores are maxed out) but an overall slower transcoding time (+25% for a Quarter Good Debayer, 1080p 23.976 DNXHD 36) compared to what was doing 1.2 version...

Anyone else experiences the same thing ?

PS : I remember Michael telling us 1,7x rushes time with an 8-cores using Half Good debayer, i'd like to see that now (relatively speaking, i have only a quad core)...

 

[MichaelP]

What drives are you using? Drives can be a huge bottleneck - like reading and writing from the same FireWire will not gain you much processing time - especially on a quad core. Go to the configuration settings and try scaling back some of the threads. MetaFuze goes full bore and can overload the drive capacity resulting in slower results at times.

My tests were done on a 16TB RAID.

Michael

 

[Henrik Cednert]

Hmm... I see something similar here. :troll:

This is an quote of myself from an old thread and it is about an alpha of 1.2

Computer: 8 core Xeon 3.2 Ghz (2x5482)
Source on 2.5TB RAID 0 (8 drives).
A 4k 2:1 R3D @ 00:01:04:23 and 25fps
Transcoded to DNxHD40 (guess they mean 36?)
Debayer Low
Burn in TC and key
Letterbox Center

minutes:seconds (rouded to closest second)

Eight good: 00:49
CPU @ 70-75%

Quarter good: 01:34
CPU @ 70-75%

Half good: 08:51
CPU @ 50-55%

Half premium: 08:55
CPU @ 50-55%

Full premium: 29:24
CPU @ 40-50%

If we do some math on those numbers on the 1.2 Alpha we get approx:
Eight Good: 0.75:1
Quarter Good: 1.45:1
Half Good: 8.17:1
Half Premium: 8.23:1
Full Premium: 27.14:1

Today on the very same computer, but with another clip that's 1545 frames long I get:

Eight Good: 1.33:1
Quarter Good: 1.59:1
Half Good: 2.38:1
Half Premium: 4.51:1
Full Premium: 13.66:1

And on this I didn't do the "burn in" which seems to add about 10 seconds to the transcode. But still, it's interesting to compare these group of numbers. The new version seems slower on the lower qualities but faster on the higher qualities. But note in my quote above that the higher qualities on the alpha version of 1.2 was not using the procs so well. This new version, 2.0, max out all my eight cores, but still slower in general? :nopity:

Other then that it feels like a stable release. There's still a lot to ask from the GUI and general workflow but that might end up in another thread, one of my wish list threads. :thumbsup:

 

[MichaelP]

Again.. what is the drive situation in these cases? V2 will overload the drives when it is going full bore resulting in slower times. In those cases, you need to scale back the number of threads.

 

[Henrik Cednert]

Again.. what is the drive situation in these cases? V2 will overload the drives when it is going full bore resulting in slower times. In those cases, you need to scale back the number of threads.

Think it was up in my first post.

Source on 2.5TB RAID 0 (8 drives). And that is also what I transcode to. Exactly the same environment as the 1.2 alpha tests. Will do a test where I transcode to the FC attached SAN.

 

[MichaelP]

Have you reduced the number of threads?

 

[Tom.Wong]

also ram can be a bottleneck too with every thread you use. general rule of thumb is 2 gigs of ram for every thread for max efficiency especially when it's cooking away maxed out.

 

[Igor Ridanovic]

I don't have time to compare numbers this morning but I did a very comprehensive speed test with a benchmark in respect to Redcine 20.0.0 yesterday. For reference, "low, half good" is running 2.5 times faster on MetaFuze than on Redcine. I tested it with different drive controllers (SAS, Firewire, ESATA).


MetaFuze 2.0 Speed Test Table

The table HTML formatting is messy, I just published this late last night and will fix it.

 

[Henrik Cednert]

Have you reduced the number of threads?

I don't know how threads are supposed to work here. But I instinctively thought that the number of threads would limit the number of cores it will send job to. Guess I thought that since that's how other apps I've used have done it, one thread per core. But maybe you branch 2 threads out to all 8 cores in mfuze? Anywayz, no difference in speed if it's set to 2, 4 or 8. Same same.

Some more speed tests:

SAN is a 4GB/s fibre attached Bright Systems SAN.
RAID is 2.5TB RAID 0 (8 drives).
C:\ is a single 10krpm SATA disk
The source is a 1545 frame R3D.

SAN to SAN: 02:27.7
RAID to SAN: 02:24.3
SAN to RAID: 02:26.9
RAID to RAID: 02:28.0
RAID to C:\: 02:24.4
C:\ to C:\: 02:31.1

Not so scientific and I should have ran the tests like ten times each and calculated an average. But honestly I don't have the time. And using the iPhone as a timer from beginning with isn't exactly water proof. :bath: So the only thing I would say here that disk I/O in my case isn't it. Sure, maybe that's the cause of C to C being a tad slower, but in the other cases I can't see that being the case. Or?

The computer have 8GB of RAM. And it isn't even near to hit the roof when transcoding.

And all I'm really saying here is that quarter good and eight good was faster on the 1.2 Alpha. I dunno if it's related to what's on the individual frames that it converts or so, because its not the same clip I'm testing in 2.0 as in 1.2. I had missed to log the name of the clip tested in 1.2. =/

 

[Henrik Cednert]

Great work man. Stumbled across the list earlier today. =)

I don't have time to compare numbers this morning but I did a very comprehensive speed test with a benchmark in respect to Redcine 20.0.0 yesterday. For reference, "low, half good" is running 2.5 times faster on MetaFuze than on Redcine. I tested it with different drive controllers (SAS, Firewire, ESATA).


MetaFuze 2.0 Speed Test Table

The table HTML formatting is messy, I just published this late last night and will fix it.

 

[Igor Ridanovic]

There is a very good explanation of the threads settings straight from the source. This is not CPU multithreading related but the number of reader threads that has to do with source media access.

http://community.avid.com/forums/t/74073.aspx

 

[Henrik Cednert]

There is a very good explanation of the threads settings straight from the source. This is not CPU multithreading related but the number of reader threads that has to do with source media access.

http://community.avid.com/forums/t/74073.aspx

Reading that discussion I can't came to any other conclusion that setting the threads in my case with R3Ds wouldn't change a thing? Since it's automatically set to the maximum in this case...? Right? And that would also explain why I didn't notice any difference.

 

[Igor Ridanovic]

Reading that discussion I can't came to any other conclusion that setting the threads in my case with R3Ds wouldn't change a thing? Since it's automatically set to the maximum in this case...? Right? And that would also explain why I didn't notice any difference.

Right it's auto with R3D files.

The discussion is good because these "threads" are not CPU threads that we thought they were. Same name, different thing.

 

[Marcus Vasques Osorio]

The fact that half good does 2.3 reeltime exporting is pretty impressive. Keep in mind that to do a QT DNXHD 36 standalones (REDRUSHES with RSX look) takes about 3 times real time plus the added bonus of having to batch them into MC afterward.

Marcus AKA mahvo

 

[Igor Ridanovic]

The fact that half good does 2.3 reeltime exporting is pretty impressive. Keep in mind that to do a QT DNXHD 36 standalones (REDRUSHES with RSX look) takes about 3 times real time plus the added bonus of having to batch them into MC afterward.

Marcus AKA mahvo

But let's not forget that for a fair benchmark you need two systems with equal hardware.

 

[Marcus Vasques Osorio]

Will be intresting to benchmark the Rocket when we get ours. SDK + Metafuze + Rocket will be a god send. Imagine MXF DNXHD 36 in reeltime... or faster

 

[MichaelP]

We'll see! Waiting for the SDK for RED ROCKET.