6 ROCKETS and REDCINE-X PRO v9

[Michael Cioni]

If you're like me, it's no secret that the trend of higher shooting ratios is on the rise. With more cameras, more coverage and more footage to process, the need for additional speed may be welcomed by independents and facilities alike. Recently, my friends Torrey Loomis from Silverado Systems and Eric Fiegehen from Cubix helped us out by allowing us to test the latest Cubix XPander (http://www.cubix.com/products/gpu-xpander/gpu-xpander-rackmount/rackmount-2) with REDCine-X Pro v9. We then tested the Rocket acceleration performance with 1, 2, 3, 4 and 6 Rockets in a MacPro (read and write disks were separate volumes). And thanks to Rob and Matt at RED for their advisement and hard work in optimizing RCX-Pro v9 performance. After some extensive testing, the Cubix XPander is a great tool for people who are looking to increase their REDCine-X Pro transcode performance to be much faster than real-time.


If you routinely render ProRes, DNx or DPX files, the below chart will show you what an investment in such technology will buy you. You'll see that the sweet-spot for most of these processes is about 3 Rockets. After you go up past 3 Rockets, performance decreases significantly due to bus limitations, disk I/O limitations and encode limitations (on compressed codecs such as ProRes). But you'll also notice that the results across different codecs is encouraging since the encode times are very similar.


EPIC 5120x2700 4:1
internal raid 0 (560MB/s) to sas (580MB/s)

69.5 fps 1080 prores PROXY 4x rockets
59.4 fps 1080 prores PROXY 3x rockets
43.7 fps 1080 prores PROXY 2x rockets
23.2 fps 1080 prores PROXY 1x rockets

63.1 fps 1080 prores LT 4x rockets
58.3 fps 1080 prores LT 3x rockets
43.8 fps 1080 prores LT 2x rockets
23.1 fps 1080 prores LT 1x rockets

65.7 fps dnx 36 4x rockets
59.9 fps dnx 36 3x rockets
43.5 fps dnx 36 2x rockets
23.1 fps dnx 36 1x rockets

64.6 fps dnx 115 4x rockets
60.1 fps dnx 115 3x rockets
43.6 fps dnx 115 2x rockets
23.3 fps dnx 115 1x rockets

44.5 FPS 1080 10 BIT DPX 4x rockets
44.1 FPS 1080 10 BIT DPX 3x rockets
42.7 FPS 1080 10 BIT DPX 2x rockets
23.3 FPS 1080 10 BIT DPX 1x rockets

 

[Steve Johnson]

Wow. Amazing tests here Michael.

So I take it that the expander connects to a MacPro x16 slot and from that, 4 Rockets gives these performances. I guess for speed, this is going to cut through footage like a knife through butter. Did you do any tests in how this would cut through footage for 3D work?

)

 

[Michael Cioni]

3D would be the same results in terms of fps (within a couple of percents). You can mux 2 R3Ds at approximately the same speed so long as your network is able to read 2 streams of R3Ds. That means 2 rockets will render 3D in just short of real-time.

m

 

[Bruce Allen]

Awesome fun!

Interesting that DPX is slowest - due to drive write speed limitations obviously because DPX files since they're not hard to encode? Those DPXes are just too big!

If you routinely render ProRes, DNx or DPX files

Two questions:

1. How about OpenEXR, B44? And B44 in 4:2:2 mode? Come on Michael - you know I was thinking of you and your 54TB of disk space for TGWTDT when I asked Brendan to write that ProEXR importer / exporter with timecode for AE and Premiere

2. Does that sucker work in Resolve?

Bruce Allen
www.boacinema.com

 

[Subhadip Sen]

Interesting! The Mac Pro + Cubix solution is extremely PCIe bandwidth starved, however, which is the bottleneck past 3 Rockets even though Rocket is not as hungry as modern GPUs. (Of course, for DPX it is disk I/O) Will be interesting to see 6 Rockets going on a Sandy Bridge-E/EP system.

 

[Stephen Lovett]

Thanks very much for this Michael.

It is something that I've been interested in but frankly its not many of us have the resources to test the permutations to this extent.

I've been holding out for a thunderbolt based expansion chassis to allow both cart based and portable (MacBookPro) scenarios.

But... that limits total throughput to Thunderbolt throughput.

Perhaps I need to rethink that.

Is it your thinking that its PICe itself that is the bottleneck here (or one PCIe 2.0 x 16 lane in the MacPro), disk I/0, resource marshaling, or is there enough information to pin that down?

This is super cool, maybe even cooler than the new outdoor kitchen

Steve

 

[Subhadip Sen]

Thanks very much for this Michael.

It is something that I've been interested in but frankly its not many of us have the resources to test the permutations to this extent.

I've been holding out for a thunderbolt based expansion chassis to allow both cart based and portable (MacBookPro) scenarios.

But... that limits total throughput to Thunderbolt throughput.

Perhaps I need to rethink that.

Is it your thinking that its PICe itself that is the bottleneck here (or one PCIe 2.0 x 16 lane in the MacPro), disk I/0, resource marshaling, or is there enough information to pin that down?

This is super cool, maybe even cooler than the new outdoor kitchen

Steve

Since this test is on a Mac Pro with a Cubix Xpander, PCIe bottlenecks are natural and expected. Rob has previously confirmed the PCIe bandwidth will be the bottleneck: http://reduser.net/forum/showthread...e-Build-9-Beta&p=906633&viewfull=1#post906633. With a more suitable system we could see 6 Rockets scale.

Thunderbolt offers 1/4th of a PCIe 2.0 x16 port, so it would be suitable for a single Rocket at best.

 

[Mark L. Pederson]

With a more suitable system we could see 6 Rockets scale.

And such a system exists in the skunk labs over at JMR right now

 

[M Most]

I understand the fun of a technlogy demo, but for the price of 6 Rocket cards, I coukd buld at least 12 reasonably hefty PC's, which would not only allow similar (or better) transcoding performance for dailies purposes, but would do a whole lot of other things, and for files ofher than R3D. I understand the demo purely for curiosity's sake, but Red Rocket cards cost almost $5000 each and only do one task with one type of camera file. This approach seems pretty cost intensive for a single purpose solution. To me, a render farm is much more sensible, practical, and versatile.

 

[Ketch Rossi]

He he, love you for it Michael...

I understand the fun of a technlogy demo, but for the price of 6 Rocket cards, I coukd buld at least 12 reasonably hefty PC's, which would not only allow similar (or better) transcoding performance for dailies purposes, but would do a whole lot of other things, and for files ofher than R3D. I understand the demo purely for curiosity's sake, but Red Rocket cards cost almost $5000 each and only do one task with one type of camera file. This approach seems pretty cost intensive for a single purpose solution. To me, a render farm is much more sensible, practical, and versatile.

I do see your point, a valid one up to a certain point, more machines also a lot more problems to come, but also if one goes down here you have more, but problem with your solution is that by by portability any were...

Oh and did I mention I absolutely despite PC's... Sorry.

 

[Subhadip Sen]

I understand the fun of a technlogy demo, but for the price of 6 Rocket cards, I coukd buld at least 12 reasonably hefty PC's, which would not only allow similar (or better) transcoding performance for dailies purposes, but would do a whole lot of other things, and for files ofher than R3D. I understand the demo purely for curiosity's sake, but Red Rocket cards cost almost $5000 each and only do one task with one type of camera file. This approach seems pretty cost intensive for a single purpose solution. To me, a render farm is much more sensible, practical, and versatile.

Definitely, link them up with InfiniBand. Unfortunately, R3D CPU decode does not seem to scale past 16 threads. A solution for massive multithreading is definitely possible though. 6 Rockets is worth over 100 Sandy Bridge-EP cores and that is just tremendous power when you realize just 4 of those cores already do 1/2 res debayer for 5K to 2.5K and 6 can get you 4K real-time. Best of all - they do a whole lot more than just R3D decode.

Two 8Cx4P towers will get you 64 SB-EP cores for $20k - so it's just 2 workstation towers, not 12 PCs. With extensive multithreaded implementation of the SDK (this is the big if) that will be much faster than 6 Rockets.

 

[Blair S. Paulsen]

Post Rocket world

Post Rocket world

I have been thinking about this issue quite a bit lately as I am working out the details of creating a rig capable of playing back Epic 5K FF (2,700 x 5,120) R3Ds scaled to 4K (2,160 x 4,096 or 3,996 or 3,840) in real time. I own one Rocket at the moment and could potentially buy a second one to create a dual Rocket based solution, but, as others have noted, that's a hefty investment for something that only does one thing.

I join the call for expanding multi-threading options in the SDK. I understand that RR sales might suffer a bit, but based on their trajectory it is clear that RED understands very well that the march of technology is relentless and that the benefits to the entire RED program of real time decodes on newer workstations without the requirement of a RR is a big deal. When post pros and production companies can manipulate R3Ds natively, ala Premiere with scaled monitoring and faster than 3x real time output rendering, on a $3,000 box (plus storage) running AVID, FCP7/X, etc then the overblown blowback of the transcode step hassle in the typical RED workflow becomes moot.

I am not accusing RED of deliberately handicapping the SDK to sell Rockets. At this stage I think they are more concerned with the stability and reliability of the SDK than pushing the envelope, which is the right priority in my book. I also would assume they are using RCX-Pro as a test bed for things like multi-threading support within an app they control prior to pushing such things into an SDK that needs to be generic enough to play nice with a plethora of hardware and software platforms.

I look forward to some serious breakthroughs in 2012 with the newer CPUs, GPUs, better core utilization and faster busses. Of course I still dream of a rack mountable Mac with modern internals that can support professional content creation, but how long can one wait...

Cheers - #19

 

[Alex MacLean]

Fun Test. Thanks for sharing

 

[roryhinds]

More Multitheading please and a Linux version of Red Cine X Pro.
Linux Linux Linux

 

[Bruce Allen]

I join the call for expanding multi-threading options in the SDK. I understand that RR sales might suffer a bit, but based on their trajectory it is clear that RED understands very well that the march of technology is relentless and that the benefits to the entire RED program of real time decodes on newer workstations without the requirement of a RR is a big deal. When post pros and production companies can manipulate R3Ds natively, ala Premiere with scaled monitoring and faster than 3x real time output rendering, on a $3,000 box (plus storage) running AVID, FCP7/X, etc then the overblown blowback of the transcode step hassle in the typical RED workflow becomes moot.

I am not accusing RED of deliberately handicapping the SDK to sell Rockets. At this stage I think they are more concerned with the stability and reliability of the SDK than pushing the envelope, which is the right priority in my book. I also would assume they are using RCX-Pro as a test bed for things like multi-threading support within an app they control prior to pushing such things into an SDK that needs to be generic enough to play nice with a plethora of hardware and software platforms.

I look forward to some serious breakthroughs in 2012 with the newer CPUs, GPUs, better core utilization and faster busses. Of course I still dream of a rack mountable Mac with modern internals that can support professional content creation, but how long can one wait...

Cheers - #19

Agree again! To be honest, I don't understand it.

Did RED really sit down in 2006 or whenever and say "Let's design a codec that even in 6 years' time, no computer can play back realtime without an add-on board"?

I always assumed that RED was designing for GPU or CPU playback in realtime a few years in the future.

EG that their plan was that by 2010 or so we'd have realtime playback on any modern desktop.

Personally, I think that if it turns out that REDCODE can't be decompressed in realtime by, say, 2 modern nVidia Kepler GPUs or a modern dual-Xeon... RED should figure out how to alter REDCODE so that it is more-easily decompressible.

I don't see anything about the basic idea of lightly-wavelet-compressed 4K or 5K RAW imagery that instantly says "impossible to GPU or CPU decode in realtime".

Folks can already do 2K JPEG2000 GPU-assisted wavelet decode of RGB imagery in realtime on current nVidia cards - and that's 3/4 of the pixels of 4K RAW, right?

Bruce Allen
www.boacinema.com

 

[Rob Anderson]

Excellent information. Looks like I'm going to be adding a third card soon.

 

[Sven Seynaeve]

I think this is mostly the cpu manufacturers fault, not red.
if they would have found a reason already to have it spreaded accross more threads, they would have done it.

in the other thread about the prores module , i have did reconsider my opinion. Was thinking to myself today,

we wanted a 5K camera, then why for godsake do we have to settle back and record straight away to hd files; when we can have 5k files, and just have to recode them when really needed for someone ?


I guess when people start to consider using this module as a way of faster editing and more people being able to do editing and post at home or settle for less quality,
then i guess jim's idea is totally misunderstood, and in every specific situation we feel threatened by a competition that has been way slower then Red but have better, more agressive marketing strategies;
i even think red has become the victim in some way of it's potential against these dinosaurs.

not so long ago i've been told as well that mistika does'nt need the rockets that much as other systems do, and it has it's debayering more efficiently spread accross all available cpu's, but still have to wittness this, maby Mark Pederson could chime in on this.

 

[Subhadip Sen]

I think this is mostly the cpu manufacturers fault, not red.
if they would have found a reason already to have it spreaded accross more threads, they would have done it.

CPU manufacturers give you several useable threads to work with. It is completely up to the software developers to utilize all of these threads, or not. Perhaps one could blame Intel for not being as proactive in pushing for multi-threading (and GPGPU) for common desktops apps. This is simply because AMD CPUs offer far superior value in multi-threaded apps across the board and their GPU tech is on the very bleeding edge and several years ahead of Intel. Today, if every program was perfectly multi-threaded and GPU accelerated, that would be catastrophic for Intel. $140 Fusion APUs would be cleaning house with $1000 Intel CPUs. Anyway, all of this is another can of worms. Back to the topic - the R3D decode is adequately threaded for what is likely the most common computer solution around these parts - Mac Pro. It close to maxes out the prime threads, while the hyper threads are loaded somewhat. However, it does not scale for more powerful computers. I considered using an (relatively inexpensive, ~$5k) AMD Opteron 32-core machine (also for other usage) but wherever I tried R3D playback it simply couldn't utilize past ~30% of the CPU, at which it was doing 1/2 res for 5K. The system offers monstrous X264 (which maxes out all 32 threads) performance, 2.5x that of the fastest 12 core Westmere Xeon systems, while R3D performance is much the same as a 6 core Sandy Bridge-E system! Which, in turn, is not that much faster than Gulftown systems from 2010 or very cheap quad core Sandy Bridge systems. All that said, I am content with what a mainstream ~$1.5k PC can offer today - real-time previews at 2.5K, that's how far my monitors go anyway. It's just a shame that spending 5 times as much isn't going to help playback much and I can't help but think there's even greater untapped potential. Not to mention, the GPUs are mostly idle.

On a slightly related note, some blokes from NCSU managed to get non-GPU accelerated apps to run on an average 20% faster, upto 110%, on AMD APUs by simply using software to schedule and distribute load with both GPUs and CPUs.

 

[Blair S. Paulsen]

Information please...

Information please...

If I have this right there are several discrete steps in the process:
1) Decompress
2) Debayer
3) Scale to output and/or monitor resolution

I don't have the citation handy but IIRC Rob or Stuart suggested that the decompression was a challenge for even fast multi-core CPUs, and not well suited to GPU resources (driver issues perhaps?) so a special purpose card like the Rocket is a big help.

I would love to know if you could set up a Rocket to ONLY do the decompress, not the debayer, scaling or monitor feed - if so, would it then be able to hand off 5K FF Epic files in real time?

Debayering seems to be in reach of the newer CPUs with proper core utilization, but would love to know the real deal...

Scaling would seem to be a good job for GPU resources.

If anyone out there can break these issues down in greater detail I would welcome the knowledge...

Cheers - #19

 

[Gabriele Turchi]

Excellent information. Looks like I'm going to be adding a third card soon.

people , this is why read is not interested on having R3D optimized for CPU multi core or GPU playback ...i bet the make as much money from RR than from cameras (if you consider how much cost them camera development and RR development (ended basically over 2 years ago (and 5K for a single component card that cost them ...well not 5K....)

ps: cineform RAW (visually lossless ) was playback 4K realtime 2 years ago in a single CPU core i7 (quad core) ...

g