Building System For REDCine. Need advice

[Jeff Kilgroe]

Looks good, Zeke.

Although, I'm not sure why you would want the ATI FireGL cards... I'm trying to hold off on purchasing any new systems, especially ones with 45nm CPUs until Intel fixes their problems. Probably another month or so...

 

[galexander]

Since the goal of this is pretty application specific, there is no arguing the importance of testing.

I have an episodic project coming up and our needs our pretty damn specific. Namely, we need to get about an hour or more worth of .r3d's through the pipe line daily (starting just after wrap) to editorial overnight, 5 days a week, for as long as the show runs. I am really seriously considering that there is no single machine that will accomplish this fast enough. I am considering putting several (perhaps up to 4) machines on the task.
Kevin Halverson

it depends on how much you have to spend. of course there are single computer systems that can chew through the data. we used to process spectral hypercubes in flight, in real-time, with a mini-SGI beowulf type cluster.

These can be handled by one of the "over night" guys as it won't require much baby sitting. It just has to be done well before everyone arrives the following morning.

....

The testing criteria is a 4k .r3d output to 1080p DNxHD, process is 'standard' (pulling from the 2k wavelet) minimal color changes. This can process a frame in about 910 ms (.91 second). I am still testing other outputs, but this seems to be a minimum in quality terms for our needs.

One of these machines would take nearly 22 hours to process the shots daily. With 4 on the task, they would finish in about 5 1/2 hours. This seems reasonable considering a 8 1/2 hour shift. All machines will be writing to the same file server/raid stack.
Kevin Halverson

the solution doesn't necessarily scale by throwing more machines at the problem. your software needs to be able to "fork" or parallelize the input. if the solution space is linearly independent, then there won't be too much of a hit at the file server end but there will be some network and processing overhead, i would add about 10 to 15%. if your computations require information from other processed data sets and you screw up your shared memory management, you'll probably arrive in the morning with a big file of garbage.

 

[Gavin Greenwalt]

Alexander I think Khmuse is planning on sending a frame to each render node which would require a 1.2MB network request and then be completely CPU independent until a ~1.2MB? write after processing so the processing end should be fine like you say as long as you can build a network capable of delivering at least 1Gbps consistantly. That is actually something that is not a simple task and would highly recommend talking to someone like Avid or a specialized network consultant if you don't have something for real-time already. We looked into building it ourself and the difference between our homebuilt option and the Avid solution with 'seemingly' the exact same setup was night and day. The home built introduced an enormous amount of latency in consecutive calls. It did very well in big bursts but very poorly in frame hosting.

The only other problem I'm seeing with khmuse's current plan (and I think he knows it based on his Boxx Apexx question earlier) is it's practically impossible to write to quicktime or avi such as DNxHD from multiple systems. You would need to render out to a frame sequence and then have a seperate machine assemble DNxHDs from a debayered, color corrected and rescaled frame sequence. Which could very easily become a bottleneck unless you use a lightning fast system and might end up being slower than a handful of very fast render nodes with 3 or 4-way SLI 8800s churning through their own r3d file.

*Speaking of which has anyone tried REDCine on an SLI setup?

 

[Mark L. Pederson]

One might want to consider the following:

.r3d files loaded onto single volume on SAN of high speed storage

Four separate 8-core machines each read from the same volume - each runs REDCINE on a 25% selection of the .r3d files and each writes to it's own volume on the SAN.

We have not finished testing - but we are strongly leaning towards dividing up .r3d files over multiple workstations with as many CPU's as possible - each workstation processing independently.

 

[Deanan]

Overloading the server/san with lots of writes from different machines
can really clog things up. One solution is to render locally to each node
and do a pull from the server instead of pushing simultaneously from many nodes.

deanan (built a 120 processor infiniband debayerfarm in a previous life)

 

[Kevin Halverson]

Alexander I think Khmuse is planning on sending a frame to each render node which would require a 1.2MB network request and then be completely CPU independent until a ~1.2MB?

Hey Gavin,

Actually, I was going to send a batch of .r3d's to each machine on the 'farm' and let each write to a single server/raid array. Each machine would have its own local smaller raid and would operate independently of the others. Assuming an adequate disc channels to the server, each machine will run at their maximum native speed regardless of the total number machines (up to a reasonable limit). Having 4 $1500 - $2000 machines should beat out a single $6,000 - $8,000 one and the extra reliability is not to be discounted either.

Kevin Halverson

 

[Mike McCarthy]

This seems to have got a bit off track, but I will offer GPU advice on the original question.
Assuming RedCine is a primary App, you need Nvidia. For Render speed, you want a Geforce 8 series, or Quadro equivalent. They are about 3x more powerful than the best 7 series. The 8800 is the only varient worth using, and from there, the 8800GT is the best bang for the buck, with the NEW GTS512 a good option as well. The 8800GTX as slightly faster, but way louder and power hungry, I have one and am not totally satisfied. The Quadro4600 is hardware equivalent to the 8800GTX, but may have advantages in regards to video overlay and dedicated OpenGL. (And for merely triple the price

I recommend the 8800GT for most purposes, including Redcine. The Quadro4600 will have few advantages in RedCine, but may improve your editing experience, depending on your app.

If I was setting up a workflow for serious Red work on PC, I would get a dedicated render system for RedCine/AE renders, with an 8800GT, and an edit system with a 7950GT for proper Premiere overlay.

 

[John Tissavary]

The biggest argument against the geforce 8 series is total lack of hardware overlay. This has been discussed at length on the cineform forum, where numerous users have had to exchange for 7 series because of this.

And there are some significant openGL operations not available in the geforce line, most noteably openGL logic (end of pipeline) operations, more clip planes, hardware overlay, openGL anti-aliased points & lines, etc...

I'm not sure which of the openGL functions will have a direct effect on RedCine/Scratch... but I do know that a lack of hardware overlay makes the geforce 8 series undesirable for me.


cheers,

jt

 

[Nathan Buxton]

You can pick up a quadro 4500 on ebay for a lot less, and there's absolutely no benefit from running two cards unless you have software that specifically uses both cards, or you have a crapload of monitors.

I bought an fx5500 oem out of a dell system for under $1k... and it works perfectly.

Whatever you do, DO NOT BUY an 8 series geforce. The overlays are completely disabled in that series. Best bet would be a 79xx of some sort, but the drawback with all geForce cards is crippled openGL.

Since Assimilate software relies on openGL gpu performance any geforce is going to be compromised in that area. I don't know if RedCine relies as heavily on openGL as Scratch does, for one thing it deals with a lot less simultaneous operations on the image... perhaps Lucas can enlighten us?

cheers,

jt

No offence but I don't know how accurate some of this advice is.

There is real benefit in running two video cards in SLI mode for GPU intensive applications. Don't blame me for assuming RedCINE would use the functionality as i think it is a hardware level feature and not a software dependent one (after all, both cards feed into the same monitor essentially).

Please elaborate on what overlays are disabled. Do you mean the funtionality to adjust the preferences for overlays? I think you are incorrect. That is a driver-level issue and features such as that can be added with different versions. Maybe the ones supplied by your OEM card manufacturer were crap. I don't have experience with an 8xxx series GeForce, but i think it is safe to assume that the overlay layer is not completely disabled thus rendering windows media player useless.

Another thing: DirectX was not the standard when GeForce was developed. OpenGL has never been a problem for me on GeForce series cards. Often times it performs better than D3D.


I would highly reccomend a high-end GeForce series card in any video-editing PC.

 

[Mark L. Pederson]

I think that if you are putting together a system ONLY for processing footage in REDCINE vs. a system you will use for other applications your graphics card choice can vary.

Rob or Deanan can correct me if I am wrong, but last time I checked, in REDCINE, the GPU is processing COLOR. The CPU(s) are processing debayer and transcode. I strongly suspect that most very high-end graphics cards are over-kill if your box is only going to be used for .r3d "processing".

I am just pointing this out to make the point that IMO - CPU(s) + DISC SPEED are really the key here.

 

[Joe Carney]

Looks good, Zeke.

Although, I'm not sure why you would want the ATI FireGL cards... I'm trying to hold off on purchasing any new systems, especially ones with 45nm CPUs until Intel fixes their problems. Probably another month or so...

Latest generation of ATI OpenGL cards(5600, 7600...) are getting great reiviews and are a LOT less expensive than the Quadros.
I didn't know Intel was having problems with their 45nm Xeons. I'm not planning on purchasing till Jan anyway. Could you expand on the problems?
Is it the cpu or the chipsets?

 

[Jeff Kilgroe]

The biggest argument against the geforce 8 series is total lack of hardware overlay. This has been discussed at length on the cineform forum, where numerous users have had to exchange for 7 series because of this.

And there are some significant openGL operations not available in the geforce line, most noteably openGL logic (end of pipeline) operations, more clip planes, hardware overlay, openGL anti-aliased points & lines, etc...

I'm not sure which of the openGL functions will have a direct effect on RedCine/Scratch... but I do know that a lack of hardware overlay makes the geforce 8 series undesirable for me.

Whoah, there cowboy...

Hardware video overlay is indeed gone. And gone for good. Never coming back, it's a legacy feature and software vendors should have quit using it long ago, but you know... lazy programmers and all that.

I agree, this overlay issue is a serious deal until software catches up. Does Adobe CS3 use video overlay? If so, f**k Adobe, it's not like they didn't know this was coming. What about Cineform? Upcoming cards from ATI aren't going to support this either. It wasn't even supposed to be supported in Windows Vista, but Microsoft still put it in for legacy support because too many beta testers complained. IMO, MS should have left it out... Along with a laundry list of other legacy crap that should have been done away with long ago.

What do you need with more than 2 hardware clipping planes? I don't know of any application that uses more than 2, which the GeForce 8800xxx provides, the 8600 provides 1. Up to 8 planes are offered via software.

Hardware-accelerated points are not there. Never have been in a GeForce card. Only app I have that make use of them is XSI, if I choose to turn it on. Pro/E and SolidWorks use it too. Hardware AA lines are there, not sure what you're talking about, at least they are present in the 8800 line. They've been a part of the GeForce GPUs since the 6800 Ultra was released. Hardware GL overlay (not to be confused with the video overlay that is missing) you are provided one overlay plane on GeForce 5900 and newer. Quadro cards support more. Most GL apps make use of only one, but Maya, XSI and a few others can use more.

Everything else you mention are Quadro-specific functions that no GeForce card has ever had. If you're not running software that uses these functions, then why spend the extra money for the Quadro?

For a system primarily for NLE use and REDCINE, there is no reason what so ever to buy a Quadro card. Now, the video overlay issue with newer cards is a serious concern and I would make sure whatever NLE or other software that will be run in addition to REDCINE will work with the newer cards. If not, get a 7950GTX Ultra 512MB card and save yourself some cash. it's still a very good video card and pushes nearly the same amount of geometry as the 8800GTX. It doesn't have the texture and pipeline abilities, but you won't notice or care if you're not a gamer. I don't think the higher-performance GPUs are doing much for REDCINE... The bottlenecks are going to be your CPU(s) and storage system.

 

[Jeff Kilgroe]

Latest generation of ATI OpenGL cards(5600, 7600...) are getting great reiviews and are a LOT less expensive than the Quadros.
I didn't know Intel was having problems with their 45nm Xeons. I'm not planning on purchasing till Jan anyway. Could you expand on the problems?
Is it the cpu or the chipsets?

I'll take another look at the ATI FireGL cards to see what's up... They have always had great hardware, but the software/drivers have been terrible. If they cleaned up the software and improved their OpenGL support, then yes, definitely worth a look.

Intel has put a delay on shipping the rest of their 45nm CPU models that were due to ship by the end of this month. They're having some stability issues with the CPUs as well as the new chipsets. That's all we know publicly right now... Oh, and they're probably not in a hurry since AMD's latest offerings have fallen flat on their face, so no direct competition right now and no incentive to hurry to market. Hopefully it's a quick fix and they still can push the new CPUs and chipsets out the door. I need to buy a new Mac Pro, but I can't justify spending the current prices on what Apple is offering right now. Ouch. I'll probably have to pick one up on eBay or something because I'll need a new one within the next 30 days.

 

[Gavin Greenwalt]

ATI is putting out stable drivers? Has hell frozen over?

 

[Joe Carney]

ATI is putting out stable drivers? Has hell frozen over?

Why.....yes it has. They even have non beta drivers for Vista 64.

 

[Rob Lohman]

ATI (X1900) for Mac, NVidia for Windows. You don't really need to get the ultra expensive Quadro just for REDCINE. If you get Scratch then they have a list of graphics cards they advice.

 

[JustMe]

ATI (X1900) for Mac, NVidia for Windows. You don't really need to get the ultra expensive Quadro just for REDCINE. If you get Scratch then they have a list of graphics cards they advice.

Yet RedCine is Scratch, so it needs what Assimilate wants even at its base...:bleh:

 

[Mark L. Pederson]

Yet RedCine is Scratch, so it needs what Assimilate wants even at its base...:bleh:

I have both - and I can assure you - RedCine is NOT Scratch. You do NOT need what Scratch needs to run REDCINE at max performance.

 

[JustMe]

I have both - and I can assure you - RedCine is NOT Scratch. You do NOT need what Scratch needs to run REDCINE at max performance.

We can all see that Redcine is just the core of scratch with the "redcode" node being a modified jpeg2k import.

the same base api's apply.
Scratch has never supported some cards and now has to via the Redcine program.

 

[JustMe]

but just for giggles a dell m65 with a QFX 350m works