Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

How much horsepower needed to edit 4k?

IMGentertainment said:
Hey Rob I don't remeber if you or anyone stated on DVXuser. But is REDCINE going to be able to acess a cluster of servers and utiliz them for processing...
Click to expand...

That was my very next question. Please say I can use my render farm... Puhleeeze. ;)
 
AppliedVisual said:
Whoa, there... lay off the Sony kool-aid. I've personally worked with Cell and I'm working with it right now on an embedded control systems project. I knnow what it can and can't do. Cell is a very nice CPU, but in reality it's just a PowerPC 907 series that has all the legacy junk stripped off so it's not backwards compatible with anything. It's trimmed and lean and mean and has some nice abilities like running individual cores at arbitrary speeds. More cores (like 3, 7 and 9 core variants of Cell) are easily created on the CPU die because it's trimmed down and with their current 65nm process, they can fit more of these smaller CPU cores on a single chip. Intel won't hit unified quad-core and 8, 12 and 16 core CPUs until they transition x86 to 45nm later this year.

As for what PS3 can do, the true power of PS3 is not the Cell processor. We heard all this same type of crap the last time around with PS2 and the EmotionEngine CPU - (hybrid MIPS R5400). You can chalk up the Trillions of operations per second claim with the PS3/Cell right along with the "it's illegal to export PSX and PS2 to the middle east because they're classified as supercomputers" rumor. Sony is making the trillions of operations per second claim based on the number of cores * number of possible pipeline steps * MHz. It's a bogus claim, but there is no industry standard that specifically declares what an "operation" is. Perhaps I've spent too much time in game development and I'm glad I don't do it anymore, but we have to look at the hardware for what it is... In the PS3's case, it is first and foremost a game system. It is designed to play games and it does that rather well -- although the current crop of games are junk. If Cell were half as revolutionary as the Sony hype juggernaut makes it out to be, then people would be tossing x86, Itanium/EPIC and just about every other major CPU architecture in the trash in favor of Cell.

The real power of the PS3 is the nVidia GPU, which has been tweaked to work in tandem with Cell, the onboard memory architecture and common HD video resolutions. Additionally, game developers can specifically optimize code for the PS3 platform and it allows them more freedom to create visually impressive games than it does with current PC hardware due to no need for maintaining compatibility with countless hardware configurations. 6 months from now, PC games will look far better than PS3 games - just as what happened with the PC vs. PS2 and XBOX. IMO, there's a few PC games right now that are on par with what PS3 is showing.

And let's not forget all those revolutionary Cell-based products and turn-key systems that Sony has been bragging about for the last three years. None of which have come to market, let alone have even been demonstrated with any amount of reassurance that they will come to market. Remember, we're talking about the same Sony that was busted at CES in '99 for hyping up PS2 and EmotionEngine by showing a demo of a real-time rendered ball covered with over 2 million interactive hair fibers. ...When I say they got busted, they had to fess up and admit to the demo running on an SGI Onyx2 with dual RealityEngine graphics pipes and 4xR10K 200MHz CPUs (think $80K computer at the time) was tucked under a table behind the booth. And then they had the brass cajones to claim it was representative of what the EE/PS2 could do when released. Even though the EE was nothing more than an R5400 CPU with partially integrated nVidia graphics chip.

OK, I'm off my Sony rant... Sorry everybody. But their CE division and PlayStation / Media marketing has no shame.
Click to expand...

*Puts the koolaid down*. I really dont know much about the PS3 but thanks for illustrating the way it works. But I am glad to know that NVIDIA was the hero in this one. Which is why you may have noticed that my rant was about GPU's in the first place.

The point I was trying to make is that there is a paradigm shift from raw processing power to clever processing power. Why use a "virtual" processor so to speak when you can have the most common functions you need already hardwired ready to go as in the case of the GPU. My point is that undeniably, this is where computing is heading. Cell or Not.
 
REDCINE will not support a render farm, we may have some other options but can't say much about that at this point in time.

If you have multiple machines split footage (clips) over these machines and have each process a part of what you shot. I know this involves a bit more work on the user's part, but it does give you more control (and allows you to generate QuickTime files which is not possible in a render farm setup).


On editing with REDCODE, keep in mind that it's not just huge resolution RAW file. That would be easy. Wavelet decompression is expensive to do (cpu wise) and at these high levels debayer time starts to really make an impact as well.
 
acehole said:
The point I was trying to make is that there is a paradigm shift from raw processing power to clever processing power. Why use a "virtual" processor so to speak when you can have the most common functions you need already hardwired ready to go as in the case of the GPU. My point is that undeniably, this is where computing is heading. Cell or Not.
Click to expand...

Yep. There's been lots of talk within the industry for years about using GPU functions to aid in intense processing tasks. Over the years, several have tried it, but few have succeeded in making something worth-while. But I think GPU power is finally at a stage where it is truly powerful. The texture/image processing functions on most current GPUs are incredibly powerful and they offer abilities in real-time that take even the latest CPUs several minutes or even hours to render. This is why we're now seeing advanced GPU functions being utilized in upcoming OS releases from MS and Apple. The GPU is no longer relegated to "that chip on your video card which helps you play cool games". It is another processor in your computer that deserves equal attention and should share the same status as your CPU because in many ways it's just as powerful, in some ways more and should be used for far more than pumping texture polygons into the latest version of Quake.

As for Cell, I don't mean to bash it, it's a pretty good product and can fit so many uses a lot better than x86 chips and Cell is much more powerful than ARM or older Motorola and MIPS chips commonly used sor such things. Think compact, streamlined embedded applications. Cell makes perfect sense for autonomous applications for manufacturing systems and robotics. Think robotic arms in atuo assembly lines where each motor/joint of the arm has it's own "brain". Cell's biggest strength is how easy it is to scale and link collections of the processor to form larger node-based CPU clusters. Hence its name... If Cell continues to develop and follow IBM's intended path, you won't see it become the CPU in a desktop workstation. But that new 32-core Mac pro you buy in 2012 could very well have various forms of Cell processors in it to control secondary device functions like RAID subsystems, coordinate external interfaces like multiple FW800 busses, etc..

The reason why the Sony hype machine uses Cell to promote PS3 is because it's a cool sounding new thing that most people are unfamiliar with. It's all marketing... They can make it out to be what they want people to think it is very easily. They can't hype the exclusive nVidia GPU, even though that's pretty cool in itself. Why? Every gamer knows that the nVidia chip, which Sony has exclusive rights to today, is nothing more than a $300 PC video card within the next two months. And there will also be a $550 model at that time that's nearly twice as good. Actually this has already happened. We're just waiting for PC software to catch up to GPU tech...
 
Rob Lohman said:
REDCINE will not support a render farm, we may have some other options but can't say much about that at this point in time.
Click to expand...

Hmmm... OK. The anticipation over RED and how you guys are going to make this all work is killing me. :)

If you have multiple machines split footage (clips) over these machines and have each process a part of what you shot. I know this involves a bit more work on the user's part, but it does give you more control (and allows you to generate QuickTime files which is not possible in a render farm setup).
Click to expand...

...And such is often the workflow for larger animation projects. Can't just dump everything to the render farm, so I'm already used to this sort of thing. As for building QT files on a render farm, I know it's not possible to have multiple render node processes trying to build on the same QT file. But it's possible to build QT and other video files in a farm setup. I have it working here with software I wrote. It's actually nothing special, just one CPU or render node will start to write the desired video file from the rendered frames as they come in. I think there's a few freeware utilities out there that do the same thing. :D

Hmmm... Now I wonder how REDcine would perform in a cluster environment.

On editing with REDCODE, keep in mind that it's not just huge resolution RAW file. That would be easy. Wavelet decompression is expensive to do (cpu wise) and at these high levels debayer time starts to really make an impact as well.
Click to expand...

Yep. I'm aware of that... And in an editing scenario, it's all about what the editing workstation can handle. Hence my questions of support for multiple CPUs. That 8-core, stoakley chipset Mac Pro can't get here fast enough. :cool:
 
Well Rob that brings up the question.. How dose apple's compressior do it? I have it set up across an xserve cluster and it makes me Quicktime files when its done... Is it not really using the cluster? If so I think I found out how to pay for my camera...
 
As indicated by Jeff it's certainly possible, but more difficult (to implement).

I was a bit quick to say it's not possible. However, to make that work we'd have to dig deep into QuickTime to generate such files and I'm not sure that's worth it.

Whether or not compressor is using your xserve cluster for QuickTime generation I don't know. Shouldn't be too hard to find out through activity monitor and some network monitoring tools.

Rest assured we are looking into all kinds of directions & technologies.
 
I think it's probably relatively easy to do with CBR codecs, as you can generate a placeholder Quicktime of black frames with the correct duration and frame size, then the frames can just be overwritten with the rendered frames as they are generated. Obviously with a VBR codec (such as REDCODE) it can't work the same way. That method could however potentially be used by a (future) networked version of REDCINE to generate editing proxys using a codec like DV or DVCPro HD.

Nick

EDIT: I say "relatively easy" but I am talking from a non programmer's point of view about concepts, not implementation. My last serious bit of programming was in Z80 assembler for my Sinclair Spectrum, so what do I know?!
 
Hey thanks Rob.. I checked and it its utilizing the Xserves so for now I'll keep them. The question is right now you are not looking a supporting this feature and maybe in the futer you would. I am just wondering if we are going to see allot of gain. AKA the RED bar moving faster in processing images. I aske this beacuse I know when we shoot features and for two company's we work for all the time. They want dailys the next day or day of if possible. If we are shooting all day on two RED cameras. I wonder if it would be extremely benifital to send that footage through a "Render Farm" vs just one machine... What do you think... And even for final render of the project that could be going out say DPX would then it be easier..
 
Rob Lohman said:
If you have multiple machines split footage (clips) over these machines and have each process a part of what you shot.
Click to expand...

How will this be affected by any licensing that may be used by REDCINE? Will the license that we get with the camera permit us to install the software on multiple machines to divide up tasks? I would hope that at the very least it will permit use on one MacBook Pro for use in the field, and one Mac Pro in the edit suite, but multiple machines would obviously be very useful.

Nick
 
There are several ways to write video files with frames or intermediate data coming from multiple sources. Most 3D rendering applications out there will allow fro some forms of video files to be written from their node-based render modules. However, I almost always make it a practice not to do it -- one node crashes and it just throws everything else off... Whereas rendering individual frames is much more reliable and fault-tolerant. As for compressor, the primary application instance still takes care of writing the QT file when in a clustered configuration. It assembles it as the data from the other cluster nodes comes in and the next piece in order is available. It's not a huge trick to do this, that's how just about every other node-based program that encodes video files works. All it can do is wait for each frame or next sequential piece of data to come in and then tack it to the end of the video file. For complex encoding tasks such as MPEG2 compression in Compressor or other apps, often data between cluster nodes must be compared and averaged before resulting data can be written to the file. Once again, not a huge deal, but as data sizes, clip duration and cluster sizes grow, it becomes increasingly less efficeint.
 
Thanks Jeff that is helpfull.. So I guess the best configuration would be to have several different machines running processing different clips and dumping them onto one RAID (or several for back up). I guess if REDCINE has some kind of batch processing where we can set a number of clips and Queue them up so the next clip starts I don't see this to be an issue, If indeed a cluster type situation is less effecient.
 
Indeed, for now use multiple machines with REDCINE. We are still working out licensing for REDCINE and so on. We do want you to use your footage so even if you do need additional licenses they will be in line with the rest of the pricing.
 
A cluster or farm of nodes isn't necessarily less efficeint, but that depnds on the work being done. Where nodal compute configurations lose their efficeincy is in transaction speed and bandwidth. If you're spending more time transferring data back and forth between nodes than actually computing results, you're spinning your wheels. This can happen pretty easily when trying to have separate nodes generate data which depends on data generated by other nodes. It is far more efficient to have the nodes work in a linear manner. All nodes are presented with the core data set such as a RAW video clip or 3D animation scene. Then individual nodes can be instructed to begin processing a single frame or small linear sequence of frames. They complete, deposit the frames to the target storage volume and request new instructions from the host node. It's fast, it's efficient. If nodes are suddenly processing intermediate data for long GOP or IFrame video formats where they have to await full frames from other nodes and quantize their data produced with that of other nodes, we start trafficking large volumes of data. There's ways to combat this effect too when processing large data sets. Such as staggered processes where different nodes handle different render phases, etc..

I'm assuming that REDCINE has some form of batch processing, it would have to.. Or if nothing else, a solid command-line interface so that it can be adapted to various batch / render / workflow managers like Chalice, Spider, AlienBrain, etc..

Anyway, I trust Rob and the rest of the RED team to deliver something that will do what most people need. As with any new software, there should be more features and abilities to come with future revisions. I'm sure there will be requests for features which weren't obvious during development, but make perfect sense once the camera is in the hands of users. For now, we don't know much and we just have to wait and see what NAB and the release of RED will bring us.
 
Very True Jeff. I think that Rob and the guys at read are doing a fantastic job and REDCINE will be a top of the line product and all this talk has proably already been discussed numbers of times in the RED war room.

Thanks ROB, one last question if you can answer it. In Jeffs post he mentions having a node based farm it would be more efficient to have the nodes work in a linear manner. Will REDCINE be able to controll a host node and tell the other nodes to render and drop the footage on a Raid...
 
No, the current version of REDCINE only knows about itself and nothing else. We'll see what the future brings.
 
Thanks Rob for the insight.

Yes the future is looking brighter and brighter everyday and I think you guys are doing a killer job. And just like every new production pipeline. You just find the best way to work with it and make it work. REDCINE no dought will grow and different methods will be thought of and that is all part of the process.
 
Emanuel said:
Here is some info related:

http://www.reduser.net/forum/showpost.php?p=390&postcount=10

http://www.reduser.net/forum/showpost.php?p=704&postcount=16
Click to expand...
http://www.reduser.net/forum/showpost.php?p=390&postcount=10

Mardi_Gras said:
zakaree said:
is 2k too much for an at home online session?
or will it pretty much have to be 1080 if u wanna do it yourself?
Click to expand...
Onlining 2k assets at home, like its bigger brother, 4k, don't come cheap. Even getting a good 1080p online at home is a big deal. You can easily run up thousands of $$$ before cutting a single frame of media. Sh*#@ don't come cheap! My best guess is that most non to average techies like me will be offlining @ 480 and exporting for a professional 1080p posting, just like Jaime outlined above:D
Click to expand...
$5,000 for 4K online editing is the David Newman's estimated figure:

«We just demonstrated real-time online 4K RAW editing (dual stream dissolve with Dalsa Origin footage) to Adobe management under Premiere Pro 2.0 running on a Dual core dual proc Xeon Woodcrest system (about $3K PC with two drives in RAID-0.) So it can be done today, although we aren't shipping this technology yet, but we intend this be available for Red customers as the alternative workflow (for those who can't stand offline workflows.) This a simple extension of the now shipping CineForm RAW workflow. So even with the purchase of our software the system costs will be doable at $5k for a Red online.»

LINK

FOR MORE INFO >> CLICK HERE

http://www.reduser.net/forum/showpost.php?p=704&postcount=16

David Newman said:
1) Two 4K video streams of CineForm RAW can be played on a dual proc dual core Intel Woodcrest -- quad core may do better (note: our 3Ghz Woodcrest, was slightly out-performing a 2.4Ghz dual proc quad core.) With half res'ing wavelet tricks more real-time layers will be supported.
2) Color correction, transitions, titles, etc in RT. The basics are in realtime.
3) Around 1-2 frames per second on a Woodcrest Intel system -- this lowish render speed is due to the demosaic required to convert form RAW to RGB.
4) No real need. You source is compressed (whether REDCODE or CineForm RAW) and for post renders to CineForm Intermediate this can have as little compression as the user desires. If you wish to decompress to DPX then conform those, the quality is not visibly different even for extreme workflows.

David Newman
CTO, CineForm
blog : cineform.blogspot.com
web : www.cineform.com
Click to expand...
 
You must log in or register to reply here.
Back
Top