WHY!!!! are people still using color baked DPX's F&%*@#%*#K!

[Nick Shaw]

Maybe from RED's perspective (i don't know) a prores module that records 1080P is a patchwork, temporary solution that can already be accomplished in many ways if that is what you want.

What would probably be better (if RED were open to the idea) would be for the 3rd party recorder manufacturers to make a version of their recorders in a DSMC Module form factor, so it sat neatly on the back of the Epic/Scarlet, drew power from the module connector, and passed through all the pins of that connector to other modules.

I imagine that power, meta-data and a 1080p signal (for the I/O module) are probably all available on the module connector, but I don't know if the pin-out is publicly available. I also imagine RED would want to approve any 3rd party manufacturer who made such a module.

 

[David Battistella]

Nick,

That is still R&D, product development, prototyping, FCC electrical device approval, working with a third party (which RED tends to do in a very limited way). By the time that module was "ready" it would probably be obsolete.

I'd say the bar is 4K "ready to edit files" probalby the fasted way to that end is opening up GPU acceration or making the ROCKET so cheap that everyone just buys one. That might be far more productive than tons of engineering time toward another module.

Modules are nice but I like the EPIC brain form factor, The darn EPIC will look like Santa's Sleigh with all these modules attached....

We are trying to get AWAY from the size and weight of teh RED one.

16mm flexibility 35mm picture quality. Bring that on!


David

 

[Mark Toia]

I just like having so many options at my finger tips already ))))

I'm happy ( if I just do it myself )

 

[Brice Ansel]

I'd say the bar is 4K "ready to edit files" probalby the fasted way to that end is opening up GPU acceration or making the ROCKET so cheap that everyone just buys one. That might be far more productive than tons of engineering time toward another module.

Modules are nice but I like the EPIC brain form factor, The darn EPIC will look like Santa's Sleigh with all these modules attached....

We are trying to get AWAY from the size and weight of teh RED one.

16mm flexibility 35mm picture quality. Bring that on!


David

Absolutely!

 

[Martin Stevens]

I have read most of this thread (Great thread!) and I noticed no mention of Render Nodes to avoid using a RedRocket Card.

Is this not an option with the Red Software, as in, you cannot send the files to be processed through a Render Farm etc.?

 

[JanneJansson]

It's not 'slightly more sharp' for me.
On RCX beta 10, I have to take the Rocket version and size it down to 70% , then back up to 100% to see about the same detail level.
So a 4K file has about 2.8K worth of detail. Sharpening the CPU version is not getting the detail back, no matter what de-convolution fancy/shmancy sharpen I tried.

Janne : Can you try beta 10 to see if you see something different ? Perhaps Beta10 broke something.... Thanks !

I had big problem with beta 10, could not enable full debayer. Returned to beta 6.

 

[Subhadip Sen]

There is a different side to what Mark is saying and I am surprised that nobody mentioned it yet!

It will come down to OSX and WIN again but I am among many people who have never even touched PRORES once for a job. I never used it, I never needed it and I know MANY people who feel the exact same way. PRORES is presented as the ultimate solution in this thread when in reality R3Ds are much easier to deal with for people who work with PPro or SonyVegas on Windows.

I also would like to add that Arri's ProRes workflow is not as easy for everyone as people in this thread suggest. I actually KNOW and have discussed with DPs who are not RED camera owners like me and who actually prefer to shoot with RED just BECAUSE they don't want to have to deal with Prores!

There's no doubt that ProRes exists only because Final Cut Pro upto 7 was not capable of real-time performance on native acquisition formats. Vegas and later Premiere Pro could, which is why I moved away from Final Cut many years ago. It is just much easier and better to edit natively than going through a lossy format. Granted, the finishing app relinking back to the native format means there is no quality loss, but it still means massive time and space is wasted on the transcoded material.

Until recently, there was no way to read ProRes on Linux or encode ProRes on Linux and Windows. That it has become a standard deliverable format is most unfortunate for non-FCP / non-OS X users, but at least workarounds exist now. Hopefully the industry will come to its senses and choose a cross-platform and preferably open standard format for future deliverables and acquisition.

It's not 'slightly more sharp' for me.
On RCX beta 10, I have to take the Rocket version and size it down to 70% , then back up to 100% to see about the same detail level.
So a 4K file has about 2.8K worth of detail. Sharpening the CPU version is not getting the detail back, no matter what de-convolution fancy/shmancy sharpen I tried.

Janne : Can you try beta 10 to see if you see something different ? Perhaps Beta10 broke something.... Thanks !

Like Janne says, pretty sure it's a bug in Beta 10. I haven't used Beta 10, which is why I couldn't see the significant difference you are seeing. The perils of using beta software!

 

[Nick Shaw]

There's no doubt that ProRes exists only because Final Cut Pro upto 7 was not capable of real-time performance on native acquisition formats.

I don't think I agree with that statement.

ProRes dates back to when the vast majority of edited material was digitised from tape (not that long ago!) FCP was always (well, since v4 if memory serves) perfectly capable of capturing uncompressed to a fast RAID using a BlackMagic or AJA card, and working with that material in real time. ProRes was created as a compressed format, to allow editors to work with HD material on drives which would previously only have been capable of sustaining uncompressed SD playback.

It is true to say that some code optimisations were made to FCP with the introduction of ProRes which permitted better real-time performance with ProRes than other codecs.

 

[Mark Pugh]

If the Alexa just shot ARRI RAW like the D21... I wouldn't be having this conversation with you.

I just wish RED took a one leaf out of there book, only one little page... And that would be the PRO RES LOG bit .. (with no module hanging of the back).. choice of 4k or 2k with a touch of a button would be even better.

Yep. Without Prores the Alexa would still be a great camera, but if Red had taken up Prores, as an option instead of Arri, we'd be living in a different universe.
Jim's built the camera that he wants, and his likely response wil be, as always, is that if you want Prores, go get it over there...
For many of those that own his cameras, though, life would be that much better with a LOG Prores 4444 option. Rental returns would be way up also. Red's sales would.be up immensely, without a doubt.

A module solution will only do half the trick, though. It will record an off-speed shot only at project frame rate, unless ingesting the shot via playback,

 

[Scott Crawley]

ProRes was created as a compressed format, to allow editors to work with HD material on drives which would previously only have been capable of sustaining uncompressed SD playback.

Yup. IE: Laptops.

ProRes enabled us to take it on the road. As I recall, that was it's primary purpose in the beginning. When folks discovered it was good enough to finish in, they began to take advantage of it on other systems as well.

 

[Jeff Kilgroe]

For various reasons, "maximum achievable picture quality" is not always at the top of the list when making post decisions, it gets weighed against other costs and/or time restrictions, plus it's not always the highest priority with some producers, production companies, and distributors. And it's not just a post issue, it starts in pre-production. You don't always rent the best locations possible, you don't even go to the best city to shoot possible, you don't get the best actors in the world, you don't always hire the best crews possible, you don't get the best lenses possible, on and on and on. From Day One, for most productions, it's a process of minimizing the damage caused by an endless series of small and large compromises. And yes, some decisions are less justifiable than others, some are even downright stupid.

Word. And... AMEN!

Actually you need to mention #4.
Iridas demoed real time R3D GPU debayer 2 years ago at NAB only to receive a phone call from Red to knock it off.
One more thing. Red debayer actually quite a bit more difficult in comparison to ARRI RAW on the account, that ARRI is uncompressed. And we all know, that R3D is. Big diff.
But, as Iridas already showed, it is doable, even on a two year old GPU...

Actually, the Iridas demo was 3 years ago. Same time Cineform was showing their realtime native R3D solution. They were told by RED to stop it, which RED had the right to do, because it involved reverse-engineering of the codec. This was all before RED made the SDK available.

Now we're back to the GPU discussion again... Neither the Cineform, nor the Iridas solution (then, or now) used the GPU for the wavelet decompression. And no one is going to. It's just not well-suited to a GPU task. And yet, it accounts for probably 60% or better of the amount of compute cycles needed to decode and display an R3D frame. On the other hand, GPUs are great for processing Bayer-pattern RGB data. The RED SDK does an admirable job of allowing developers to multi-thread the frame decoding and seems to be well-optimized for the wavelet decoding itself. Unfortunately, demosaic and Bayer operations are still all CPU-bound (or offloaded to the RED Rocket when available). Looking at it from RED's point of view, it makes sense. This way, they can guaranty the same results across platforms. With the GPU-accelerated solutions this has not been the case in the past. It's hard to explain it to people, but precision varies between CUDA (and OpenCL) implementations on different cards, even from the same vendor. RED would have to profile and tweak settings for each card they would "certify" and then they would be reliant on these third-party manufacturers to continue producing these cards and not change things down the road with future drivers.

OK, that was all fine and good last year, even a plausible argument a few months ago, but it's starting to wear thin. The current revision of OpenCL has evolved quite a bit and now, unlike CUDA, it actually solidifies precision across GPU platforms (at a performance hit with many GPUs, but for the greater good). nVidia's just-released CUDA update does this for the first time as well, but only across GPU families. And it of course only works on nVidia GPUs. So, IMNSHO, I feel that RED made the right choice by not supporting generic GPU decode and acceleration in the past. However, it is now time for them to embrace it. I feel the Rocket has run its course, and honestly speaking, I don't feel it has ever lived up to its potential or intended purpose. Outside of REDCINE-X, we can't use it for monitoring. We can't access the Rocket frame buffer to manipulate the image onboard... Even with it accelerating, it's near impossible to get reliable info and data to/from the card to give us reliable software scopes in real-time. While I have dabbled with the SDK quite a bit and created a few things, I haven't released anything publicly, but it's taught me enough to see where the shortcomings and pitfalls are. I'm still impressed with what the Foundry achieved with Storm. Even though it always felt like they were just making another REDCINE-X.

Anyway, I think it's time RED should start moving to OpenCL acceleration for their Bayer processing. Sure, the sales for the RED Rocket will all but stop cold, but let's face it. Other than full-rez transcoding, the Rocket is of little use right now. Any current top-end PC that can be had for less than the price of the Rocket, can just about keep up for transcoding purposes. When the SB-E series 8-core Xeons drop here in the coming weeks, along with the new chipsets, I really wonder who will be buying a RED Rocket...? Sure, if your software still relies on it, but how long is that going to last. The 2.8GHz with 3.4GHz turbo variant of those CPUs are going to be the same price as the 6-core 3.06GHz Westmere Xeons and not only will they have two additional cores and a larger cache, they are approximately 22% faster per core, per clock. That's nearly 50% faster when under load per core and turbo is kicked in. I can edit 4K and 5K in NLE's like Premiere and Vegas on some pretty lowly hardware via the lower-rez extractions right now, I've got a PC here that will all but do 4K full rez at 24fps in Premiere. It will handle 2 streams of 4K or 5K with editing rez set at 1/2 just fine. I bet in the next 30 to 60 days, I'll be able to build a new PC for much less money than the Rocket that can actually transcode faster than a Rocket, and doing so by using brute-force, hammer the CPUs approach to do it...

SDKs and GPUs aside, I will have to add to what others have said here. One of RED's biggest weaknesses is documentation. Efforts like Reducation and even the continuous input by many of us here on the forums will help to educate. However, there are a great many who only go off of hearsay or need to see the current info in a PDF or white paper of some sort that lays everything out for them.

And when we talk about RAW and debayer processing, GPU acceleration, etc.. Let's not get too overzealous with what we think the competition offers. ARRI doesn't just let anyone create their own GPU-accelerated debayer for ARRIRAW. They take the same measures (or very similar), that RED does in controlling how it's done. It's easy to demonstrate real-time ARRIRAW processing. First and foremost, there's no compute-expensive wavelet decoding that needs to happen to get your bayer frame data out of its container. Next up, it's maximum current size is 3072x1620... For the mathematically challenged, the 3K ARRIRAW frame is only 59% the size of a RED 4K 2:1 frame and 7% smaller than a RED 3K 16:9 frame. I've been able to push around 3K R3D files at full rez for years now... My old 15" MBP from '06, using a lowly eSATA drive for workspace in the field, could edit (in Adobe CS4) 3K at half-rez... No problems. Seriously, let's see what ARRI delivers for a 4K camera (or 5K, 6K, etc..) and how well that performs.

Sorry if I'm ranting a bit here, but I've run into this a couple times this month already... "We can process ARRI RAW in real-time, but our systems choke on EPIC 5K... RED sucks...". Never mind that ARRIRAW frame is only 36% of the number of pixels and 36% of the total uncompressed bayer data, 36% of the total resolution or detail and has relatively no decompression overhead. 5K full on EPIC is 13.8MP in size... 16bit RAW... That's 26.4MB per FRAME for RAW and uncompressed 80MB per FRAME as 16bit RGB. That is 1.9GB/s for 24fps motion... Which leads us to one other concern with GPU acceleration. Most GPUs are notoriously slow for getting data off the GPU card and back onto the PCIe or system bus. On a PCIe 2.0 bus, you need the dedicated outbound capacity from your GPU of 4 lanes to accommodate that and to put that into perspective, most all current GPUs, even the Quadro 6000 and GeForce GTX585 don't have that sort of outboard data throughput. Sure, they exist as 16-lane devices and can swallow massive amounts of incoming data and process it for display to a monitor via their onboard path, but most of them have little more output capacity than what is required to build and send out framebuffer data to support portions of a ~2K framebuffer used in SLI mode for games. Most SLI configs choke on displaying to dual 27 or 30 inch displays because of the outboard bandwidth from individual cards. It usually takes 3 cards in SLI to swallow dual 24" and larger (1080p +) screens.

...Really. I think RED should jump on the OpenCL bandwagon here and offer it as a solution. I think many people will be surprised by what they find and not necessarily in a good way. Of course, it will all still be RED's fault. It sucks being pioneers in an industry at times. RED is no different than those who came before in the audio (or even video) industry, who decided to disrupt the incremental flow of technology. Sure, in another 2 to 5 years, we'll all be tossing around 4K video clips on common destkop PC's, even (dare I say it, on tablets), like they're no big deal. Until then, it is what it is... Just like real-time 16bit audio processing in the early '90s, it takes a dedicated accelerator card to do certain tasks now. This is rapidly changing. How many of you here bought MPEG-2 accelerator cards so we didn't have to wait for 3 days for our Pentium-II systems to encode a DVD? Come on, it's OK, y'all can admit it.

 

[Jeff Kilgroe]

And well, crap... After I wrote all that, I saw Rob posted this:

Good discussion. We are listening.

I want to correct a piece of wrong information. The reason we have a Rocket has nothing to do with the debayer/demosiac, it has everything to do with the decompression. Bruce: GPU can't do our decompression faster, trust me, I've tried. All those other cameras don't do compression + 4K/5K. ProRes 1080p ain't the same game.

So if you get bored reading my massive rant above, here is one of my key points in a nutshell...

Thanks, Rob.

 

[Sergio Perez]

I did have a Pinnacle DV500 back then for real time dvcam editing with multiple streams...

 

[jake blackstone]

Let's not get too overzealous with what we think the competition offers. ARRI doesn't just let anyone create their own GPU-accelerated debayer for ARRIRAW. They take the same measures (or very similar), that RED does in controlling how it's done.

Jeff, as always great post.
But on Arri RAW 3rd party access I'd have to disagree.
http://www.gluetools.com/products_arriraw.html

 

[Paul Russell]

I like the way Premiere has integrated Red into the workflow. Using 'source properties' i get almost all the power of redCineX.
Once I get a Red Rocket (and the necessary mobo upgrade to acccomodate sufficient PCIe slots), I'll be editing in Raw with no transcoding, hopefully sometime between 4-10th February.

 

[Subhadip Sen]

Absolutely agree with everything Jeff has said about modern hardware, the non-requirement of Rocket, the move to OpenCL, etc.!

Sorry if I'm ranting a bit here, but I've run into this a couple times this month already... "We can process ARRI RAW in real-time, but our systems choke on EPIC 5K... RED sucks...". Never mind that ARRIRAW frame is only 36% of the number of pixels and 36% of the total uncompressed bayer data, 36% of the total resolution or detail and has relatively no decompression overhead. 5K full on EPIC is 13.8MP in size... 16bit RAW... That's 26.4MB per FRAME for RAW and uncompressed 80MB per FRAME as 16bit RGB. That is 1.9GB/s for 24fps motion... Which leads us to one other concern with GPU acceleration. Most GPUs are notoriously slow for getting data off the GPU card and back onto the PCIe or system bus. On a PCIe 2.0 bus, you need the dedicated outbound capacity from your GPU of 4 lanes to accommodate that and to put that into perspective, most all current GPUs, even the Quadro 6000 and GeForce GTX585 don't have that sort of outboard data throughput. Sure, they exist as 16-lane devices and can swallow massive amounts of incoming data and process it for display to a monitor via their onboard path, but most of them have little more output capacity than what is required to build and send out framebuffer data to support portions of a ~2K framebuffer used in SLI mode for games. Most SLI configs choke on displaying to dual 27 or 30 inch displays because of the outboard bandwidth from individual cards. It usually takes 3 cards in SLI to swallow dual 24" and larger (1080p +) screens.

AMD's Radeon HD 7970 changes all of that. Both directions 16GB/s, partly thanks to PCIe 3.0 but also brand new uncore and I/O. 3GB framebuffer @ 260 GB/s to asymmetric Render Output units which are no longer tied to specific memory channels (this is where much of the bottlenecks were in the past and led to some ROPs remaining idle). It is absolutely awe-inspiring to see a single HD 7970 drive three 24" 1920x1200 monitors - effective resolution 5760x1200 - with the most demanding game out there today (Battlefield 3) at Ultra settings at over 30 frames per second. There are reviews out there which show less demanding games (but still contemporary DX11 games) run just fine with 3 x 30" monitors at almost 8K resolutions on a single HD 7970. Theoretically, 4 x HD 7970s in CFX is capable of 16K - waiting for someone to test that out! Of course, AMD has always had the leadership in multi-display and high resolution since Eyefinity but it is amazing how powerful GPUs are getting! Not to mention the 7970's new GCN architecture shows incredible performance and potential for compute (DirectCompute or OpenCL) applications. Perhaps GPUs were not suitable for R3D debayer/decompression on GTX 285 only a couple of years back. But it is 2012, HD 7970 is 4 times as fast, with a much more compute oriented architecture. If I may request - it is perhaps time to give GPU acceleration another shot. At least move the debayer to the GPU. I don't know the technical specifcations of REDCODE or JPEG2000 decode, but I have used CUJ2K which noticeably accelerates JPEG2000 encoding using GPUs. (that probably has nothing to do with REDCODE decoding, but just saying...) In any case, the GPU must be utilized to whatever extent possible - 4 Teraflops of raw power on a single GPU is too much to be wasted. Even limited it will surely take some burden off the CPU.

 

[Les Dittert]

. . Any current top-end PC that can be had for less than the price of the Rocket, can just about keep up for transcoding purposes. When the SB-E series 8-core Xeons drop here in the coming weeks, along with the new chipsets, I really wonder who will be buying a RED Rocket...? Sure, if your software still relies on it, but how long is that going to last. The 2.8GHz with 3.4GHz turbo variant of those CPUs are going to be the same price as the 6-core 3.06GHz Westmere Xeons and not only will they have two additional cores and a larger cache, they are approximately 22% faster per core, per clock. That's nearly 50% faster when under load per core and turbo is kicked in. I can edit 4K and 5K in NLE's like Premiere and Vegas on some pretty lowly hardware via the lower-rez extractions right now, I've got a PC here that will all but do 4K full rez at 24fps in Premiere. It will handle 2 streams of 4K or 5K with editing rez set at 1/2 just fine. I bet in the next 30 to 60 days, I'll be able to build a new PC for much less money than the Rocket that can actually transcode faster than a Rocket, and doing so by using brute-force, hammer the CPUs approach to do it...
.

Jeff,
I can maybe get 6 frames per second transcoding a full decode/bayer when I script several command line versions simultaneous, in an attempt to saturate the cores of my SB-E running at 4ghz. I'm going from and to ramdisk for speed tests. RCX does a poor job of multithreading, cant get the speed the easy GUI way. When I run Premiere it got nowhere near realtime either. Are you seeing those speeds actually saving images ? Playing a timeline does not do a full decode, right ?

On decompressing, yep, I was able to play 4K RGB Cineform on an i7 a long time ago, as you said it is wavelet also.
But I understand the basic problem: Red is a small company, and these little fires are popping up all the time. It is no easy task to do all these things, and try to prioritize must be a chore as well.

 

[Bruce Allen]

Sorry if I'm ranting a bit here, but I've run into this a couple times this month already... "We can process ARRI RAW in real-time, but our systems choke on EPIC 5K... RED sucks...". Never mind that ARRIRAW frame is only 36% of the number of pixels and 36% of the total uncompressed bayer data, 36% of the total resolution or detail and has relatively no decompression overhead. 5K full on EPIC is 13.8MP in size... 16bit RAW... That's 26.4MB per FRAME for RAW and uncompressed 80MB per FRAME as 16bit RGB. That is 1.9GB/s for 24fps motion... Which leads us to one other concern with GPU acceleration. Most GPUs are notoriously slow for getting data off the GPU card and back onto the PCIe or system bus. On a PCIe 2.0 bus, you need the dedicated outbound capacity from your GPU of 4 lanes to accommodate that and to put that into perspective, most all current GPUs, even t

OK well... that's actually an argument in favor of GPU for me.

If you're grading EPIC at full resolution... you have to send the full decoded 5K RGB file to the GPU anyway. This seems like a horrible waste of bandwidth.

At the very least, do the wavelet decompression on the CPU... then send over the 5K RAW file (which is only one 16-bit value per pixel, instead of Red, Green and Blue) and do the de-Bayer (AKA inflation to massive RGB file) on the GPU?

And yes, the future could even be:
- Transfer over the compressed R3D
- de-wavelet on the GPU, de-Bayer on the GPU, do effects and grade on the GPU, then do high-quality GPU scale down to 1080p or 2160p or whatever, do H.264 compression on the GPU...
- Transfer back a nicely-compressed H.264 file to the CPU

I think that's where nVidia is pushing. Just a little ARM CPU connected to massive GPU power. Not sure that'll happen. But the alternatives (fast CPU connected to fast GPU via very high speed link, maybe even on the same chip) sound good too.

...Really. I think RED should jump on the OpenCL bandwagon here and offer it as a solution. I think many people will be surprised by what they find and not necessarily in a good way. Of course, it will all still be RED's fault. It sucks being pioneers in an industry at times. RED is no different than those who came before in the audio (or even video) industry, who decided to disrupt the incremental flow of technology. Sure, in another 2 to 5 years, we'll all be tossing around 4K video clips on common destkop PC's, even (dare I say it, on tablets), like they're no big deal. Until then, it is what it is... Just like real-time 16bit audio processing in the early '90s, it takes a dedicated accelerator card to do certain tasks now. This is rapidly changing. How many of you here bought MPEG-2 accelerator cards so we didn't have to wait for 3 days for our Pentium-II systems to encode a DVD? Come on, it's OK, y'all can admit it.

I can admit worse: I once convinced my high school to buy a video IO card with MJPEG acceleration that could handle a whopping 25fps at 384x288 (1/4 of PAL).

If you defragged the hard drive religiously it could even play back without dropping frames

Bruce Allen
www.boacinema.com

 

[Carter Cammack]

Kudos to Rob L for his intrest and input here.
I still don't get why people go to ridiculous pains to avoid or resist buying a Rocket card, and building a system around it. Total investment with software (as REDs software is FREE!) is 10-12 grand for a new tower with Rocket and raid. Don't forget the amazing deal you got on your Epic-X. Even a new one ordered today without the trade-in deal is about half of reasonable. Still not convinced? Go post some film.

It's the Scarlet camp that may need some workflow options. And Vegas is a pretty good one.

 

[jake blackstone]

I still don't get why people go to ridiculous pains to avoid or resist buying a Rocket card, and building a system around it. Total investment with software (as REDs software is FREE!) is 10-12 grand for a new tower with Rocket and raid. Don't forget the amazing deal you got on your Epic-X. Even a new one ordered today without the trade-in deal is about half of reasonable. Still not convinced? Go post some film.

Film world is much bigger, than just the Red. Not every post house is Red-centric, like Light Iron. Buying hardware, that does only one thing is the thing of the past. And not everyone purchased Red camera to take advantage of the RR discount. Post like yours clearly demonstrates just how much Red in desperate need of major attitude adjustment and good old fashion charm offensive in order being better accepted as a good citizen in the post world.
Yes, I'm not convinced to buy RR by a long shot and I love posting film.