SSD: more $$$ Confusion ????

[JohnF]

Sorry to continue off topic:

That would be true if the processor were the only component of the system, but at the data rates you would be working with, the time required to move that much data in and out of memory would likely be be more significant than the time savings from dealing with a lossless compression algorithm.

I disagree, In my experience with most VFX software uncompressed (UC) video speeds up rendering of complex sequences by a considerable degree.

There is a time penalty with compressed video for each layer of video in a sequence which can add up to quite some time.

For sequences with many layers I've found it faster to transcode the layers to UC and use those for the render. By doing this I have reduced reported render times by up to 50% for the really complex sequences (ie a many tens of layers sequence from a 24hour render to just 12hours).

This not only allows for a faster assessment of the sequence but saves money in man-hours, equipment use and electricity.

This is no criticism of REDCODE just a workflow observation of complex VFX sequences.

JohnF

 

[David Battistella]

True.

Uncompressed is faster than compressed because there is no compression step, therefor no codec, compresseion and decoding going on to slow CPU, etc.

Just look at FCP and RT extreme, UC timelines allow for much more RT performance.

But BRINGING this back to REDCODE.

REDCODE is ALWAYS compressed . It is BUILT on a WAVELET COMPRESSION SCHEME which at 2.5:1 IS MATHEMATICALLY (and VISUALLY0 LOSSLESS.


David



David

 

[Martin Weiss]

Manufacturers specs don't match real performance so we had to pay a premium for guaranteed performance. Deanan has been pulling the remaining hairs out of his head working on this. It amazes all of us how loose the memory guys are with specs.

 

[Tim Whitcomb]

Any chance on getting an answer to this?

Both Jim and Jarred said "soon". I'm sure all this will come together in a new announcement as they get their production ducks in a row (which will determine their final costs). We all know RED will pass whatever savings they can to us.

 

[Rob Schmitz]

Not even Angry Birds is helping...

Jim

Angry Birds FTW!

 

[Gavin Greenwalt]

VFX people have said that they have never had better footage to work with than the RED ONE M-X (at 7.5:1). Why would they need 3:1? Just curious... I can see 5:1 no problem.

Jim

Because there is "best footage" and "bester" footage.

 

[D Fuller]

Sorry to continue off topic:



I disagree, In my experience with most VFX software uncompressed (UC) video speeds up rendering of complex sequences by a considerable degree.

There is a time penalty with compressed video for each layer of video in a sequence which can add up to quite some time.

For sequences with many layers I've found it faster to transcode the layers to UC and use those for the render. By doing this I have reduced reported render times by up to 50% for the really complex sequences (ie a many tens of layers sequence from a 24hour render to just 12hours).

This not only allows for a faster assessment of the sequence but saves money in man-hours, equipment use and electricity.

This is no criticism of REDCODE just a workflow observation of complex VFX sequences.

JohnF

I see what you mean. My experience is with much-less complicated sequences where data retreival speeds are still significant.

In your complex sequences, does it matter how great the compression ratio is or is just a matter of what codec you're unpacking? Clearly h264 will take linger than Redcode, but will RC2.5 be faster than RC10? (I know this is off-topic, and I apologize for continuing, but I'm curious about this.)

 

[Michael McVicar]

Currently, with the SSDs you can shoot 5K @ 24fps at REDCODE 3:1... although I shoot 5:1. 5K @ 24fps HDRx™™ is 6:1 (makes sense... twice 3:1).

Jim

So is the speed of the media the current data bottleneck? Meaning with the Epics ASIC will we be able to handle faster frame rates (with HDRx™) at the same time, same compression, as faster media becomes available?

 

[JohnF]

Hi David,

It varies from from codec to codec. So yes, it is just a matter of which codec is being unpacked. Though I've not measured the time taken against the compression ratio.

Long G.O.P codecs seem to be the worse offenders.

Re: Data retrieval speeds(and heading sort of on-topic) there has been some interesting developments with PCIe SSD Raids. They appear to be heading into a more realistic price range with capacities suitable for film makers and with quite amazing speed potential.

Read: this link from anandtech.com about OCZ's IBIS and HSDL interface

This piece of hardware combined with RED's SSD means a production could conceivably go solid state for both production and post with an added (serious) speed increase to boot.
IMO a very interesting development.

All the best

JohnF

 

[Brian Ferguson]

Call me crazy but I want to capture in the least compression possible. VFX or not I want uncompressed! I guess as years went by and watching previous projects shot in the 80's and 90's on different formats would of been amazing to see what it really looked like in 5K. I know in 20 years from now it will be nothing compared to 30K but still some serious quality. I'm a quality whore, sorry.

If you ever saw firsthand the early days of RED development, "Frankie", you would never - ever say you want uncompressed. First time I saw REDCODE I was flabbergasted. It was the secret sauce that made 4K possible for the many. Uncompressed RAW was a ton of data.

Bandwidth and drive speed is better than it used to be in PC's but not that much better than back then. I am calling you crazy.

Edit:
Ok I thought about it for a while, I understand the "uncompressed is faster in post" point of view. Yes uncompressed SD/HD-SDI was faster than compressed in post because of no codec overhead. I saw that myself firsthand...

But from a practical aspect on the set - at 4k RAW (if memory serves me) uncompressed 4K is right around 1GB a second at 24 fps. Jim said in this thread that 5K is 60% more. So 1.6GB a second? So if you have a 250GB SSD drive you get about 156 seconds? A couple of minutes + on the set, 10+ minutes on a Terabyte? And to play this in real time for viewing? I don't know.

On the original 8GB CF on the REDONE you could get 4-5 minutes.
I am just talking as a DP here. Who wants to do that many reloads no matter what VFX wants? I am assuming actors are in these VFX scenes. It is more at play than just pure visual aesthetics.

 

[Gavin Greenwalt]

Speaking from experience: REDCode without REDRocket is a bad idea in comps. If you have 4 or 5 REDCode streams trying to debayer all at once your CPU will be pegged. It's best to output to half float EXR and pull it off of a 200MBs raid.

Most modern compositing applications use windowing for their reads anyway so if you have a 6k plate but you have a defined working region it'll only read off the 1k x 1k square you're currently working on. With REDCode it has to decode tiles of data first which also screws up the multi-threading architecture of the compositing app.

For example in Nuke if you are zoomed in to 10 pixels it'll only read those 10 scanlines from the files (where possible depending on file format).

In theory I'm all for the "CPUs are evolving faster than HDDs". But in practice I've resorted to just rendering off uncompressed to a RAID for workability. It'll get there. Eventually. But for now HDDs are faster in part because as soon as pixels get into the software the CPU is needed to start processing the RGBA channels. 200MBs of pixels can effectively keep an i7 plenty busy. If you add in debayer and decoding then you're using the CPU that could be working on RGB data. So there is a delay before the computer can even start working on the composite. While it's crunching on the composite the RAID can be caching the next scanline into RAM. Yes it's slower maybe than debayer redcode but it's time that the HDD would otherwise be idle.

If REDCode is 3GBs and a cheap RAID can deliver 300MBs then as long as your comp is taking more than half a second per frame to render your bottleneck is going to be in the render not the file reads and it makes sense to use uncompressed. Even if you had a 3GBs RAID of SSDs the CPU would just be sitting there churning on the last frame anyway.

Obviously editing is a whole'nother ball game since there is minimal processing and just getting the RGB data as quickly as possible is the goal no matter the CPU cost. You also need hours of footage where as a composite might need a few minutes at most.

Then again as mentioned before that assumes there was an uncompressed-debayered option. Which there isn't and wouldn't be so mathematically uncompressed or Redcode 5:1 would both be the same speed wise.

 

[Steve Dent]

Speaking from experience: REDCode without REDRocket is a bad idea in comps. If you have 4 or 5 REDCode streams trying to debayer all at once your CPU will be pegged. It's best to output to half float EXR and pull it off of a 200MBs raid.

Most modern compositing applications use windowing for their reads anyway so if you have a 6k plate but you have a defined working region it'll only read off the 1k x 1k square you're currently working on. With REDCode it has to decode tiles of data first which also screws up the multi-threading architecture of the compositing app.


.

That's why I think the first full 4K release with 4k workflow will NOT be an effects heavy movie. The processing involved at the moment is just too penal, from transcoding to comping to (especially) SGI.

I used to do demos (in SD 10 years ago), with uncompressed footage and 5:1 MJPEG, distressed 5 generations. Almost no one could tell the difference (all professional editors).

 

[Michael Olsen]

I'm pretty sure I just missed this somewhere, but I wanted to confirm.

The side SSD module (which takes the place of the CF module) for Epic is $2500. The regular module is $1500.

Couple of questions:

1)Why is it more expensive? I thought the "side" version was sort of the economy choice?

2)Is this pricing the same for Scarlet? I assume so.

 

[Michael Hastings]

I'm pretty sure I just missed this somewhere, but I wanted to confirm.

The side SSD module (which takes the place of the CF module) for Epic is $2500. The regular module is $1500.

Couple of questions:

1)Why is it more expensive? I thought the "side" version was sort of the economy choice?

2)Is this pricing the same for Scarlet? I assume so.

Michael:

Everything has changed - there is no side CF module anymore or new larger CF cards - SSD will be standard - may be no change in price for side ssd over side CF but not determined yet. A CF to SSD adapter will be available to allow use of legacy CF cards.

Subject to change of course.

 

[fALk Gärtner]

VFX people have said that they have never had better footage to work with than the RED ONE M-X (at 7.5:1). Why would they need 3:1? Just curious... I can see 5:1 no problem.

Jim

Yes especially with the low noise 5:1 is equal to a good filmscan (at least if not even miles better - no film jitter etc etc) as stated most post is done at 2k (that virtually lossless with 5:1 (if 2.5:1 is lossless)) - the few IMAX films that have a 4k post have enough excellent manpower to handle "unclean edges" coming from a 5:1 compression or will have no problem getting a custom rig from red to record at 2,5:1 if they soooo desire. Mostly though the actual physical greenscreen on set is worse then the compression problems anyways (as long as you don´t shoot with h.264 or HDV or other crap like that and green/bluescreening is the only reason you need low compression rates anyway.
For most film & vfx people in germany 1080p uncompressed coming out of alexa is already heaven - go figure.

 

[Gunleik Groven]

Because the logC output is cineon compliant, like redlogFilm?

Just finnished first VFX tests with RLF EXR output.
Made a difference...