Welcome to our community

Be a part of something great, join today!

Log in
  • Hey all, just changed over the backend after 15 years I figured time to give it a bit of an update, its probably gonna be a bit weird for most of you and i am sure there is a few bugs to work out but it should kinda work the same as before... hopefully :)

Weapon compression - WOW!

Whose name is on those patent documents? Do you not think I know what I'm talking about here? Original Red One did 12bit R3D, but that's how many years ago now and things have changed since then.

Graeme
 
Graeme Nattress said:
Whose name is on those patent documents? Do you not think I know what I'm talking about here? Original Red One did 12bit R3D, but that's how many years ago now and things have changed since then.
Graeme
Click to expand...

I know you are the man behind the curtain, I do not understand why files do not match what you are writing.

The specification for Dragon/Weapon lists PSNR as 80db, (80-1.76)/6.02 = 13bits of precision.
Is the PSNR listed wrong, or do you calculate it in some other way?

If an R3D file contained 16bits of data, at Dragon's min compression ration of 4:1, the data rate would be 233 MB/sec which is well above Dragon's specified limit of 200 MB/sec. Actual data rates of recorded files average 175 MB/sec
For an individual frame 6K 4:1 the file size is 8.01MB, that is too small to hold 14bits of data
For an individual frame 6k 5:1 the file size is 6.5MB, that is too small to hold 14bits of data.

Unless you are discarding pixels and interpolating?
That would fit 16bits in the file size, and that would explain the color aliasing on high contrast black/white subjects.
I like to understand the tools I'm using, blind faith in technology is never a good idea


HVPixelsBit DepthSize (bits)Size (MB)CompressionPCM/MetaR3D Frame Size
R3D Actual File Size
R3D Actual File Size

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]12[/TD]
[TD="align: right"]232980480[/TD]
[TD="align: right"]29.03545364[/TD]
[TD="align: right"]4[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]7.28286341[/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]13[/TD]
[TD="align: right"]252395520[/TD]
[TD="align: right"]31.45507478[/TD]
[TD="align: right"]4[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"] 7.887768694 [/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]14[/TD]
[TD="align: right"]271810560[/TD]
[TD="align: right"]33.87469591[/TD]
[TD="align: right"]4[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]8.492673978[/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]15[/TD]
[TD="align: right"]291225600[/TD]
[TD="align: right"]36.29431705[/TD]
[TD="align: right"]4[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]9.097579262[/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]16[/TD]
[TD="align: right"]310640640[/TD]
[TD="align: right"]38.71393819[/TD]
[TD="align: right"]4[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]9.702484546[/TD]

[TD="align: right"] 8.01 [/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]12[/TD]
[TD="align: right"]232980480[/TD]
[TD="align: right"]29.03545364[/TD]
[TD="align: right"]5[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]5.831090728[/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]13[/TD]
[TD="align: right"]252395520[/TD]
[TD="align: right"]31.45507478[/TD]
[TD="align: right"]5[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"] 6.315014955 [/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]14[/TD]
[TD="align: right"]271810560[/TD]
[TD="align: right"]33.87469591[/TD]
[TD="align: right"]5[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]6.798939182[/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]15[/TD]
[TD="align: right"]291225600[/TD]
[TD="align: right"]36.29431705[/TD]
[TD="align: right"]5[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]7.28286341[/TD]

[TD="align: right"]6144[/TD]
[TD="align: right"]3160[/TD]
[TD="align: right"]19415040[/TD]
[TD="align: right"]16[/TD]
[TD="align: right"]310640640[/TD]
[TD="align: right"]38.71393819[/TD]
[TD="align: right"]5[/TD]
[TD="align: right"]0.024[/TD]
[TD="align: right"]7.766787637[/TD]

[TD="align: right"] 6.5 [/TD]
 
:emote_popcorn: Include all math in each rebuttal so I can follow along.
 
While I don't know how an R3D is formatted, I think the way that an EXR half float (16-bit) was designed was smart. 1 sign bit, 5 exponent bits (log), and 10 mantissa (linear) bits. EXRs 16-bit format supports 30 stops of dynamic range, which is WAY more that any camera can capture for the foreseeable future.

A pure linear file is kind of wasteful.
 
Christopher : It would not surprise me if a modern A2D used multiple passes for each frame - in which case the number of reads (log2) would add to the effective bit depth.

Media speed : A technology around Intel & Micron 3D Xpoint maybe the way forward. Probably enough for 1000fps x 8k ...

AJ
 
Doesn't look like Graeme will explain the mystery of how 16 bits of data fits in a 12bit file size.

Graeme says R3ds from Dragon forward are 16bit. Red says Dragon is 16bit.
Gavin and Stacey did a very nice test demonstrating 16 stops or so of engineering range, which would need 16bits.

16bits can not fit in the space of the 12bit file sizes that are being recorded, and it would exceed Dragon/Weapon bandwidth specs, unless...

What if your primary goal was dynamic range, and were willing to throw out some color and/or resolution to achieve that.

If I post here the math of what I think they are doing, I'm pretty sure it will just be deleted.

I would suggest anyone interested just read Graeme's video processing patent updated 4/28/2015
(Hint search for Green or Delete)

Again, not trying to kick the hornet's nest just trying to understand the tools I use so I can work around their problems. I'm pretty sure now the chroma aliasing problem that is very annoying for the projects I work on is very related to all of this, has it been addressed in weapon? Specs do not indicate any change.

WEAPON 6K (CARBON FIBER)
DYNAMIC RANGE 16.5+ stops S/N RATIO 80 dB
EPIC DRAGON
DYNAMIC RANGE 16.5+ stops S/N RATIO 80 dB


With enough public complaining Sony is finally adding uncompressed raw as an option to their craw/compressed raw on their DSLR cameras, so maybe Graeme can fix the chroma aliasing with enough public awareness too.
 
Phil Holland said:
RED does a few things very "right" with MINI-MAGs. They have a proprietary method of writing to the media with specific firmware and hardware backing it up. On top of that there's a bit of headroom left it appears to allow for safe writes. And even on top of that they do some insane stress testing. I mean those things toast up like Hot Pockets and they survive happily.

This is a different method than standard CFast writes and if you've seen the Ursa "dubstep red screen" error that you get occasionally while using CFast cards you know where that bus stops with actual production work. I know a few Amira shooters with several CFast cards that have failed for some reason or another, likely due to the media, but who the hell knows. Not saying I don't trust them, just saying it's a different brand of reliability.

Nothing is bullet proof when it comes to media and media bays, but when the system is clean and running right life should be good. I've seen all sorts of nasty things though, like "hey my DIT spilled his Coke on my MINI-MAG" and the "hey we left the camera on the sand at lunch with no mag or tape blocking the media bay". I face palm sometimes when I get those phone calls at 3am asking why something has gone "weird". Nearly always some sort of oversight to external contamination it seems.

Much of our job deals with electronics and optics. As much as we can ride them rough, we need to treat them like a special lady and never forget that if possible.
Click to expand...

Amen to the bus stopping there, Let's not forget the days before redmags and red undead, where all footage that was lost due to a reading or writing "error" was just gone 9 times out of 10. Forget that man, i'll keep my toasty little hot pockets any day for the extra $$. (now i'm hungry)
 
Christopher*Seguine said:
Doesn't look like Graeme will explain the mystery of how 16 bits of data fits in a 12bit file size.

Graeme says R3ds from Dragon forward are 16bit. Red says Dragon is 16bit.
Gavin and Stacey did a very nice test demonstrating 16 stops or so of engineering range, which would need 16bits.

16bits can not fit in the space of the 12bit file sizes that are being recorded, and it would exceed Dragon/Weapon bandwidth specs, unless...

What if your primary goal was dynamic range, and were willing to throw out some color and/or resolution to achieve that.

If I post here the math of what I think they are doing, I'm pretty sure it will just be deleted.

I would suggest anyone interested just read Graeme's video processing patent updated 4/28/2015
(Hint search for Green or Delete)

Again, not trying to kick the hornet's nest just trying to understand the tools I use so I can work around their problems. I'm pretty sure now the chroma aliasing problem that is very annoying for the projects I work on is very related to all of this, has it been addressed in weapon? Specs do not indicate any change.

WEAPON 6K (CARBON FIBER)
DYNAMIC RANGE 16.5+ stops S/N RATIO 80 dB
EPIC DRAGON
DYNAMIC RANGE 16.5+ stops S/N RATIO 80 dB


With enough public complaining Sony is finally adding uncompressed raw as an option to their craw/compressed raw on their DSLR cameras, so maybe Graeme can fix the chroma aliasing with enough public awareness too.
Click to expand...

what do you think they are doing can you explain that little bit?
 
egon touste said:
then they are ignorant. it makes little difference, except to offload times , and storage costs .
Click to expand...

When you're on a multi million dollar project you don't want to risk screwing up a shot due to compression artifacts. VFX
You're right on low budget & youtube stuff. No need for anything more than 8:1.
 
For my YouTube videos I still shoot as low compression as I can, it helps really bring out the fur in my cat videos!
 
egon touste said:
then they are ignorant. it makes little difference, except to offload times , and storage costs .
Click to expand...
'

No. Not exactly. REDCODE RAW compression ratios also effect final image quality and "workflow" time. Meaning transcodes and renders. The less compression, the longer everything takes.

I'm more focused on quality for the work I do typically and shooting 6K 3:1 thus far on Weapon has been very nice.

For me as long as labor and time expenses are cheaper than shooting film it works out for me.


egon touste said:
dear red..do let me know when i might expect a meaningful sensor upgrade for my dragon ..
Click to expand...

Sounds like you have have a year or two before that happens according to a recent Jarred post. This is a tick-tock cycle like he's mentioned before. DSMC2 style bodies seem to be on the "tick" side. Likely the next "tock" cycle is a new sensor.

That said, there are visual improvements to the images Dragon can create via Weapon. CMOS smearing is vastly minimized. The Optical Path is much more streamlined. There's some other things going on as well that help produce cleaner images at all ISOs due to some of that. And yes, the lower REDCODE RAW compression ratios produce cleaner "fuller" images.
 
Phil Holland said:
No. Not exactly. REDCODE RAW compression ratios also effect final image quality and "workflow" timeI'm more focused on quality for the work I do typically and shooting 6K 3:1 thus far on Weapon has been very nice..
Click to expand...

I would have to agree with Phil, Dragon 6k 4:1 is visibly significantly better than 8:1 - when watched in a theatrical 4k environment. J2k compression softens the image at higher compression, it doesn't alias and stand out and look bad - but watch back to back, you can see how details mush.

Currently shooting less than say 7:1 was a big problem for narrative work on Dragon- the fans would just be too loud for a normal closed set, less compression = more heat = too loud a camera.

Hopefully that was addressed with the new fan design in Weapon.



Kemalettin Sert said:
what do you think they are doing can you explain that little bit?
Click to expand...

Its all in there, Graeme explains it better than I can

"the camera 10 can be configured to delete or omit some of the green image data. For example, in some embodiments, the image processing module 20 can be configured to delete 1/2 of the green image data so that the total amount of green image data is the same as the amounts of blue and red image data"


As you know, with a Bayer pattern sensor you have twice as many green pixels as Red or Blue, so if you just throw out half the green pixels, it makes it much easier/cheaper to compress, and you can then fit 16bits in the space of the 12bit file sizes we have with Dragon.

Obviously you just threw out a bunch of resolution in favor of dynamic range - maybe you can do some tricks to help you interpolate that resolution back, but then you have problems like chroma aliasing like we see with Dragon.

Decreased compression on Weapon is great, but it does not seem like the pre-filtering done before compression has changed based on the data rate limitations and compression ratios.
 
Please don't quote me out of context. We don't "throw out half the green pixels".

"like chroma aliasing like we see with Dragon" - if you look to the thread where that phrase emerged you'll find it was an erroneous misattribution. The issue was axial chromatic aberration from the lens (as best as we can tell).

Graeme
 
Christopher*Seguine said:
As you know, with a Bayer pattern sensor you have twice as many green pixels as Red or Blue, so if you just throw out half the green pixels, it makes it much easier/cheaper to compress, and you can then fit 16bits in the space of the 12bit file sizes we have with Dragon.

Obviously you just threw out a bunch of resolution in favor of dynamic range - maybe you can do some tricks to help you interpolate that resolution back, but then you have problems like chroma aliasing like we see with Dragon.

Decreased compression on Weapon is great, but it does not seem like the pre-filtering done before compression has changed based on the data rate limitations and compression ratios.
Click to expand...

For a number of reasons you just lost my attention here...
 
Graeme Nattress said:
Please don't quote me out of context. We don't "throw out half the green pixels".

"like chroma aliasing like we see with Dragon" - if you look to the thread where that phrase emerged you'll find it was an erroneous misattribution. The issue was axial chromatic aberration from the lens (as best as we can tell).

Graeme
Click to expand...

I did see this post where you conclude it is chromatic aberration from the lens.

I was waiting to try again with weapon and also with one of kippertie's zero pass olpf to see if there is any difference - but comparing .cr2 from the 5dsr and 6k .4:1 r3d from a dragon, same otus 55mm lens, same 5.6 exposure, both critically focused - chroma ghosting/halo(presence of color where there is none) on black/white high contrast object in the r3d, none in the canon - no lens corrections or goose fat applied. maybe not caused by compression or demosaic, maybe by pre-filtering.

When I film black and white objects in a color scene - graphite drawings in a wood paneled room for example - I would like black and white - to be black and white, not back/white with color halo. I know, maybe not a common event in hollywood, but quite common in the museum world.

If your saying the r3ds are 16bit and the compression ratio is accurate, the file size is a known quantity, you cant fit the square peg in the round hole without shaving off some of the edges.

Remember the early days when you could just open up an R3d in matlab and see exactly what was going on, you can still do that with everyone else - arriraw, sonyraw, cr2, nef.

Given there is some trickery going on to make 16bit fit in the space of 12, maybe you can build on the trickery to predict where the chroma problem is likely to occur and compensate.
 
You must log in or register to reply here.
Back
Top