SSD run times

[Daniel Browning]

I just wish that the 10 to 1 compression option was the Mathematically Lossless option,

Mathematically Lossless doesn't mean better.

There are some types of compression that lose absolutely no information whatsoever, and no artifacts are visible even under the most extreme once-in-a-lifetime processing you can imagine. Yet they are not mathematically lossless. I'll give you an example.

Let's say you're trying to measure how long it takes for a ball from a "perfect" throwing machine to go 100 feet. It always throws the exact same way at the same speed (2.0000000 seconds), but we'll say we don't know that so we have to measure it. Your stop watch precision is down to one ten-millionth of a second. So you start recording numbers:

1.95435435
2.05223848
2.10438524
1.97448923
2.02385385

But is the data really that *accurate*? Were your eyes and muscles so finely tuned that you knew to push the button to within one 10 millionth of a second? One way to "compress" that data is to simply throw away the wasted precision:

1.95
2.05
2.10
1.97
2.02

Now there is much less data, but it's still just as "accurate". All you've done is throw away wasted precision.

Same thing with image sensor compression. Let's say you're shooting a colorchecker test chart. We know from the manufacturer the color of a certain patch to such an exact extent that on the certain 14-bit sensor we're testing it should render a green channel raw value of exactly 8192.

But when we actually measure it with the camera, we get all sorts of raw values from the pixels inside the colorchecker patch:

8282
8102
8237
8147

Why are they jumping around so much? Because light itself is random. A phenomenon called photon shot noise means that the *mean* raw value will be 8192, but the actual raw values are quite random.

The values jump around so much that 14 bits is far more precision than necessary for this tonal level (1 stop below clipping). As long as the random jumps are greater than the quantization step, you can throw away precision without having lost *any* information whatsoever. It's exactly like the stopwatch analogy above: your eyes and body are introducing so much randomness to the measurements that recording 10 millionths of a second is not really giving you any additional accuracy -- just wasted precision. In the same way, light itself adds randomness to the measurement, so wasting 14 bits on that tonal level is just wasted precision.

One of the simplest and fastest compression methods is a bit depth LUT (Look Up Table). For low tonal values, you need all 14 bits to record the entire noise floor. For high tonal values, you only need 7, 8, or 9 bits. The LUT only needs to be tuned once for each sensor design (so that the LUT curve matches the SNR curve for the photon shot noise limited part of the dynamic range). This is exactly like changing "1.99423953" to "1.99".

So just because something is not mathematically lossless doesn't mean it's any worse. Sometimes it's just dumping the wasted precision.

 

[Elsie N]

Mathematically Lossless doesn't mean better.

There are some types of compression that lose absolutely no information whatsoever, and no artifacts are visible even under the most extreme once-in-a-lifetime processing you can imagine. Yet they are not mathematically lossless. I'll give you an example.

Let's say you're trying to measure how long it takes for a ball from a "perfect" throwing machine to go 100 feet. It always throws the exact same way at the same speed (2.0000000 seconds), but we'll say we don't know that so we have to measure it. Your stop watch precision is down to one ten-millionth of a second. So you start recording numbers:

1.95435435
2.05223848
2.10438524
1.97448923
2.02385385

But is the data really that *accurate*? Were your eyes and muscles so finely tuned that you knew to push the button to within one 10 millionth of a second? One way to "compress" that data is to simply throw away the wasted precision:

1.95
2.05
2.10
1.97
2.02

Now there is much less data, but it's still just as "accurate". All you've done is throw away wasted precision.

Same thing with image sensor compression. Let's say you're shooting a colorchecker test chart. We know from the manufacturer the color of a certain patch to such an exact extent that on the certain 14-bit sensor we're testing it should render a green channel raw value of exactly 8192.

But when we actually measure it with the camera, we get all sorts of raw values from the pixels inside the colorchecker patch:

8282
8102
8237
8147

Why are they jumping around so much? Because light itself is random. A phenomenon called photon shot noise means that the *mean* raw value will be 8192, but the actual raw values are quite random.

The values jump around so much that 14 bits is far more precision than necessary for this tonal level (1 stop below clipping). As long as the random jumps are greater than the quantization step, you can throw away precision without having lost *any* information whatsoever. It's exactly like the stopwatch analogy above: your eyes and body are introducing so much randomness to the measurements that recording 10 millionths of a second is not really giving you any additional accuracy -- just wasted precision. In the same way, light itself adds randomness to the measurement, so wasting 14 bits on that tonal level is just wasted precision.

One of the simplest and fastest compression methods is a bit depth LUT (Look Up Table). For low tonal values, you need all 14 bits to record the entire noise floor. For high tonal values, you only need 7, 8, or 9 bits. The LUT only needs to be tuned once for each sensor design (so that the LUT curve matches the SNR curve for the photon shot noise limited part of the dynamic range). This is exactly like changing "1.99423953" to "1.99".

So just because something is not mathematically lossless doesn't mean it's any worse. Sometimes it's just dumping the wasted precision.

Works for Me!:emote_popcorn:

 

[Martin Stevens]

Mathematically Lossless doesn't mean better.

Same thing with image sensor compression. Let's say you're shooting a colorchecker test chart. We know from the manufacturer the color of a certain patch to such an exact extent that on the certain 14-bit sensor we're testing it should render a green channel raw value of exactly 8192.

But when we actually measure it with the camera, we get all sorts of raw values from the pixels inside the colorchecker patch:

8282
8102
8237
8147

Why are they jumping around so much? Because light itself is random. A phenomenon called photon shot noise means that the *mean* raw value will be 8192, but the actual raw values are quite random.

The values jump around so much that 14 bits is far more precision than necessary for this tonal level (1 stop below clipping). As long as the random jumps are greater than the quantization step, you can throw away precision without having lost *any* information whatsoever. It's exactly like the stopwatch analogy above: your eyes and body are introducing so much randomness to the measurements that recording 10 millionths of a second is not really giving you any additional accuracy -- just wasted precision. In the same way, light itself adds randomness to the measurement, so wasting 14 bits on that tonal level is just wasted precision.

One of the simplest and fastest compression methods is a bit depth LUT (Look Up Table). For low tonal values, you need all 14 bits to record the entire noise floor. For high tonal values, you only need 7, 8, or 9 bits. The LUT only needs to be tuned once for each sensor design (so that the LUT curve matches the SNR curve for the photon shot noise limited part of the dynamic range). This is exactly like changing "1.99423953" to "1.99".

So just because something is not mathematically lossless doesn't mean it's any worse. Sometimes it's just dumping the wasted precision.

My desire to have a Mathematically Lossless compression setting on the Epic
is not an argument about capturing imagery with a zero loss in mathematical precision.

I'm not, as you are, trying to compare the real world with the finished file.
Instead I'm trying to compare two files, with one being Uncompressed,
and the other being Compressed but at the same time Mathematically Lossless.

To illustrate my point..........hypothetically take an original uncompressed TIF file through
a million cycles of compression/decompression. If the compression algorithm is
truly Mathematically Lossless then the final TIF will be the same as the original TIF.
No change....period.

However, all of this is to be taken as mere intellectual fun.

I will use my Epics at 5 to 1 etc., with no worries.

Also, sorry if I have made the assumption that you were referring to my
previous posts regarding this topic.

 

[Daniel Browning]

I'm not, as you are, trying to compare the real world with the finished file.

No, that's not what I'm doing. I'm comparing the wasteful precision file with the normal-precision file. Here's another way to think of it.

Let's say I design a 12-bit camera with 12-bit mathematically lossless files. But I want to market it as a "16-bit" camera, so I add four bits of completely random data and store 16-bit mathematically lossless files.

If you take those 16-bit files and round them down to 12-bits, you are not using mathematically lossless compression. But you are not throwing away anything useful or informational either. You can do all the most extreme comparisons you can think of, and you still wont find the mathematically lossless version any better than the mathematically lossy version. Because while it's mathematically lossy, it's not "information" lossy (I don't know a better term).

The only difference between that and what I'm talking about is that the "completely random data" comes from light itself.

 

[Martin Stevens]

So Daniel......do you wish for a Mathematically Lossless compression setting on the Epic?

I can see no reason not to have this option.
You can always just not use it.

 

[Jannard]

So Daniel......do you wish for a Mathematically Lossless compression setting on the Epic?

I can see no reason not to have this option.
You can always just not use it.

Martin... just because you are not letting this go (and we will have a 2.5:1 setting with about 18fps to SSD at some point), I have to ask you the question:

If you test for yourself 2.5:1 vs. 5:1 and cannot tell the difference no matter how close you look, will you still shoot 2.5:1 just because it has less compression?

Jim

 

[paulherrin]

with numbers guys, who usually tend to be into mathematics, science, tech, etc... there can sometimes be an ideal pursuit of the more pure data. sometimes that really helps out... other times it's more like an unhealthy obsession with numbers.

for example, in my work with computer graphics, i tend to use numbers that make sense... for example, the Kelvin number in RCX... i will not allow myself to dial in random numbers with the sliders... that things not going to be say, 6846... it will be 6845, or more likely 6850 or 6800/6900. You'll never see me let some random 10th decimal place values go through. I typically use the number pad instead of sliders. I typically use numbers like... 8, 16, 32, 64, 256, 512, 1024, 2048, 4096, etc. (powers of two) when it makes sense to do so, and especially for intense computations.

kind of silly. partly it's academic, and part of it is maintaining a logical order and just plain not liking to have wild entries, the rest of it is knowing that those little tiny things will add up to very subtle increases in computation speeds/quality.

mostly it's just the idea of the thing. and for most of us these days, the ideal has been to acquire uncompressed 4:4:4 RGB. this isn't that, and we're beginning to understand how much better it is. up until this point, compression is viewed primarily as a very negative thing. so like i said earlier, let's please continue to be patient with one another. we're moving into a new realm here.

just a sweet draught of perspective.

 

[Martin Stevens]

Martin... just because you are not letting this go (and we will have a 2.5:1 setting with about 18fps to SSD at some point), I have to ask you the question:

If you test for yourself 2.5:1 vs. 5:1 and cannot tell the difference no matter how close you look, will you still shoot 2.5:1 just because it has less compression?

Jim

I will use, and advocate the use of, 5 to 1.

However, I know that having the 2.5 to 1 option will help sell and convert over
die-hard film emulsion fans.

 

[paulherrin]

that is a good point, martin.

you can't count on the naysayers doing actual research, however simple... just another thing for people to use as a weapon.

however, the other route is to get blown away by the image. at that point, it's all visceral, and you just know it's right. it will trump the numbers every time.

i suggest open 4k screenings at multiple strategic venues around the globe... above reproach.

also, perhaps a more public (broader than just reduser, i.e. on the red home site for the naysayres) r3d's/rcX release. or a really cool image comparison app on red.com, where you can virtually grade an image...

"if it looks good, it is good"

 

[Jannard]

I will use, and advocate the use of, 5 to 1.

However, I know that having the 2.5 to 1 option will help sell and convert over
die-hard film emulsion fans.

So the fact that John Schwartzman was converted from film to EPIC recording to 5:1 doesn't count? John had never shot anything but film until the Green Hornet pick-ups with a RED ONE M-X and then Spiderman with EPIC. The Green Hornet was shot REDCODE 42 (7:1) and Spiderman at 5:1.

Jim

 

[Steve Sherrick]

My guess is that John used his eyes. And I would also guess that they took some files through post and VFX to see if they could break it. And the fact that they are shooting at the compression ratios indicated means they eventually found their comfort zone. Seems like 5:1 is going to be that comfort zone for a lot of films like this. For everyone else, 7:1 or 5:1).

 

[Deanan]

I will use, and advocate the use of, 5 to 1.

However, I know that having the 2.5 to 1 option will help sell and convert over
die-hard film emulsion fans.

That's funny because emulsion is compression also

 

[Gunleik Groven]

That's funny because emulsion is compression also

as is air, Vinyl and tape in the analogue audio world...

 

[I Bloom]

Martin... just because you are not letting this go (and we will have a 2.5:1 setting with about 18fps to SSD at some point), I have to ask you the question:

If you test for yourself 2.5:1 vs. 5:1 and cannot tell the difference no matter how close you look, will you still shoot 2.5:1 just because it has less compression?

Jim

Following this thread closely.

I kind of agree with both sides. I agree with Jim that we often overreact to compression based on principle rather than experience. I also agree with Martin that there's no reason not to have options that the camera is physically capable of. I can also see Jim would like to prevent customers from needlessly fucking themselves with low compression workflow and then turning that bad experience into FUD.

Imagine being the DIT on a set with limited resources where the DP insists on as little compression as possible without regard to the effects on storage and turnaround etc. I think this will be the rule rather than the exception.

All that being said my hope is that the Epic-X has no limitations beyond what is physically possible. I would hope to see 4K 2.5:1 and 2K 1:1 among the options as well as 5k 1:1 at low frame rates. I think to not allowing this means making assumptions about the purpose for which the camera will be used.

Regards,
IBloom

 

[Terry VerHaar]

Sometimes I think it is only smart for a company to "limit" its product's use when it is clear that the brand will be damaged by allowing people to misuse it. As several people have pointed out, when the producer ignorantly insists on uncompressed (or something equally unnecessary), just because he/she can, but without regard for or complete knowledge of the consequences, that producer will "blame" the shortcomings of the RED camera rather than understanding that they caused the issues.

In this case, when huge files lead to workflow slowdowns or low compression leads to other limitations, who get's dinged? As it is, I don't really think RED is limiting anything, realistically. The only issue is the mindset of the individual who demands something that doesn't really make any sense. To paraphrase Jim, maybe RED does need to protect people from themselves - or from the demanding, but ignorant, producer.

 

[Martin Stevens]

So the fact that John Schwartzman was converted from film to EPIC recording to 5:1 doesn't count? John had never shot anything but film until the Green Hornet pick-ups with a RED ONE M-X and then Spiderman with EPIC. The Green Hornet was shot REDCODE 42 (7:1) and Spiderman at 5:1.

Jim

YES, it does count. It counts so much so it is profound!
In fact, it is so impressive that I have been spreading the good word.

Your position really doesn't need defending. Your vision is sublime,
and your products are empirically the best of the best.

However, what was almost indefensible was the George Lucas/Sony camps
statements that 1080p was "indistinguishable from film".

Their 1080p is more like a poor quality release print at best, and not EPIC!



Oh, and by die-hard emulsion fans, I meant people like Spielberg etc.

 

[Steve Sherrick]

YES, it does count. It counts so much so it is profound!
In fact, it is so impressive that I have been spreading the good word.

Your position really doesn't need defending. Your vision is sublime,
and your products are empirically the best of the best.

However, what was almost indefensible was the George Lucas/Sony camps
statements that 1080p was "indistinguishable from film".

Their 1080p is more like a poor quality release print at best, and not EPIC!



Oh, and by die-hard emulsion fans, I meant Spielberg etc.

Spielberg's reasoning for sticking with film is a bit deeper than compression. Steven has some philosophical differences with digital.

 

[joshua csehak]

for example, in my work with computer graphics, i tend to use numbers that make sense... for example, the Kelvin number in RCX... i will not allow myself to dial in random numbers with the sliders... that things not going to be say, 6846... it will be 6845, or more likely 6850 or 6800/6900. You'll never see me let some random 10th decimal place values go through. I typically use the number pad instead of sliders. I typically use numbers like... 8, 16, 32, 64, 256, 512, 1024, 2048, 4096, etc. (powers of two) when it makes sense to do so, and especially for intense computations.

That's funny. I can get the same way, but I have the opposite reaction. I purposefully don't look at the sliders or numbers when I adjust them. I don't want my intellectual brain influencing my visual brain. I realized a while ago while mixing music that it was only when I shut my eyes when I tweaked anything that I could get a decent mix, or EQ. Of course, YMMV.

I always use the highest compression ratio that looks good and holds up to intense color correction (like if I shoot at the end of the day and need to make it look earlier). Smaller files translate into faster transfers and more capacity, and saving time and money is always a good thing. I wonder though, do different compression rates process at different speeds? Has anyone done any tests?

 

[paulherrin]

yeah, that may be some good advice.

 

[Thomas Akira Nakasone]

Quick question, can you record to ssd to 100% capacity or do they function like a spinning drive where it is recommended you leave some "breathing room"?