Cinematography of 2030s: Ultra HFR! I have witnessed realtime 1000fps on real 1000Hz.

[David Peterson]

The ASC article on Billy Lynn's Long Halftime Walk (Dec. 2016 issue) may have some interesting information for you.

Something that caught my attention was the fact that they were starting to experiment with dynamically changing the displayed frame rate in shot.

Since they were starting with a HFR source, they could start a shot at "24FPS" but then when a bomb goes off, they could change to "120FPS", to give a heightened sense of realism, all without cutting.

Damn that would be very cool! Frame rates changing within the same shot

 

[Eric Z]

Mark,
Thanks for your info on Ultra-HFR.
I'm still waiting for a 32" UHD 10bit HDR 180Hz monitor to come out.
With that said, 1000Hz (1ms) is the holy-grail.
And like you said, that's at least 10 years from now.

 

[Les Hillis]

Fascinating stuff Mark, thanks for sharing.

 

[Mark Rejhon]

You're welcome!

I'm glad reading about the future is helpful (especially as it is already beginning to affect a number of engineering careers today, and is liable to filter down to videographers for specific-applications in the years/decades to come).

Today, this thread is literally reading the equivalent of 1980s Japanese MUSE HDTV experiments -- whoosh -- four decades before 1080p becomes extinct from Best Buy (that almost exclusively sells 4K today).

The Hertz race to successfully leap the uncanny valley will be a monumentally difficult one taking years/decades to successfully complete.

 

[Marc Wielage]

The thing is - any film or a movie should be projected at the speed it was shot, otherways you're changing the way you precieve the basics of the motion in the film. There is absolutely no need in "BLURLESS" image, more to say - I personally find a blurless or a HFR image distracting and repulsive. Speaking of the thing's I've tested - 60 fps/hz look awful, 120 fps/hz look even more distracting. It literally thows you away from the story into the rushing image.

I agree 100%. I'm totally opposed to any kind of "blurless motion" that takes you out of the story. To me, 24fps is so ingrained as the look of dramatic scripted motion pictures, any deviation at this point is going to be weird. I think it's embedded in human DNA.

However I don't have a problem at all with high frame rates used for Theme Park Rides, documentaries, travelogues, special venues, live events, concerts, news, and so on. In fact, I think it's a fantastic idea, very much along the lines of what Doug Trumbull has proposed since 1980 or so for Showscan. But I found the HFR used in Blly Lynn's Long Halftime Walk to be very off-putting. It just reminded me that I was watching a projected image -- it never let me get fully "immersed" in the experience. It was an interesting image and I think all the technical work on it was terrific, but I just don't think it added to the story-telling.

And I'm also very opposed to processing existing films to artificially frame-double or otherwise use motion-smoothing to eliminate the characteristics of 24fps film or digital. To me, that's F'ing with the image too much.

 

[Karim D. Ghantous]

To me, 24fps is so ingrained as the look of dramatic scripted motion pictures, any deviation at this point is going to be weird. I think it's embedded in human DNA.

We all grew up exposed to both interlace and 24fps, and, without prejudice, we always preferred 24fps. I don't have to appeal to history, I just have to appeal to aesthetics.

However I don't have a problem at all with high frame rates used for Theme Park Rides, documentaries, travelogues, special venues, live events, concerts, news, and so on.

Absolutely. I would argue that it actually is preferable for live sport to have (genuine) HFR.

I'm also very opposed to processing existing films to artificially frame-double or otherwise use motion-smoothing to eliminate the characteristics of 24fps film or digital. To me, that's F'ing with the image too much.

You say it more politely than I can.

 

[Mark Rejhon]

And I'm also very opposed to processing existing films to artificially frame-double or otherwise use motion-smoothing to eliminate the characteristics of 24fps film or digital. To me, that's F'ing with the image too much.

As the director intended!
I often turn off stuff like "Motionflow Interpolation" on my TV, since I prefer to watch movies as cinematographers intended. 24fps for traditional films for me.
(That said, sometimes cinematographer intent is to eliminate blur, for various special applications).

We rarely intentionally degrade our resolution to VHS for movies (Except for niches like Blair Witch Project or Apollo 13), so when Ultra HFR becomes freebie in the whole chain (storagewise-camerawise-displaywise-distributionwise).

New director intents will be invented by Ultra HFR that we don't even dream of yet.

Instead of limiting our video file format resolution, we can use camera blur to degrade resolution by choice for specific scenes. Likewise, for Ultra HFR, we can intentionally to degrade temporal resolution by choice, like sticking to plain 60fps most of the time, using 1000fps only for specific fast sequences wherever motion blur may produce more more nausea in current prevailing population. Ultra HFR gives us that full choice and flexibility to use whatever temporal resolution we want, and intentionally degrade it accordingly.

For example -- most of the time, initially, Ultra HFR might be played with mainly 24fps, 48fps, 60fps, 120fps sequences, with 1000fps sequences only for specific sections. If it's a fully VRR-compliant Ultra HFR, the framerate can even gradually ramp up (24fps->25fps->28fps->40fps->60fps->80fps->100fps->200fps->300fps->600fps->1000fps realtime) for a less-jarring framerate-transition effect that is not sudden, and does not require a scene change. This effect can be quite dramatic, and I've actually partially simulated this effect in a motion animation:

Want to see over 100 different framerates changing in the SAME shot, in a fully seamless manner?
See Variable Refresh Rate Simulation

(Recommended: NVIDIA or AMD GPU on the Chrome web browser: You need a stutterless web browser)

This animation simulates a variable refresh rate display demonstrating a stutterless smooth framerate ramping, simulated via interpolation (requires a modern GPU-accelerated web browser to play this specific TestUFO animation without artificially added web browser stuttering). If you're using a newer iPad or iPhone, it'll even work on those too!

NOTE: Yes, there's some minor interpolation-blend artifacts. Please forgive the interpolation artifacts (because browsers do not support variable refresh rates). A real VRR displaly (and one without defective overdrive) does this framerate-ramping much better than my interpolation simulation, but this animation demonstrates that framerate-changes can be fully seamless with zero jarring effects!

This web based animation is best seen in Chrome on a 240Hz, 144Hz or 120Hz computer monitor, where you actually see the motion blur successfully thin from 1/30sec blurring slowly down to 1/240sec blurring, unbottlenecked by a display limitation. It's still useful animation to witness on any 60Hz display although if you test this animation only on a 60Hz display, your range is limited from 30fps to 60fps). This is all theoretical of course -- but applicable to future possible cinematographer intents.

There are thousands of possible tricks that can be done with Ultra HFR, not just neat "seamless gradual framerate-acceleration/deceleration" effects.

The venn diagram of Ultra HFR lets it play traditional 24p too, or pretty much any useful framerate, and sometimes in ways that the framerate transitions are not even jarring (as seen, for example, in the gradual framerate-blending animation above). Many things people haven't yet imagined as cinematographers.

Given proper technique, it is scientifically possible to have 1000fps sequences will be inserted in 24fps films, without the audience immediately noticing where the transition occured because it was so seamlessly done; the motion blur and stutter simply "fades away". Much like a panning shot decelerating -- except the decreasing-motion-blur effect is occuring while the camera is still panning the same speed! (as demonstrated in the animation above!). Figuratively; imagine this: One second you see 24p-like blur. Scene still panning, the scene is 48p-like blur. Scene still panning at same velocity, half a second later it is 120fps-like blur..... scene is still panning, now it's CRT-crystal-sharp (zero blur, zero stroboscopics) while the scene is still panning. One nonstop pan at constant velocity, but the blur gradually decreased seamlessly from 24mm film look to perfect CRT look in a 2 or 3 second gradual blendover. It's a magical effect, not too different from the Wizard of Oz effect where the black and white turns color, and vice versa.

Also, even if you don't have a VRR display, 1ms frame-blending techniques are so extremely fine-grained -- so you can theoretically simulate any low framerate in Ultra HFR via multiple kinds of framerate-antialiasing algorithms not too different from the one in the link above that I developed to try to simulate VRR on a non-VRR display. (Mine is a crude algorithm too, yet it works so well).

(This post makes a lot more sense if you play the above TestUFO VRR simulator animation on a fast GPU-acclerated (chrome://gpu all green) web browser on a 120Hz display. If you have a HFR display, test the animation, and try Slow Framerate Ramp, Fast Framerate Ramp, Random Framerate, Struggle At Max -- you will better understand fully seamless & stutterless gradual framerate changes (even for random: It's surprising how a 10% randomization of framerates look visually stutterless).

Anyway:

The moral of the story: All kinds of magical future cinematography techniques is made possible via Ultra HFR, too.
Not just what I write about in this post, it's just one of many theoretical examples (And already demonstrated in experimental motion tests).

 

[Mark Rejhon]

I think it's embedded in human DNA.

More the "Grew up with" factor than the "DNA" factor, I think.

I have observed many millenials preferring HFR because that is now what they grew up with their TVs in their default Motionflow/interpolated modes, and they don't like the classic 24fps look. The first TVs that had the interpolation came out 15 years ago, and there are now adults today who has rarely had to spend long times with unretouched 24fps content.

we always preferred 24fps.

I grew up with 24fps, and I prefer 24fps for movies, even if I appreciate HFR for other content.

BUT....an important observation: Talk to people who almost exclusively grew up with interpolation (people who spent 10 years never watching 24fps or sub-HFR motion)....and....most of those people say 24fps is junk.

Interpolation invented almost 20 years ago. Now we have budding cinematographers no longer preferring 24p. Some are niche and indies, but they're climbing the ladder. If you grew up with a specific frame rate, you're used to it, and 24fps looked junk to them. It's funny how civilization works this way. But the bottom line -- the new generation (on average) is less 24fps-hardcore than I am. This may be a early-canary HFR signal too. Or it may not be.

It's easy to assume 24fps is DNA related, but the pattern shows it is not.

Grandkids Career Key Info: People who grew up childhood never being conditioned to 24fps tend to no longer prefer 24fps. (e.g. Spent childhood sitting in front of permanent-interpolation TVs, over one decade of exposure).

So it appears, from this observation, 24fps is not in human DNA, but more from long-term human conditioning.

Bet it's very annoying and hairpulling for cinematographers -- kids today who has never seen original director intent -- because of permanent-default interpolation in a Walmart HDTV.

Related conditioning factors
--> If you've spent a lifetime staring at flickerfree LCD displays, you're likely statistically more likely to get nausea/headaches from suddenly having to watch a CRT tube (than one conditioned to CRT).
--> If you've spent a lifetime staring at HFR content, you're likely statistically far less likely to get nausea like a 24fps cinemaophile suddenly watching HFR content (than one conditioned to 24p).
--> Etc.

Never before in humankind, we have had kids growing up with exclusively-HFR-like content (e.g. permanent interpolation TVs).
It's unfortunately happening, whether we like it or not.

Such kids and young adults now find HFR more comfortable than 24fps on displays larger than a smartphone's. We see more and more reports. I've met literally hundreds of people like this already and we shouldn't dismiss these kids as "lesser cinemaphiles" -- some are intelligent and some are actually beginning to work for places like Apple or Facebook or Disney. So, from a "portent-of-the-future" perspective -- infuriating and so goddamned maddening as it may be to cinematographers. We'll stick with 24fps for probably a hundred years, but it might not be humankind's permanent dominant framerate in the decades or centuries to come. At least, smarter-and-smarter interpolation is finally making interpolation more and more true-HFR-like, but that's still not director's intent. I agree!

Possibly in fifty years from now, you can simply just tell a Holodeck algorithm to intelligently convert and build a 3D environment from a single movie (for virtual reality), THEN export it back to a virtual camera angle back to a perfect different framerate (and completely new, exact amount of preferred source blur) with no Motionflow or SOE artifacts with perfect identicalness to as if the film was actually mastered that way. You want to convert 1000fps to 24p sufficiently visually perfectly? Sure. You want to convert 24p to 1000fps HFR sufficiently visually perfectly? Sure. Who knows? The wondrous inventions of the future cannot be predicted, but SOE/Motionflow artifacts will at least eventually and gradually disappear. For better or for worse...

At first, more than 50% hated HFR-look. Now that's rapidly falling (to 25%). Still big enough to produce big negative HFR reviews. But at some point, a critical mass may be hit in a future decade where the benefits of HFR outweighs the bad of HFR, for an increasingly HFR-inoculated population. HFR is still faddish. But it may not be forever, if humankind eventually suddenly (by majority) prefers HFR when all the HFR problems are five-sigma'd out (e.g. Ultra HFR) and such. Who knows?

At least, to understand, that it is a conditioning factor (Not assuming it is a DNA factor) -- it is to be forearmed with the info, in a multi-decades-timescale evolutionary moves in cinematographer intents, including theoretical migrations away from a 24fps-standard in a far-future decade or century.

The Good News: Your 24fps career is (probably) safe for now!

The good news is that the framerate progress is slow, and framerate flexibility gives director choice. But these kids and young adults, tomorrow's cinematographers, creating tomorrow's films, may not be sticking to 24fps forever.

Many of us will have happy 24p careers lifelong, even 20 or 30 years from now. But our grand-grandkids may now be facing a world where 24p is not the most popular/profitable framerate to use cinematography techniques with. Mark these words, time-capsule this, and visit this thread in year 2068...

But, what this means, is your great-grandkids almost definitely may not make as much profit with 24p as with other framerates. Initially Ultra HFR will probably niche applications, the feature is built in for free in TVs/projectors of the distant future. Feature available for free in cameras. Temptation by cinematographers. More demand from a HFR-conditioned public, etc. All dominoes in place to not bother with 24p. Just like 4K has become much easier today than even just 5 years ago. 10 years ago Apollo mission, today pocket phone. The same is now happening to slo-mo cameras (smartphones with 240fps, 480fps and 1000fps highspeed cameras), and now we've got 120Hz iPads.

Today, us 24p-loving cinematographers.... We try to keep teaching the new Motionflow Generation to do 24p, but some cinematographers are resisting. But they'll have kids too, and they won't be as 24p religious, and Motionflow Generation Mommies/Daddies (who were reluctantly convinced by *todays* cinematograph in 24p) will no longer indoctrinate mandatory 24p cinematography to their own next generation kids into that. The great-grandkids generation are, thus, free of 24p encumberance. Bigger percentage of HFR content shown to bigger-percentage used-to-HFR audiences. By that future, only the 70 year old grumpies complain about it not being 24fps. Eventually, herd HFR immunity kicks in. See?

Pick your timescale -- I only use two human generations as an estimate -- ~50 years -- but you might prefer to imagine 100 years as the 24fps-deprecation timescale. In twenty years, niche purposes. In one hundred years, ultra HFR becomes the mainstream default format. Or by then, a future H.269 or H.270 or H.271 video compression format is a framerateless 3D geometry format that can be played at any frame rate with flawless legacy 2D-plane 24fps film look but also maps to 2D 1000fps or to fully 3D Holodeck/VR/holograph/etc too. Imagine that being the great-great-great-grandkids director intent as the cinematographic technique then! Who knows?

Fully consistent industry-wide de-coupling from 24p standard may not happen in my lifetime, but may be in great grandkids lifetimes because of circumstantially widespread Ultra HFR because of a True-Hz Moore's Law that finally got ignited only ten years ago in gaming monitors (Hertz doubling in gaming monitors every 5-8 years; now 240Hz market and 480Hz experimental) that eventually infects lots of other displays.

Freebie-ized features
High speed video is now almost a freebie feature in cameras (at 120fps & 240fps scales) and will be a freebie feature at 1000fps scales in every single billion smartphone within 5-10 years.
4K is now reaching freebie feature status, and excess fps/Hz may also be too -- all the pieces of chain are going to fall in place for Ultra HFR within one or two human generations.
Now combine that with the Motionflow Generation, huger population inoculated to to prefer HFR over 24p, and when the HFR curve goes up sharply it finally leaps some uncanny-valley factors, and become much more useful than today's HFR once the cost of 1000fps realtime HFR is sufficiently low by then, like 4K is today. So, who knows? A multi-decades-log Rube Goldberg journey, of domino effect, if you will. As mentioned -- bookmark this thread and have your surviving kids come back here in year 2068 via Internet Archive if this text survives. I'll probably have (mostly) been more prophetic than you've thought.

It's probably is a losing battle by 50 years, mark my words. We probably should not make the mistake of assuming 24p is the predominant new cinematographer framerate, given all of these observed trends.
24p will still be used in fifty through a hundred from years ago, but 24p may be the niche format then. Who knows?

Meanwhile, I'm old enough. I enjoy my damn movies in original-director-intended 24fps format. :iagree:

 

[Karim D. Ghantous]

(e.g. Spent childhood sitting in front of permanent-interpolation TVs, over one decade of exposure).

Making kids watch vision which has been motion-interpolated is child abuse.

 

[Mark Rejhon]

Making kids watch vision which has been motion-interpolated is child abuse.

Ah, yes: Interpolation may be evil today....

....but let's not forget that H.264 and H.265 video formats (E-Cinema! 4K Blu-Ray! Netflix!) are very heavily interpolation-mathematics based, because it is the mathematical basis of video compression. For example, H.264 is formed of I-Frames, B-Frames, and P-Frames, using predictive interpolation techniques, motion vectors. Including nearly flawlessly jiggling-around parts of previous frames on different parts of a new frame.

As long as you don't compress too much, throwing away more than 99% of picture information brilliantly succeeds with the interpolation math embedded in modern video file compression formats. Otherwise we'd be blasting a stone's throw of 10 gigabits per second just do 4K 24fps cinema at full color depth. Even E-Cinema isn't delivered uncompressed.

In an "Understand the devil, know the devil" fashion...

Correct: we don't consider H.264/265/EVC video compression as being interpolation technology. But it is.
The mathematic concepts are similar, in motion vector handling and many related concepts.

That said, the name of the "interpolation-is-good" game is fully-source-aware interpolation (e.g. H.EVC compression, or geometry/parallax-aware 3D framerate-increasing algorithms) is often desirable, while black box heavier-guesswork interpolation forced into the chain (e.g. Sony Motionflow) is often undesirable.

One day into the future, when we have framerateless 3D-geometry-based video file formats (imagine a H.269 or H.270 or H.271 video file format) that can be reprojected to any frame rate, in a fully-geometry aware manner (no parallax problems, no unwanted framerate-change problems, no edge artifacts, etc) -- this argument will be pretty moot when the evil "Soap Opera Effect" is no longer artificial and reprojected framerate is fully natural and source-aware; i.e. it properly looks like an original-mastered framerate. As one thoeretical possible progression path for a future ultra-advanced video file format several generations ahead of today's H.264 and H.265.

Graphics rendering can finally do that now (you can play the exact same videogame virtually flawlessly at a higher or lower framerate, with more or less powerful GPUs, and many games compensate visually flawlessly and stutterlessly -- at least on a variable refresh rate display like G-SYNC and FreeSync. But within our grandkid's lifetime, the same thing may happen to video if a framerateless method of recording & video file format occurs. It'll probably be a long time (possibly decades) before such a video file format appears in humankind, but the discussion exists now at least theoretically.

Mathematically, a framerateless video file format is possible!

Theoretical timecoded photon cameras (and certain camera concepts) are framerateless too. Playback is then, thusly, judder-free and flawless at any framerate you so desire. A video file is framerateless when it is stored essentially as analog-motion-vectors or as accurately timecoded individual photons. For a theoretical framerateless video file format, then what is interpolation? Projecting further out in the future: One thus ponders the meaning of the word "interpolation" in the concept of a framerateless cinematography world. A theoretical video file format that plays back perfectly at any your preferred/desired frame rate (no divisors needed, can play the exact same file to 24fps, 55.31245fps, 59.94fps, 89.123124fps or whatever, no divisor worry, infinite choice of framerates are all perfect. When considering theoretical analog-motion-video-encoding or theoretical timecoded-photon file formats).

This may not happen, nor within our lifetimes, but the theoretical talk exists now. The word "interpolation" may not be part of the vocabulary for a framerateless video file,

In the interim, virtual reality researchers are doing some amazing research in lagless frame rate increasing algorithms that skips scene-rerenders, but still uses original geometry/positional information and multilayered-awareness (e.g. knowledge of what's behind objects), to produce the necessary intermediate frames. Much like what you now have to do on E-Cinema, Blu-Rays, and Netflix with today's interpolation-math-based video codecs.

 

[Hrvoje Simic]

1000 fps image acquisition for a mass market makes exactly zero sense.

50/60p and 100/120p should be just fine for motion interpolation algorithms due to absence of motion blur.

More the "Grew up with" factor than the "DNA" factor, I think.

More due to the principles of vision and psyche and less due to the conditioning part.

 

[Mark Rejhon]

1000 fps image acquisition for a mass market makes exactly zero sense.

50/60p and 100/120p should be just fine for motion interpolation algorithms due to absence of motion blur.

While interpolation (to >1000fps) will be more practical, there comes a point where it's favourable to have as much temporal data recorded as possible for various niche applications. What is "mass market" and what is "niche" is subject to debate -- until you consider that things like VR and holodecks may be popular in the future (where ultra HFR may indeed eventually become more mandatory, if acquisition is sufficiently inexpensive). For niche applications, there's still too much motion blur in 120fps HFR -- if one needs to use blurless 360-degree shutter combined with blurless sample-and-hold for passing the Holodeck Turing Test (e.g. future virtual reality that looks like true reality).

There are some situations why it's favourable to use 360-degree shutter recording (source) on sample-and-hold displays (destnation), to avoid 100% of all stroboscopic effects. Now, when you try to also simultaneously eliminate motion blur simultaneously with eliminating stroboscopic effects at the same time -- the mathematics fully point to ultra-high Hz being mandatory for this specific purpose.

Also, for 360-degree shutter and sample-and-hold displays, the source-based blur and the destination-based blur are additive:

60fps@60Hz = 1/60sec camera shutter + 1/60sec display persistence = 1/30sec final motion blur
120fps@120Hz = 1/120sec camera shutter + 1/120sec display persistence = 1/60sec final motion blur
240fps@240Hz = 1/240sec camera shutter + 1/240sec display persistence = 1/120sec final motion blur
480fps@480Hz = 1/480sec camera shutter + 1/480sec display persistence = 1/240sec final motion blur
1000fps@1000Hz = 1/1000sec camera shutter + 1/1000sec display persistence = 1/500sec final motion blur
2000fps@2000Hz = 1/2000sec camera shutter + 1/2000sec display persistence = 1/1000sec final motion blur

You can try to use interpolation, but then you can have difficulties with things like complex parallax effects (e.g. driving through a forest) -- see this motion animation which is impossible to interpolate, but actually becomes 8x more detailed at 480Hz versus 60Hz in actual tests. Once cameras can record everything behind every objects (e.g. record the scene essentially holographically, or convert to virtually flawless 3D geometry) interpolation can then much more practical as an Ultra HFR solution.

Regardless, the higher resolution (retina resolution) the more framerate you need to fix interpolation artifacts. It's like flaws showing up more in HD than in SD, and that goes with interpolation too -- and when we scale it to retina Holodecks, that becomes super-demanding -- to the point where 1000fps Ultra HFR becomes much more essential to pass the Holodeck Turing Test (aka, ability to trick a human into being in a virtual world). If you want to read more, see my replies in the last few posts.

Have you read this article and also this -- and what is your opinion?

 

[Hrvoje Simic]

While interpolation (to >1000fps) will be more practical, there comes a point where it's favourable to have as much temporal data recorded as possible for various niche applications. What is "mass market" and what is "niche" is subject to debate -- until you consider that things like VR and holodecks may be popular in the future (where ultra HFR may indeed eventually become more mandatory, if acquisition is sufficiently inexpensive). For niche applications, there's still too much motion blur in 120fps HFR -- if one needs to use blurless 360-degree shutter combined with blurless sample-and-hold for passing the Holodeck Turing Test (e.g. future virtual reality that looks like true reality).

Mass market is the majority and niche a minority. And performance to resources graph would show an S curve.

And as the performance reaches the peak the amount of resources needed grows exponentially. Rule of the thumb.

Relation of temporal to spatial resolution is a good point. VR argument stands but in the context of CGI. Ultra high resolution acquisition in those extreme framerates to cater that domain is far from reality. Holodeck is science fiction at this point and I don't see the premises pointing otherwise.

Thanks for the effort invested in research and in sharing your findings.

 

[Mark Rejhon]

Temporarily setting aside the commodity versus niche argument:

Let's focus on the inevitability side instead.

I'm not sure data acquisition will always be too much effort because 4K was science fiction 20 years ago when the world were using huge broadcast trucks just to barely broadcast 1080i in the first HD Superbowl or Olympics broadcast. Today, we can realtime compress and record 4K on our pocket smartphones.

Now, today a Sony Experia XZ Premium smartphone can do 1000fps in your pocket (albiet only for 5 seconds of slo-mo due to the data firehose). Within five to ten years, every single smartphone will have a high-def ultra-slow-mo feature capable of 1000fps continuous.

Also, DLP projectors use pixels that can modulate at up to 10KHz, and many scientific projectors have repurposed them to 500Hz and 1440Hz projectors. Intrisinically, it costs only $200 to add 1000Hz capability to a DLP projector -- to a certain point (bandwidth of video input).

The "4K circle" has only recently become cheapish-enough (cameras, storage, and displays).

I expect the "1000fps circle" to hit affordability within 20-40 years (the "two human generaitons" I posted in Page #3 of this thread) -- cameras, storage and displays having access to 1000fps/1000hz feature for "free" -- the more data, the more temptations to use the feature, the easier interpolators can do the rest of the work, etc.

This time around, cameras with 1000fps will go cheap first, instead of displays, but even displays will gain 'free' 1000fps eventually for whenever they're used for niche purposes. (Hey, it's an additional marketing bullet for a display, if the engineering is cheap, huh).

Refresh rate is getting cheaper, too.
I have also run into cheap $1200 DLP projector with an undocumented free unadvertised "240Hz" feature (via an unusual trick known to only a few). The extra true refresh cycles are indeed getting cheaper. And more than a quarter of HDTVs in year 2014 in Best Buy were overclockable to 120Hz - the undocumented HFR feature was trojanhorsed because of 3D glasses (60Hz/60Hz per eye) and those in the know were able to unlock this 120Hz HFR feature for videogames and such. The extra refresh cycles is not that expensive, once the bandwidth is there.

Let's ponder this for a moment: How many floppy discs does one need to store just ONE frame of 4K video? How many minutes to play it back and decode using late 1980s or early 1990s computer performance? From that, even with the slowing Moore's Law -- extrapolated a human generation or two -- tomorrow's storage and performance likely still handle 1000fps with no problem. If the feature is available for free, the temptation exists to use it anyway, potentially even for 24fps mastering.

There are many vectors that 1000fps/1000Hz will sneak into our certain areas of lives by the end of this century, in a rapid commoditization progression. How niche / not-niche is up to debate, but the inevitability is definitely there, as described. Slo-Mo will be the biggest marketing thing, with 1000fps arriving to every single smartphone cameras ever manufactured by the end of 2020s(ish). Displays will take longer, but also clearly inevitable, too.

Now that full-chain 1000fps/1000Hz capability seems inevitable within very few decades, one focuses on whether cinematographers want to even bother using the capability. So, let's ponder that angle for a moment, too:

If the temporal resolution is pervasively available anyway, it's simply light output. Higher framerate means fewer photons per frame. But think about it in as more abstract way: That then is solvable with smart artificial-intelligence denoisers that combines bright low-framerate with dark high-framerate. That's a much easier interpolation job than interpolating 1000fps from scratch using only 60fps or 120fps HFR. So the mantra is: Record all the photons at their maximum temporal resolution: You're still capturing the same number of photons whether it be 60fps, 120fps, 1000fps - and the file format might even be a timecoded photon file format that can be played at any frame rate. Problem solved, job easier for smarter interpolators, play the same file at multiple framerates, use the bright 24fps master, but use the photon info of the higher-temporal-resolution to much more correctly/perfectly interpolate parallax/backgrounds better, with far less guesswork, whenever a HFR or Ultra HFR version of the same file is needed for any purpose, etc. Viola. 1000fps video with full 24fps brightness, and a much more perfect interpolation with near-zero guesswork.

Heck, you might intentionally do proper 24fps from this futuristic framerateless file format, but the full temporal resolution is optionally available when remapped to other purposes (e.g. virtual reality) since the access to fuller 1000fps temporal information can technically allow for more superior and flawless interpolation. Any framerate is native framerate from theoretical framerateless video file formats (e.g. vector/curve motion 3D geometry files, timecoded photon recording, or other framerateless video files, etc) --

This might even actually one day (in 20-40 years) constitute a future VR-friendlier video codec several generations in the future such as H.269, H.270 or H.271 -- when it is eventually time to move away from macroblock-based formats to true geometry-based video formats that can be remapped to 2D, 3D, VR, holo, screen rectangle of any size or surround true 6dof 3D (not stereostopic; actual full true geometry), etc.

The more useful a 2D 24fps file is for other purposes (gaming, VR, CGI, etc) because it has all embedded original 1000fps+ temporal information via any method -- the more temptation cinematographers will have to record using a framerate-independent file format even for 24fps -- thusly making 1000fps available for virtually flawless full-brightness noise-free interpolation.

Also, with full temporal resolution (e.g. 1000fps) embedded in every file, choosing specific action to slo-mo becomes much easier too -- very convenient to just always keep filming in 1000fps if you have it for free (cheap camera, cheap storage) just so you can retroactively slo-mo later, too. The temptation is there (if eventually available for free with virtually no loss of quality) to just use 1000fps. (Who records in standard-def to save phone memory today?). Light output is not a problem if you stop treating it as a "photons-per-frame" mathematics problem, which can be bypassed through what I described above.

There's a curve that's going to cross. At some point, a near-uncompressed macroblock-based (e.g. H.264, H.HEVC) HFR 120fps >8K 360 degree stereoscopic video file, then starts to look more economical when instead compressed using a different compression algorithm into a framerate-independent 3D geometry file format. Much better parallax info, gains 6dof instead of stereoscopic, much more reuse over more frames, with an attached massive automatically-generated 3D object template library (built, say into a future video compression format, like H.269, H.270, etc) that constitutes common textures/objects but rendered to retina resolutions indistinguishable from real life. At this stage, 1000fps acquisition may actually no longer consume more data than 120fps acqusition. One then would be able to play the same video file as a 24 frame per second motion-blurred rectangle flawlessly, but the same video file would theoretically also play back flawlessly as a 1000fps surround VR headset too, or be imported into a game for rerendering. Basically GPU becomes the future playback, rather than traditional macroblock playback. The key is, all original temporal resolution can be thusly embedded in whatever video file format, to make 1000fps+ original temporal data available -- for whatever purposes (better 'perfect' interpolators that avoids temporal guesswork, better game world design, VR headsets, Ultra HFR playback, future holodecks, etc).

Even Scientific American had a 1989 article (1990?) on theory of compressing video as geometric objects (like a video game world file). It has not yet happened, but virtual reality starts to make such a video file format more necessary. Once cameras compresses, realtime, straight into 3D geometry file format, and players are all 100% GPU based -- metaphorically it's essentially converting real world into the equivalent of "flawless CGI" that can be played at any desired framerate. The estimate is that a video file format could be a reality within our grandkid's careers

From a different angle: Every single pixel of 24fps motion blur would have all the timecoded photons, so you can do virtually flawless deblurring without the normal interpolation guesswork. There are many possible directions that video recording can head in during this century, but this is one of the many directions of research.

---

Also, while holodecks is definitely science fiction today (except all these 'lab experiments' ongoing), VR headsets is not anymore with the rapid progress occuring now -- some background information: At CES 2018 they demonstrated a rudimentary "Holodeck" using VR headsets at NVIDIA's booth. Multiple people wearing VR headsets throughout multiple rooms, to interact with real-world objects in a virtual room. Apple is spending huge amounts of money on finding compact VR tech (e.g. VR headsets are as compact as sexy sunglasses) and cannot be dismissed as a potential future player if they succeed.

Although the topic subject line, does estimate the first public niche uses of Ultra HFR will likely be the 2030s, the real large expansion of pervasive (optional yet 'free') capability is still actually beyond this (e.g. 20-40 years) but within our grandkid's lifetimes as i mentioned in some follow-ups in this thread.

 

[Mark Rejhon]

Part 2/2

Here's a very, very wright brothers attempt of converting a video file into a true 3D explorable world (university research):
https://www.blurbusters.com/videos-can-now-be-converted-into-explorable-3d-rendered-worlds/

This is essentially a frame-rate-less "video file" -- TODAY.

It's very rudimentary, crude, low-resolution, like the 1920s Farnsworth TV tests. It doesn't yet use AI to fix flaws. But impressive nontheless. Project this 50 years into the future.

Another method is Microsoft's Hyperlapse, which reprocesses video into 3D geometry, then runs a virtual camera. The virtual camera can be running at a totally different frame rate independent of the original source material. That said, the more temporal resolution is available in the source material, the much more accurate it all becomes especially with heavy-parallax scenery (videoing while moving through forest foilage and its refresh-rate-induced spatial resolution limitations making good interpolation nearly impossible without 1000fps Ultra HFR).

Again, all of this is still wright brothers league. By our grandkids careers, we may pass the "Virtual Reality Turing Test" -- whereupon a video file can then be entered into a cloud processor that converts the video file into a virtual environment. And then a blind A/B test between a VR headset versus similar-looking transparent ski goggles -- people accidentally thinking real world is virtual -- and accidentally think virtual is real world.

This holy grail, potentially achievable by our grandkid's careers, is already being researched & worked on today.

Eventually -- as 1000fps gets commoditized (there's already a 1000fps smartphone) just like 4K did in phones -- eventually it becomes much more important to include 1000fps HFR data in the video compressors, for more flawless recording of 3D world data -- even if you're only acquiring 24fps.

Camera manufacturers masterplanning a 10 year business plan that includes 6dof 360 video -- and video file format designers (Moving Picture Experts Group) -- take note. If your engineers can't figure out the science papers on Google Scholar, send them to Blur Busters (and this thread) for the Easy Cole Notes 101. Everyone tells us we're the best mainstream site that explains it in an easy way.

It is not necessarily useless to acquire maximal temporal information of each pixel in a future when eventually the bitrate-hog disappears & light-loss-per-frame disadvantage disappears (as explained in previous post). If the feature exists "for free" (as explained in the previous post's commoditization journey), then all kinds of spinoff applications occur. Whether you want to slo-mo in post-process, or to produce a VR/game version of your movie, or much more flawlessly readjust your botched camera angle in post-process, or such. The 24fps still looks perfect and fully bright as if originally filmed at 24fps.

We often record at 4K or 8K, and downconvert to 1080p or 4K with benefits. The same happens when you record maximal temporal resolution (1000fps+) and downconvert flawlessly to better-quality 24fps.

We found solutions to the small pixels problem, it's simply the same number of photons spread over more smaller pixels. The same is (slowly) happening to temporal resolution, same number of photons spread temporally on the same pixels. With the appropriate eventual sensor improvements, there is theoretically no quality degradation from increasing temporal resolution in acquisition.

<SENSOR MANUFACTURERS TAKE NOTE>
Sensor readout waste is declining rapidly, and eventually we may reach the point where we've essentially (defacto) timecoded batches of photons at literally sub-millisecond timescales. Initially it will happen very crudely -- such as possibly via a multilayered stacked sensor that vertically bitshifts in sub-millisecond timescales (even 0.1ms), essentially timecoded layers that finally gets darn near-lossless readout at the bottommost layer. Basically a 3D spin of old-fashioned bit-shifting sensors. We already have SSD flash chips with 64 layers of silicon (including my 1 Terabyte Samsung 960 Pro M.2 SSD in my computer) and eventually we may gain vertically-bit-shifting stacked camera sensors for easy convenient high-speed-video recording too. The first 1000fps smartphone sensor (Sony Experia Premium phones) is also a multilayered sensor, but once 64 or even 128 layers are done, this is where the real timecoded-photon magic (at least at ~0.1ms granularity, and eventually potentially microsecond granularity) begins to finally happen, especially if the bottom-most layer of the camera sensor is essentially a GPU that does an intermediate intelligent datarate-mitigating of the data firehose for a downstream AI video compressor - most pixels do not change color in 0.1ms so plenty of opportunity for datarate mitigation in an "low-energy in-camera-sensor co-GPU" built into the bottom layer of a camera sensor - to make continuous recording much more practical for framerateless video files. And other wondrous inventions that make recording framerateless video files a reality. Sensor manufacturers are brainstorming ways to efficiently record brighter high speed video for slo-mo, but the same principles can also be applied to generating a data stream for a framerateless video file too.
</SENSOR MANUFACTURERS TAKE NOTE>

There is not necessarily a light-loss disadvantage when the problem is attacked as a framerateless mathematics problem from the video file POV. The same number of photons per second hits the camera lens, regardless of whatever framerate you record at.

List of Advantages:
A framerateless video file, as 3D geometry, has many spinoff applications:
-- Full brightness (all photons captured)
-- Original 35mm 24fps motion blur (if desired)
-- Flawless deblur (if desired)
-- Play flawlessly at any frame rate
-- Permanently removes guesswork from crappy interpolators (true full original temporal resolution)
-- True 6dof (good for VR ports, good for game ports)
-- Create slo-mo segments in post-process
-- Readjust camera angle in post-process (minor angle adjusts won't need fixing of parallax flaws)
-- non-HFR, HFR and Ultra-HFR realtime playback
-- Calculations indicate it should compresses better than H.264 when it comes to 120fps 8K+ HFR 360 retina stereoscopic
-- All original temporal resolution of all pixels for whatever purposes you want.
-- Etc.

The dream concept of a video-compression-as-3D-geometry, first widely conceptualized in an old issue of Scientific American, is slowly (in pieces) finally becoming reality. Few people (other than people like me) are recognizing the pieces converging and projecting this technology progress. The journey's of all the simultaneous Moore's Law on all the pieces, the commoditization of all the pieces, points to reality this century. (And currently: I estimate within our grandkid's careers, but feel free to fudge the decade or two forward/back).

The powerful advantages of framerateless acquisition are so such, it will lead to inevitable commoditization of full temporal resolution recording (sub-1ms timescales -- ala true 1000fps capability -- at least indirectly in a framerateless video file format) by the end of this century, even if playback in some mediums (e.g. cinema rooms) is still at director-preferred framerates (i.e. 24fps).

Blur Busters high-Hz research from year 2012-2013 have proved uncannily accurate. And even my previous research where people couldn't believe 30fps-vs-60fps -- I have been battling that myth since year 1993, with one of the world's first popular 30fps-vs-60fps demo for MS-DOS, widely distributed during the FidoNet BBS days to settle old 30fps-vs-60fps arguments.

Calculations show that framerateless video files apparently can begin to compress better once we're heading into the stratosphere of 120fps 8K+ HFR 360-degree stereoscopic/6dof video files. That's where the crossover point is (approximately). Bam, infinite framerates for the same (or less) bitrate as 120fps HFR 360 retina 6dof!! And such video still maps nicely to Cinemascope 24fps 2.35:1 rectangles -- your favourite 35mm-look motion blur included -- it is simply a crop of retina 360 6dof.

Remove the framerate (which is already happening in Wright Brothers tests), problem solved. 24fps still looks originally cinematically perfect, but you gain infinite perfect framerates from a framerateless video file.

It will take time, but it's within the lifetime of your kids or grandkids already (give or take 1 generation). That said, yes, I'm hoping my advocacy (I contact many manufacturers too, including contract work) -- to help tip scales by about one-half generation earlier by wider education.

Yes, Blur Busters is high-Hz crazy. This ain't no infomercal -- it's advocacy & education! :smiley:

 

[Karim D. Ghantous]

....but let's not forget that H.264 and H.265 video formats (E-Cinema! 4K Blu-Ray! Netflix!) are very heavily interpolation-mathematics based,
[...]
Mathematically, a framerateless video file format is possible!

1. Yes, but spatially, not temporally. Interpolation is necessary for 'bullet time' effects, but that is a different thing.

2. Yes, I agree. Actually this is old news, although I am no engineer. I recall back in my 20s thinking about that kind of thing, though I forgot what gave me the idea: using a datastream per pixel, instead of per frame. So, a rough example (good enough for a lab) would be to use an audio cassette to record a stream for each photosite on a sensor. Yes, resolution is limited, but in a 'cloudy' sort of way as opposed to an on-off sort of way. Maybe wavelets could be used temporally, not spatially. That sort of concept doesn't need 21st century technology to do, but cost is also a question.

Part 2/2
Here's a very, very wright brothers attempt of converting a video file into a true 3D explorable world (university research):
https://www.blurbusters.com/videos-can-now-be-converted-into-explorable-3d-rendered-worlds/

It caused my browser to freeze the OS for a minute. Oh well! But with light field cameras, that sort of thing will be more useful. Nobody wants to waste brain power on trying to convert 2D into 3D environments. It's retarded. Light field actually does solve a problem, although it's still a bit crude.

 

[Mark Rejhon]

1. Yes, but spatially, not temporally.

Technically, it is semantics:

A motion vector is both spatial and temporal (movement over time). By definition, motion vectors are also temporal, there's a huge amount of temporal prediction (e.g. MPEG-4 AVC/H.264 Motion Vector Prediction Rev. 4). Codecs are expanding further into temporals.

Certainly, it doesn't do between-frame temporals, so it's not framerateless.

Still, essentially temporal interpolation mathematics: Many things in H.264 already run at ultra-low frame rates (e.g. 1 frames per second of I-Frames) and uses the magic of predictive compression to fill in the remaining frames (24 frames per second). The art of using more temporal guesswork in those bitrates sitting on your BluRays and Netflix, is actually happening even more in H.EVC than in H.264. But it's not guesswork-based interpolation -- it's actually a precomputed predictiveness/interpolation based on knowing what the original source material is. So by stringently sticking to an acceptable compression ratio (noise margin), you've avoided temporal interpolation artifacts (like blackbox interpolators) with the temporal mathematics built into H.264 / H.EVC. Whereas, a blackbox interpolator needs to guess the motion vectors and try to fill-in the intermediate spots.

That's something H.264 / H.EVC does not have to do, because the compressor already had full knowledge of the source material, and essentially (defacto) backtracked from there -- creating practically flawless predictivity (at reasonable compression ratios) -- allowing you to avoid guesswork-based predictivity. Sure, interpolators can do additional techniques very different from H.264/H.EVC, but there are many temporal television-based interpolators that simply hunts for motion vectors much like an H.264 compressor hunts for motion vectors, to create the intermediate fake-frames. The venn diagram has surprising overlap...

2. Yes, I agree. Actually this is old news, although I am no engineer. I recall back in my 20s thinking about that kind of thing, though I forgot what gave me the idea: using a datastream per pixel, instead of per frame. So, a rough example (good enough for a lab) would be to use an audio cassette to record a stream for each photosite on a sensor. Yes, resolution is limited, but in a 'cloudy' sort of way as opposed to an on-off sort of way. Maybe wavelets could be used temporally, not spatially. That sort of concept doesn't need 21st century technology to do, but cost is also a question.

There are many ways to go framerateless, including light field derived techniques.

It caused my browser to freeze the OS for a minute.

You need 3D acceleration and a WebGL compatible browser.
A 6-year-old laptop or a 3-year-old cheap budget Android phone will probably just freeze.

It runs at 30-60 frames per second on my newer Dell laptop (less than 1 year old).
It runs at nearly 480 frames per second on a GeForce GTX 1080 Ti Extreme Edition OC on my experimental 480 Hz display.
Right now, it renders at almost 30-60 frames per second on an iPad Air 2 so if you have a faster tablet, give it a spin.
Or try one of your Radeon/GeForce equipped desktop computers instead.

Make sure you read the text near the bottom of the page too (it's informative):

As amazing as this is, we are still in the “Wright Brothers” era of converting video files into true VR scenery. There are still defects; we expect artificial intelligence algorithms to automatically fix defects in these types of videos that are converted to explorable 3D worlds. See other worlds on SketchFab.

In the future, there will likely be artificial-intelligence-based video processing: “AI: That looks like bark texture, let me fix that from what I know about birch and maple trees… That looks like tree canopy above; let me fix that to look real via the leaf-and-branches geometry library… That looks like sky, I’ll put that in a skybox… That looks like a road; let me flatten it out and enhance its texture from the asphalt-texture library… That looks like a historic rock wall; I learned a similar texture from my previous cloud-learning, let me fix the wall to look real… Viola, the video is now intelligently up-converted to retina-resolution 3D geometry and should now look much better!…”

Within our lifetimes, low-resolution 360 degree video files could be upconverted to retina-resolution Holodeck-like exploration via a VR headset. Imagine visiting a vacation spot via “Google Streetview” in true retina-resolution explorable 3D worlds in a virtual tour.

This will be wonderful for future video games, game developers, being able to scan a city into a 3D game, and graphics artists fancying-up the world even further with new artwork to turn an existing city retro-style or future-style, etc.

So you see, eventually -- when this plays a role -- it stops being artifacty like today's "3D scanning" from a camera.

And finally:

Oh well! But with light field cameras, that sort of thing will be more useful. Nobody wants to waste brain power on trying to convert 2D into 3D environments. It's retarded. Light field actually does solve a problem, although it's still a bit crude.

Yes, light fields may be the best way to go. However, let's consider that I now have a GPU in my computer (GeForce GTX 1080 Ti Extreme Edition OC -- a $1500 graphics card, fastest short of getting Titan series -- albiet now $1700 on NewEgg due to cryptocurrency mining, ugh) that can play almost photorealistic 3D graphics for some material -- such a GPU can -- in some cases -- do graphics better than Toy Story 2 in real time (when playing certain demos). Scale that progress about 40 years onwards, and you may be able to get VR-Turing-Test-passing graphics in a battery-friendly mobile processor (even at a degraded Moore's Law curve). Who knows? I think that's one of the many potential paths that will happen.

The iPhone X now have sub-1-watt GPU that outperforms a 10-year-old 100-watt GPU.

Let's also consider that the sheningians in H.EVC were once considered absolutely totally-bonkers crazy 20 years ago. Stupendously bonkers. The computing power required to handle today's high bitrate 8K H.265 60fps+ video in realtime once exceeded the fastest military/academic supercomputer not too long ago. And I'm talking about a supercomputer in the early 21st century too -- not a 1980 Cray supercomputer. Today's >$1000 graphics cards (ones needed to play 8K H.265 today) already outperform Lawrence Livermore National Laboratory ASCI White of year 2000 -- the world's fastest supercomputer. ~6-12TFLOPS then. Today, a single Volta GPU at 110TFLOPS!

Even the iPhone X would have entered (very high up, roughly #150-200) the Top500 Supercomputer List at the beginning of the 21st century, having a GPU capable of exceeding 120 GFLOPS in well-tuned GPU shader language programming.

We're all essentially defacto carrying pocket supercomputers now. And they're only continuing to get faster and faster. Devices have pervasive H.264 decoders, so when devices get pervasive high-performing GPUs, 3D geometry based playback starts to look attractive when it becomes power-efficient and bandwidth-efficient compared to the alternative retina 6dof options. Phones are being used as VR headsets, I was impressed how good the Qualcomm demo progressed at CES 2018 -- it was 100x better than Google Cardboard, and almost as good as Oculus Rift / HTC Vive, but all in a self-contained smartphone headset. Project this progress forward, and I can see why GPU-based video codec may be a pretty attractive path eventually.

So let's not count GPU-playback out of the future video equation. Also, there are incredible inventions occuring in GPUs that raises efficiency-per-pixel; NVIDIA recently figured out how to essentially triple virtual reality framerates without increasing transistor count much. Optimizations are rapidly iterating as we speak, and 3D rendering may migrate to a different pipeline (e.g. realtime beamtracing instead of polygons) but the 3D geometry approach of video files still remains. Video-as-3D-geometry doesn't have to be all-parallax perfect -- just enough to cover full flawless-parallax 6dof from where the human head is -- and some minor parallax offsets (and if need be, AI interpolation can play a role in autocompleting parallax flaws).

Essentially, limited-range-of-motion 3D geometry file for full 6dof at better compression ratios. And if you ever need more geometry/parallax, AI can autocomplete for you (possibly with the help of going back to capture another camera angle to autocomplete missing parallax via an AI-merge of two 3D geometry video files).

There's many parallel tech progresses going on, of which any of them, may lead to the completion of the framerateless video acquisition chain -- including light fields -- instead of the other progresses. Another potential progress is realtime denoised beamtracing as a playback of timecoded (batches of) photons.

Or a combination of all the above. If light fields doesn't make it, eventually GPU-playback could. Or vice-versa. Another technique may (e.g. Timecoded photons/photon clusters. Or traditional framerate-based files as a stand-in for flexible framerate playback, just finer granularity -- e.g. commoditized 1/960sec granularity per pixel that creates 24/60/120/etc playback). Either way. All simultaneous progressions. One of the horses will reach the necessary temporal precision first.

You may very well be right.
But let's not bet the specific horses yet -- we just know some of the horses will cross the finish line in our grandkid's lives (or earlier, give or take).

 

[Mark Rejhon]

More recent amateur-league UltraHFR experiments are occuring between a few of us.

Test System Load for 720p240fps H.264 .MP4 video files at 100 megabits/sec on i7-7740X with GTX 1080Ti
- CPU usage 18% (via Task Manager)
- GPU usage 4% (via GPU-Z from TechPowerUp)
- Predicted capability: 1080p960fps or 1440p960fps UltraHFR (to be tested this year)

240fps Players with perfect framepacing with UltraHFR files on 240Hz displays (H.264)
- Windows Media Player
- MPV player
- Chrome browser embedded player

480fps Players with perfect framepacing with UltraHFR files on 480Hz displays (H.264)
- MPV player

Failures
- Fail 480Hz+ Windows Media Player refuses to play 480fps files completely (thinks they're invalid files)
- Fail 480Hz+ Chrome embedded player microstutters too much (not 1:1 puldown) at 480Hz and up UltraHFR
- Fail 240Hz+ FireFox embedded player microstutters too much (not 1:1 pulldown) at any UltraHFR framerate or Hz
- Fail 240Hz+ VLC microstutters too much (not 1:1 pulldown) at any UltraHFR framerate or Hz

NOTE: We have found frame pacing precision is usually the video player's author fault. Sub-millisecond-accurate framepacing is absolutely essential to round-off played frames to the refresh cycles correctly. Millisecond-accurate framepacing is too low-resolution for 480Hz (frames are only 2ms, so 1ms rounding errors will often miss refresh cycles and create random-pulldown like 0:1:2:0:1:2:0:1:2 instead of correct 1:1:1:1:1:1:1:1 perfect frame:Hz sync). The conclusion is PLEASE use things like RTDSC or QueryPerformanceCounter() or std::chrono::high_resolution_clock::now() instead of old fashioned millisecond clocks, for software developers writing open source video players. Submit appropriate BugZilla ticket reports where applicable.

240Hz displays: Current true-240Hz monitors that worked with UltraHFR files (google "240Hz monitor list").
- Acer Predator XB251HQT
- Acer Predator XB252Q
- Acer Predator XB272
- Acer Predator XF270HA
- Acer Predator XF250Q
- AOC AGON AG251FG
- AOC AGON AG251FZ
- ASUS ROG PG258Q
- ASUS ROG Strix XG258Q
- BenQ ZOWIE XL2540
- BenQ ZOWIE XL2546
- BenQ ZOWIE XL2740
- Dell Alienware AW2518H
- Dell Alienware AW2518Hf
- LG 27GK750F
- Viewsonic XG2530
- Viewsonic XG2560

480Hz displays: Current true-480Hz monitors:
- Zisworks 4K120 www.zisworks.com/shop -- Supports 4K 120Hz, 1080p 240Hz, 720p 360Hz and 540p 480Hz
- A little under $1000 for the preassembled engineering sample, must supply your own VESA mount (~$20 from Staples or Monoprice)

Video workflows used
- Blender works with UltraHFR!! It can speed up videos to 240fps and 480fps. Hallelujah!!!
- Slomo cameras sped-up to realtime using instructions here
- Very few compatible workflows yet. For gameplay videos, we use an automatic module that "screenshot-every-frame" (using SSD or SSD RAID) -- have created true 480fps gameplay video files using Half Life engined games. Samsung 960 Pro M.2 SSD seems to have no problems keeping up with 480 screenshots per second (960 files written to disk per second).

 

[Mark Rejhon]

Today, presenting a new article derived from my old August 2017 thread.



Frame Rate Amplification Technologies (FRAT)
More Frame Rate With Better Graphics

Today on April 4th, Oculus launched Asynchronous Space Warp 2.0, which is a newer method of raising frame rate, by laglessly converting 45 frames per second to 90 frames per second for Oculus Rift Virtual Reality headsets, which is perfect timing for this Blur Busters featured article.

In the future, instead of low-detail 240fps, one can amplify high-detail 60fps into high-detail 240fps, allowing you to have cake and eat it too on your gaming monitor! This is also going to become a key important technology for future 1000Hz gaming monitors, since one could use a Frame Rate Amplification Technology (FRAT) to convert 100fps or 200fps into 1000fps someday in the future (e.g. by 2030).

Although this is not necessarily desirable for video, this article is well within scope because it's a technology that unlocks commercialization of ultra-high refresh rate displays.

 

[Mark Rejhon]

Also, I updated the UltraHFR article to include some new best-practices:

Diminishing Curve: Next Step After 120fps HFR should be 1000fps UltraHFR

Time-wise, the difference between 120fps HFR versus 1000fps HFR is roughly as big as the difference between 60fps and 120fps. This is because 1/60sec and 1/120sec is an 8.3ms difference, while the difference between 1/120sec and 1/1000sec is a 7.3ms difference. There is a curve of diminishing points of returns, but the massive jump up to 1000fps HFR greatly compensates.

1/60sec = 16.7ms blur
1/120sec = 8.33ms blur — observe 8.3ms improvement over 1/60sec
1/1000sec = 1ms blur — observe 7.3ms improvement over 1/120sec

As a result, a large jump up the diminishing curve of returns is required to get as dramatic improvement as the jump between 60fps and 120fps HFR. Therefore we believe that the next step after 120fps HFR should now be 1000fps HFR.
Also, UltraHFR benefits higher resolution more. 1000fps UltraHFR would not be nearly as useful for standard-definition video material (less than 1 pixel of motion blur for one-screen-width-per-second motion) but would be hugely beneficial to 8K material (8 pixels of motion blur for one-screen-width-per-second).

Very Important UltraHFR Tips

In the world of experiments with 240fps, 480fps and 1000fps real-time video playback on ultra-high Hz displays, these are very important considerations from recent research:

1. Display persistence is additive to camera persistence (shutter).
   A 1/60sec persistence display combined with a 1/60sec shutter video, generates a combined 2/60sec of human-perceived motion blur. As a rule of thumb, your common perceived blur is often at least twice the camera shutter, and often much worse (e.g. 1/1000sec shutter combined with 1/60sec display persistence). This means that on most displays, a 1/1000sec shutter at 60fps only has barely half the motion blur of 1/60sec shutter, simply because it is bottlenecked by 1/60sec display persistence! Also, a display that has a pixel response (GtG) that is a significant percentage of a refresh cycle at its max Hz — will also limit UltraHFR benefits even further.
   You must be aware of how both source (Camera) and destination (Display) persistence combines in motion blur in order to understand limitations to UltraHFR benefits
   
   
2. Resolution reduction is a big problem at higher frame rates on current cameras.
   Recommended is to oversample by recording at about 2x resolution (e.g. true 4K) and then downconvert (e.g. to 1080p). For this reason, it is hugely preferable to get a superior high speed camera instead of smartphone camera (e.g. iPhone in Slo-mo) when generating source footage.
   You must minimize resolution loss to preserve UltraHFR benefits
   
   
3. Compression is a big blur problem. Use less compression
   You must minimize compression to preserve UltraHFR benefits
   
   
4. Lens autofocus is sometimes difficult during filming moving imagery.
   It is especially amplified when your motion clarity is extremely sharp, to the point where any camera focus issues begin to become visible.
   You must make sure focus is sharp during fast motion to preserve UltraHFR benefits
   
   
5. Be aware of the vicious cycle effect where higher resolutions demands higher refresh rates to eliminate resolution limitations.
   The difference between static images and moving images become much more pronounced at 8K than it is for 1080p, since motion blur is time-based (e.g. same amount of blur per inch, affects more pixels on higher DPI displays). This is especially hard because current camera technology often goes lower resolution at higher frame rates.
   You must use a higher resolution to preserve UltraHFR benefits
   
   
6. High frame rates benefits high motion material.
   A great example is sportsfield panning, downhill skiing, BMX racing, car racing, etc. When the first HDTVs arrived, some camera operators reported they had to intentionally slow down panning and operate slightly differently because LCD HDTV blur (of the first flat panel HDTVs) was much worse than old CRT motion clarity.
   UltraHFR benefits fast motion material very significantly.

Many historical frame rate tests about frame rate benefits have often neglected to consider (1) (2) (3) (4) (5) (6) simultaneously.

Research indicates that any one of the above can be a weak link in eliminating the benefits of UltraHFR video, and one must carefully control the variables to give truly human-visible benefits of UltraHFR. If you are planning any experiments in UltraHFR, it is very important to know and understand the above variables.