Resolution vs Field Of View for VR

[Craig Parkes]

Just as a general discussion, was wondering if anyone was wanting to talk real world resolution for angle of view requirements and whether there are cost effective options for VR projects given that the HTC, Playstation VR and Occulus are now in the hands of consumers and that EPIC 8K in both Helium 35 and Dragon Vista Vision formats are in the wild (And Alexa65 is a technical possibility).

The reason I ask is trying to figure out whether there are potentially cost effective measures to shoot 360 mono environments for VR (or even say mostly forward facing 360 stereo environments for VR) using very wide angle fish eye lenses on large resolution cameras rather than using multicamera arrays.

As an example - say I have a lenses that can cover roughly 150 degrees of horizontal view and about 85 degrees of vertical view on The Dragon vista vision sensor - that gives me roughly 54 pixels per degree of vision or which is 6K resolution for the 110 degree field of view of the two major VR headsets (which then display that resolution at 1080) - so more than enough resolution, and with one camera I get another 20 degrees of viewable area to the left and right.

Two cameras set up as a stereo pair and maybe this would be enough for a convincing experience in a setup where the rest of the 360 degrees is generated by CGI - for example you're in a CG car but viewing footage of a real world racing track (side windows might be mostly blacked out - your field of view really only incorporates where the windscreen would be and maybe rear view mirrors that are rigged in place of where the rest they will be in the CGI model).

Anyone think this sort of setup would hold up as an worthy experiment? I think the decreased vertical field of view wouldn't necessarily be an issue for immersion if the car is CGI and you've got resolution to spare without the need for a full 360 rig.

Also is there any reason larger sensors might make with wider angle of view may make for better or worse experience than more small cameras with a smaller field of view once you remove resolution and latitude from the equation.

Given that there are an increasing number of mono 360 cameras and stereo 360 cameras reaching the market with reasonable resolution wide angle lenses on very small sensors, other than the quality of lens and latitude/recording format of the camera is their any inherent flaw/advantage in looking into fish eye lenses that can cover a very large sensor over a smaller sensor - apart from the obvious ones of size and cost?

I know there are seemingly perceptual differences between high resolution medium format shooting (or even plate shooting) and trying to capture the same shot with a smaller sensor size but same/similar field of view - but is all this difference just down to glass, or is there something about capturing over a larger surface area that might more or less accurately reflect out perception of the world?

 

[Ben Scott]

I came away from the recent 360 job I did thinking the 4 x GoPros yielding 16k for a 270deg viewing angle was nowhere near enough resolution. This might be a case of quality and DR as well though because it was very undrwhelming. For a car experience I think 2 x Dragons at 6K with the intention of giving you around 200-220degs of vision would be grand.

 

[Craig Parkes]

Hi Ben, 16k resolution for 270 is a slightly higher pixel density than 6K for 110 degrees and obviously much greater than the highest deliverable resolution of 1080 for 110 degrees on rift or vive so I can't imagine capture resolution being the big hurdle with your setup.

How was the project delivered?

 

[Todd T]

The presentation of VR material is directly affected by the display. Theoretically, to get maximum effective resolution using a 1080p display for 360 delivery, you should need just slightly less than a 6.5K x 3.5K source image using google cardboard, or similar phone based VR goggles.

I'm just calculating by taking into account that the fovea (the area of the retina where visual acuity is highest) has an approximately 55 degree field of view, so you would need about 6.5 times the resolution of your display to present 360 degrees at full resolution horizontally, and 3.25 times the vertical resolution of your display for maximum vertical resolution. Using goggles with dedicated 1080p screen for each eye, you would need 13Kx3.5K, otherwise phone based VR goggle use half the screen per eye so 6.5Kx3.5K. There a things i'm probably not taking into account, it would be nice to hear some other opinions or calculations to compare.

Ben, what kind of specs did you use for your rendered footage, and what did you view it with?

 

[Ben Scott]

Hi Ben, 16k resolution for 270 is a slightly higher pixel density than 6K for 110 degrees and obviously much greater than the highest deliverable resolution of 1080 for 110 degrees on rift or vive so I can't imagine capture resolution being the big hurdle with your setup.

How was the project delivered?

Ha, no it was probably the fact they were GoPro pixels and not Red pixels, plus it's very hard to light the scene when everything is in shot. This was a crime scene reconstruction. In truth it was 16k for 360 and they were going to cut a wedge out and make it 270 to hide the steadicam we had the camera unit on - so in reality probably about 15k for 270, but your point still stands.

Project yet to be released but will be on the BBC Website and I assume Youtube as well, accompanying one of their crime dramas.

 

[paulherrin]

i would add a little 4k gopro on the back for the rear-view mirror

 

[Mike Finegan]

3d pairs covering the 360 deg space is more interesting - then one issue is how close can the cameras get
Nothing 24 Weapons couldn't handle LOL

 

[Craig Parkes]

The problem with 3D pairs covering the 360 degree space is the massive amount of post work required, and I believe the fact that as you pan your head around the scene various issues crop up. For such projects I genuinely believe either some of the proposed light field solutions OR simply immersive CGI is far more appropriate.

I am really more trying to consider a cost effective scenario for a relatively narrow group of uses.

Ultimately entirely rendered environments or Augmented Reality is going to be the mainstay of most peoples VR experience. If you are trying to depict real life imagery, you are going to be doing so as a gimmick rather than for a mainstream VR audience (incidentally I would include VR Porn in this gimmick category) and as such you want your product to be the most convincing in a cost effective price bracket. Bespoke heavily post produced work is fine as a tech demo and cool if paid for as such - but keeping a pipeline to efficiently shoot, post and deliver 360 live action stereoscopic 3D an the basis of producing and delivering that content to market is a complete non starter.

 

[Wayne Morellini]

Craig, I'll skip the posts tonight. 8k as a higher end of human vision, at about 60 degree field of view. So, this s at least 24k 180 degree around, 48k 360. I might be off, I can't rember all the working out I did on this, so it might be above 64k. Halve that to 4k, and many people should be happy. But if they have bad vision halve it again, or less. So, 2k would be 12k 360.

However, let's examine the old fish eyed lens suriund caoture, as an example how things get mashed up, say one sensor capturing a good field of view. The warped kens image goes on the flat square sebsur, and has to be dewarped. Now the pixel boundaries don't match up and mix. So you could go lower resolutiin and find a solution that upresses and dewarps with good quality.

Ironic, I was going to do a thread on this resolution issue last night it the night before, but decided against it because some ignoramus would come along and wreck it, making it not worth it as usual.

 

[Mike Finegan]

actually the approximate state of the art (jaunt, facebook 360) are providing stereo views - so at least 12 cameras (2 side by side on each plane at say 8K wide angle).
The physical size of the RED brains would be limiting factor on how close focus would be able to get to the camera rig. Not to mention pricey and huge data volumes
But I am sure if you send 12 weapon or Helium 8K to Radiant Grid they can create it

 

[Blair S. Paulsen]

actually the approximate state of the art (jaunt, facebook 360) are providing stereo views (snip)

Not sure I'd call those rigs "state of the art"...

Cheers - #19

 

[Gavin Greenwalt]

VR movies are stupid. If it isn't explorable IMO it's a novelty-gag that won't go anywhere except as a way for production companies to milk some cash out of agencies who want to look like they're "cutting edge". The only value of a VR movie is the press release and headlines that go around: "_____ wowed SXSW visitors with a breathtaking VR experience!"

The optimal VR movie in my opinion is a pair of 8K cameras facing one direction for*an IMAX omnidome type experience.

 

[Todd T]

The distinction between VR and Immersive video keeps getting confused. I too feel that VR implies an interactive and responsive environment; which as filmmakers seem to go against all of the ways we have learned to tell stories. Normally, we control where the camera is pointed, what the viewer sees, and even a particular point or subject that viewer focuses all on at a pace that we set in editing. In a VR environment, much of that control is given to the viewer, which ends up resembling a video game more that a traditional movie. For that matter, game companies have essentially been creating "VR movies" for several years and we could look there for inspiration. The issue with VR is that it is extremely "intimate"; each viewer controls their own point of view, so it does not work as well for group viewing. This is more of a "stay at home" pay-per-view type delivery and it doesn't make sense for a movie theater.

Immersive video, on the other hand, can be used in a way that is similar to the way we currently create our stories and control the viewer's experience. As Gavin mentioned, this is ideal for an Imax omnidome presentation or the Barco Escape theater format; it fills the viewers field of view without being encumbered by VR goggles or similar hardware and so can easily scale from a single viewer to thousands of simultaneous viewers. A filmmakers, we will need to adapt to this new, expansive, field of view that that is now accessible to our viewers. It will be interesting to see the new techniques we create and tools that we develop to deliver our stories in this new medium.

I also see 360/immersive video being effective in informational videos (Products, services, etc.) and instructional videos, as it allows the viewer much greater control overviewing the information that is presented to them.

 

[PatrickFaith]

The problem with 3D pairs covering the 360 degree space is the massive amount of post work required, and I believe the fact that as you pan your head around the scene various issues crop up. For such projects I genuinely believe either some of the proposed light field solutions OR simply immersive CGI is far more appropriate.

I am really more trying to consider a cost effective scenario for a relatively narrow group of uses.

Ultimately entirely rendered environments or Augmented Reality is going to be the mainstay of most peoples VR experience. If you are trying to depict real life imagery, you are going to be doing so as a gimmick rather than for a mainstream VR audience (incidentally I would include VR Porn in this gimmick category) and as such you want your product to be the most convincing in a cost effective price bracket. Bespoke heavily post produced work is fine as a tech demo and cool if paid for as such - but keeping a pipeline to efficiently shoot, post and deliver 360 live action stereoscopic 3D an the basis of producing and delivering that content to market is a complete non starter.

I think the guys at hyperVR really have their act together (article here from roadtovr.com): http://www.roadtovr.com/believe-hype-hypevrs-volumentric-video-capture-glimpse-future-vr-video/

They have a super nice RED setup, but I'm doing something similar but way less costly - basically build up a static volumetric scene with a single camera then projection map a single camera footage via projections onto that volumetric surface. A cheap way to build a volumetric view is just take a 6k red and moving it around (side ways, then rotating ) and nukex will build a volumetric "static" 3d space(syntheyes can also do this, as well as a few other programs). It takes a bit of CPU to do this, but your only using one or two camera's. But the main limiting problem with this less costly approach is the cinematographer has to be lock step in line with the story & vfx for it to work.

 

[Craig Parkes]

VR movies are stupid. If it isn't explorable IMO it's a novelty-gag that won't go anywhere except as a way for production companies to milk some cash out of agencies who want to look like they're "cutting edge". The only value of a VR movie is the press release and headlines that go around: "_____ wowed SXSW visitors with a breathtaking VR experience!"

The optimal VR movie in my opinion is a pair of 8K cameras facing one direction for*an IMAX omnidome type experience.

Hi Gavin, in general I'm in agreement with this. Part of my question is can this experience be cost effectively delivered to home viewers using the current state of VR tech and how would that best work.

As a concept, imagine a VR environment where the forward facing arc is a 180 dome, and the person in the VR environment is sitting in cgi seat in a scene designed to replicate the distance and scale from the screen of an IMAX omnidome type experience, the rest of the environment is a darkened CGI void/volume type space with some themed points of interest (or for those who want a 'real world' experienece a CGI representation of an IMAX type cinema)

Currently consumer CR headsets are delivering approximately 1080 horizontal resolution per eye for a targetted 110 degrees of vision - so target this as a minimum. The question becomes how many degrees of head panning is required for a person to see the edge of the volume. If we were to put them in the literal center of the 360 scene then in VR it would be 35 degrees of head panning - and the Projected image would require more than 1780 resolution per eye for pixels not to be visible at 1:1 ratio with the VR screens pixels. Let's double that to get us some safety, we are talking a 3.5K projected image, the thing being it's required in both the horizontal and vertical.

Theoretically with both Helium and VistaVision this can now be captured with a single camera, as these cameras have a 4K vertical resolution. So now the question becomes what glass can we use?

For helium we need a lens that will project a roughly 15.77mm image circle with a 180 degree field of view. So maybe a fish eye Micro 4/3 lenses on a Helium sensor would be pretty close to this target (we lose a couple of mm on the vertical, but maybe we don't need quite a full 180 screen).

With Vista Vision we have 21mm vertical so a fish eye APS-C lens (designed for a a 22.2 mm horizontal) would be pretty much perfect.

So one question becomes can we mount Micro 4/3 on a helium or APS-C lenses on Vista vision cost effectively.

 

[Mike Finegan]

Not sure I'd call those rigs "state of the art"...

Cheers - #19

ha ha agreed - maybe I should have said 'state of that art' versus 'state of the art'
but basically 3D stereo where you pick the view
have only really seen telepresence / immersive game examples
Anyone see good theatrical content in VR yet?

 

[Mike Finegan]

volumetric capture sounds great - especially if you have static set captured independently from moving actors and objects, but the data volumes!!!

 

[Todd T]

Here is one such example of capturing the set to use as a virtual model:

It would be interesting if someone could talk them into creating a VR version of this video, especially since their assets are already 3D models.

EDIT: Actually, someone did do a VR version

 

[PatrickFaith]

volumetric capture sounds great - especially if you have static set captured independently from moving actors and objects, but the data volumes!!!

Unity3d has volumetric to spherical projection and spherical to volumetric projection, it can even blend multiple projections based on camera position. This gives a huge reduction in total file size (100x reduction).

Here is one such example of capturing the set to use as a virtual model:
...

It would be interesting if someone could talk them into creating a VR version of this video, especially since their assets are already 3D models.

I was watching a old Bob Hope film and it had a ton of projection mapping and geometry tricks (the "Road of" series). To me the whole trick to this is totally understanding the technology when setting up the shot, for example mapping this all in unity3d isn't all that hard but requires all the assets to be built and/or shot a specific way. The challenge to cinematographers is to get up to snuff on the math/geometry for doing complex projections (even though these basic math & filming techniques are fairly old, last 20 years the work has leaned too far into the VFX post side rather then production).

 

[Wayne Morellini]

The distinction between VR and Immersive video keeps getting confused. I too feel that VR implies an interactive and responsive environment; which as filmmakers seem to go against all of the ways we have learned to tell stories. Normally, we control where the camera is pointed, what the viewer sees, and even a particular point or subject that viewer focuses all on at a pace that we set in editing. In a VR environment, much of that control is given to the viewer, which ends up resembling a video game more that a traditional movie. For that matter, game companies have essentially been creating "VR movies" for several years and we could look there for inspiration. The issue with VR is that it is extremely "intimate"; each viewer controls their own point of view, so it does not work as well for group viewing. This is more of a "stay at home" pay-per-view type delivery and it doesn't make sense for a movie theater.

Immersive video, on the other hand, can be used in a way that is similar to the way we currently create our stories and control the viewer's experience. As Gavin mentioned, this is ideal for an Imax omnidome presentation or the Barco Escape theater format; it fills the viewers field of view without being encumbered by VR goggles or similar hardware and so can easily scale from a single viewer to thousands of simultaneous viewers. A filmmakers, we will need to adapt to this new, expansive, field of view that that is now accessible to our viewers. It will be interesting to see the new techniques we create and tools that we develop to deliver our stories in this new medium.

I also see 360/immersive video being effective in informational videos (Products, services, etc.) and instructional videos, as it allows the viewer much greater control overviewing the information that is presented to them.

Tod, I'll tell you about an.old story based computer adventure game system from the 1980's that could be used, and I suspect some laser disc titles.

The user was put in a progressive story line, they were presented with and told things according to script, and given choices of response, which forked the script of the story line in another direction. I didn't buy any, it made the point of adventure in it seem pointless, adventure for dummies where it is all worked out for you, but it was popular for a time. Such could be used in vr to give a little interactivity. The issue is seemingly fusing the scenes together, so that when you ask the characters seeminglessly move on o the next according to the response. So you can reposition the actors for each option, and change shot to hide it (or other, and you could use volumetric capture to smoothly realign people to their current body alignments, also on the wait scenes for the interaction). But this is still not a movie, more an individual application. Another is a fully interactive 3D environment and actors, like the holodeck. Even less a movie.

So, apart from a group activity where people are guided to what they look at, or can look around independently, what do you guys think a VR movie can be?