
Open-sourcing DeepFocus, an AI-powered system for more realistic VR images - lainon
https://code.fb.com/virtual-reality/deepfocus/
======
JaRail
DeepFocus Overview: [https://www.oculus.com/blog/introducing-deepfocus-the-ai-
ren...](https://www.oculus.com/blog/introducing-deepfocus-the-ai-rendering-
system-powering-half-dome/)

DeepFocus Paper: [https://research.fb.com/publications/deepfocus-siggraph-
asia...](https://research.fb.com/publications/deepfocus-siggraph-asia-2018/)

------
taneq
I really don't understand why they couldn't just use traditional blur
techniques. They say:

> These fast but inaccurate methods of creating “game blur” ran counter to
> Half Dome’s mission, which is to faithfully reproduce the way light falls on
> the human retina.

... but traditional Z-based blur is no less faithful than their overall "whole
screen at single shifting plane of focus depending on gaze" approach. All of
computer graphics is "more to do with cinematography than realism" anyway,
realism is nice but if you have to choose between "looks realistic" and "looks
good" you go for "looks good" every time.

Also, as others have mentioned, getting sufficient resolution for really high
quality VR basically requires foveated rendering, at which point the bits that
you're blurring are, by definition, not what you're looking at (since you're
rendering at lower resolution outside of the fovea) and a blur algorithm that
needs four GPUs to run in realtime is a complete waste of resources.

Edit: Watched the video. Their 'Circle of Confusion' map is literally just
'focus_Z - pixel_Z'. I really don't see what deep learning adds here.

~~~
RobLach
It’s research so it’s good to know how to do it really well so in 10 years you
can throw it in. Chasing photo realism is sisyphisian.

Using this now for anything like a commercial game is way past the point of
diminishing returns. In their paper the SNR between their solution and default
Unity’s blur shader is a couple percent while the computational costs are
extremely steep.

At 512x512 resolution their fast solution adds 3ms to render time of each
frame using, if I’m reading it right, 4 GPUs, which is a drop in 10fps. That’s
mind boggling slow for real-time graphics, especially VR which is even more
resolution and frame rate sensitive.

If instead of using the default Unity dof blur you really focused on tuning a
blur algorithm, which no one really bothers with because it’s not worth the
effort, you could probably make up much of the difference.

But really once you have eye tracking, status quo dof blurring will be more
than good enough.

The case they mention where someone would be in VR all day makes sense, but
let’s all hope we never reach that dystopic vision.

------
wincy
This means Oculus has this technology working. Which means foveated VR
headsets are coming soon.

~~~
kranner
This seems unrelated to foveated rendering unless I'm missing something.
Foveated rendering reduces the complexity of rendering in areas outside what
the fovea is looking at, because they are blurry to us anyway. This adds blur
to out-of-focus objects; but it seems obvious that in order to appreciate the
effect you'd have to be looking at them via your fovea.

~~~
russdill
The two are related. In order to infer the user's focal length, you need to
know what they are looking at. Which requires eye tracking, which is also a
requirement of foveated rendering

------
dharma1
What's a "DeepFocus enabled headset"? Presumably eye/focus tracking?

~~~
acrim
Yes. My reading is it would be a headset with eye tracking - such as the half
dome prototype.

We also don’t know how much hardware assistance these headsets will have. So I
it could additionally refer to a headset with the ability to process or assist
with this technique running on the hardware itself. This seems likely given
the speed of eye movements and the need to reduce latency to generally
imperceptible levels.

