
The Problem With Photorealism - blackhole
http://blackhole12.blogspot.com/2014/03/the-problem-with-photorealism.html
======
beloch
It's often been said that the difference between Steven Spielberg and Ed Wood
is merely thousands of subtleties. Tiny things add up to either make or break
created worlds.

Clipping is one of my biggest pet-peeves with current video game engines. It
really destroys the illusion when stuff that looks realistic and solid starts
magically clipping through something else that looks realistic and solid. How
about when the camera passes too close to something and you see the hollow
insides? This is especially jarring when it happens to be a character's head
and you see the backside of his/her eyeballs and teeth! Pick any AAA title
randomly, make a character, and then look at that character closely while
standing in his/her default start-of-the-game gear. The odds are good that
you'll see dozens of little bits of his/her gear clipping into itself while
that character is engaged in the default standing animation. It's telling that
even excellent artists can't solve this problem for the one combination of
gear and poses they know the player will see with 100% certainty.

The next big step for computer game realism is going to be material physics.
Simulated materials instead of textured polygons and bounding boxes. This is
going to be computationally intensive, but the boost to realism is probably
going to be far greater than we'll gain from ray-tracing.

~~~
angersock
"Simulated materials instead of textured polygons and bounding boxes."

What does this mean? Like, what do you envision this changing?

~~~
beloch
Think about how a game treats a block of stone right now. It's a hollow cube
with textures pasted on it. It's bounding box is pretty easy to calculate, but
a complex object that doesn't match it's own bounding box closely will still
clip right through it.

Now, imagine if we replaced that single block of stone with a matrix of tiny
blocks. Also, imagine that we've made the bounding box of the complex object
fit its geometry much more closely. If the same motion that caused clipping
before is carried out, we'll detect it. It's going to take a _lot_ more
computation, but we will detect it.

Not only will we detect collisions that we wouldn't have before, we'll be able
to handle them in a variety of gameplay enhancing ways. If the stone is meant
to be strong stuff, we make it's mass high and the bonds between sub-blocks
strong so that the complex object will come to a jarring halt when it makes
contact instead of passing into the stone. If the stone is meant to be softer,
we dial down the bond strength between the sub-blocks a bit so that they can
be broken. The object might dig in a bit and leave a noticeable gouge as some
of the sub-blocks are separated from the main block. If the stone is meant to
be more like dirt, we make the bond strength very low so that a blow will
cause a chain-reaction that will cause the whole block to tumble apart. We can
change the probability of blocks sticking together to simulate either wet,
clumpy earth or dust that's ready to fly off in all directions when it's
touched. Deforming the complex object is also an option. It could be similarly
be composed of sub-blocks, but with tweakable levels of
stickiness/stretchiness holding them together that would allow us to capture
things like flexing or bending.

In short, we really oversimplify the physical nature of objects in video games
at present. This greatly reduces computation, but it makes the world seem
unrealistic in how objects interact. By splitting things into smaller chunks
that more closely resemble real matter we can start to simulate real
materials, but at the cost of greater computation that isn't devoted directly
towards making things pretty. Prying resources away from the graphics engine
has long been something developers just don't want to do.

~~~
angersock
So, that's cool and all, but we aren't making simulations-- _we 're making
games_.

Nobody gives a hoot in hell how realistic the drapes on a window are if the
game isn't fun.

Many games have gone for very good realism, and it hasn't really worked out so
well.

Doom 3 had per-poly hit detection, and all it did was make many weapons feel
less accurate and multiplayer more annoying.

ArmA2, one of the most detailed military sims out there, did very accurate
character-world collision detection and as a side-effect actually moving
around inside buildings became quite hard. A community mod was created to
address that and restore the funner, less realistic behavior.

Red Faction Guerilla tried to make more realistic destructible physics, but it
mostly just made more trouble for level designers, because things they built
tended to fall apart.

Again, simulations aren't games.

~~~
loup-vaillant
Jarring details _do_ spoil the fun a little. I'm fine with lack of realism. I
am _not_ fine with noticeable _differences_ in realism.

My last example comes from _Rembember Me_ , which I have played on the PS3. An
excellent game, with very good graphics… except for self shadows: they
emphasise the underlying vertices. I'd rather have no shadow and a smooth
face, than a "realistic" shadow and a blocky face. Other titles who feature
self shadow display a horrible dithering (I have seen this in Mass Effect).

To date, the best consistency I have ever seen comes from Dragon Quest on the
PS2. It's not realist at all (its style is Japanese motion picture), but
there's next to no mistakes such as clipping. Even more recent titles such as
Ni no Kuni aren't as consistent.

------
robmcm
This guy isn't thinking outside the box. He's trying to solve the problem
using techniques from the past.

Obviously the solution isn't to manually make levels, it would be to automate
them. Where are EA on flying scanning drones around football stadia in order
to save artists thousands of man hours?

They already use these techniques to scan actors faces, I can imagine someone
from 10 years ago predicting it would be impossible to script all the
movements of lips and eyelids to produce realistic characters.

Disclaimer: I have no idea what I am talking about, but thought my off the
cuff theory would have more weight if I wrote like I did.

~~~
chongli
Scanning and other forms of automatic content generation do not replace manual
3D modeling and texturing; just as cameras did not replace painting and
sculpting.

Personally, I'm more excited about procedural generation than any of these
other ideas. Programming seems to be the most powerful tool humans have ever
devised. Why not apply it to create art?

~~~
MartinCron
If you could use automated scanning (or other means of observing the real
world) combined with procedural content generation, that could be huge.

Buildings that look and feel like buildings because they were procedurally
generated with a deep understanding of how buildings actually exist? Yes,
please!

~~~
PeterisP
Usually you want to deliberately distort the real world significantly, to
reduce the distances by an order of magnitude while keeping the 'feel'.

You don't a GTA-style game to take 40 minutes to commute from one place to
another; so you want to keep the noticeable landmarks but cut out huge parts
in the middle. You want wild alley chases to be possible, so you make many
places connected and cross-navigable in patterns that don't occur in real
life. You want to drastically increase density of 'interesting' places, not
mirror their density in real life; and leave out endlessly repeating miles of
similar suburbs or apartment blocks.

~~~
robmcm
So this is where you spend the man hours, tweaking a generated or autonomously
created world.

------
zeidrich
Just simulate the entire universe. It requires a computer the size of the
universe.

There is one freely available, however, you will have to share it with a few
other people. It's barely noticeable though because it's highly
parallelizable.

~~~
guizzy
It's rather poorly documented, though.

~~~
mikeash
And extremely buggy.

~~~
Houshalter
It's pretty bad. The other day _an entire plane_ just disappeared. And I have
yet to see someone respawn properly.

~~~
mikeash
I saw the greatest fake release notes entry the other day: "Fixed bug where
matter would sometimes spontaneously become self aware."

------
PhasmaFelis
Minecraft is an interesting case study here, in that it deliberately throws
away realistic graphics in favor of a comprehensively mutable environment.
It's interesting how that seemed like such a totally unprecedented idea at the
time.

~~~
AJ007
It wasn't unprecedented, just look at Dwarf Fortress. Tarn threw out all
graphics in exchange for focusing on an incredibly complex and deeply layered
game -- with an environment that is fully constructable and has both a
geological and cultural history.

Once the 3D accelerator become an item all gamers had, (perceptually) it
appeared all major game develop shifted to producing "photorealistic"
graphics. We ended up with an endless trail of games that basically produced
the same thing.

I wonder, what would happen if the budget of something like Call of Duty was
spent on a game with Minecraft or even Ascii graphics? Perhaps a total
disaster, but what could be produced if it was done really well?

~~~
dasil003
The current state of Dwarf Fortress probably is within an order of magnitude
of the number man hours in it as an early Call of Duty. The difference is that
the work on Call of Duty is mostly art which is trivially parallelizable,
whereas the work on Dwarf Fortress is the engine which is not parallelizable
to anywhere near the same degree.

------
coldcode
Having worked on a FPS MMO, I can't agree with this enough. People always
complained it didn't look real enough. I always wanted to ask them if would
mind buying 1,000 video cards up front and then wait 10 years buying 1000 new
ones every year until I got it right. I figured by then I could retire and
duck the problem. Reality is ridiculously hard to do right. Now try doing it
over the internet with a half second ping time and you realize that not only
reality bites, but physics bites harder.

~~~
chiph
While WoW looks the way it does partly because of it's 10 year old origins,
it's artwork also communicates to the player: "It's just a game". And once
you're far enough away from an imitation of reality, the brain stops trying to
fit it into it's model of the world and complaining when it can't.

------
Geee
I don't know if you guys know how fast path-tracing is becoming reality. It
just needs a few more years (of GPU advancement) to get rid of the noise,
watch:
[https://www.youtube.com/watch?v=aKqxonOrl4Q](https://www.youtube.com/watch?v=aKqxonOrl4Q)

For art, next-generation photorealistic graphics will be 3D scanned and/or
procedurally generated. Lots of work still (you can't scan cars or anything
mechanical), but graphics could also be reused more often.

~~~
ilaksh
Yeah this is exactly what I was going to say and I was even about to post a
Brigade engine video. LOL.

McClure kind of dismisses "ray tracing" offhand (and by implication/context,
similar techniques like path tracing) with the premise that every texture
would need to be manual developed by an artist.

The answer to that is as you mention 3d scanning and procedural generation.

There are quite a few path-tracing software (and even one or maybe a few
hardware) efforts out there, and many 3d scanning companies.

I believe that the main thing that is holding these technologies back now is
just people not knowing that they are realistic technologies, which keeps them
from being mainstream. But once things enter the mainstream consciousness of
engineers, you get an order of magnitude increase in the number of people
working on them, starting with some of the existing working ideas and you
start getting much more practical and inexpensive solutions.

I believe that within say 7 years Nvidia and ATI will either acquihire or
build hardware themselves that makes real-time path-tracing, procedural
generation, and real-time physics, convenient and efficient.

~~~
Geee
By the way, if you are interested in (realtime) ray tracing technologies,
check out this forum: [http://ompf2.com/](http://ompf2.com/)

By procedural/scanned I meant mainly procedural that is based on 3D scanning
(scans used for learning set), such as FaceGen
[http://www.facegen.com/](http://www.facegen.com/) Lots of other stuff could
be made this way, too.

------
bitwize
And that's a big part of why video games suck these days. Creating a
photorealistic or cinematic experience is extremely difficult and requires a
huge amount of artistic assets, and even then the illusion only holds up under
certain tight constraints. So game developers do as much as possible to
enforce these constraints _all throughout the game_.

~~~
lttlrck
Or it's because the games just suck and no amount of photorealism is going to
cover the cracks.

~~~
wmeredith
100 times this. If you want to know how important photorealism is to getting
your subject involved with the medium, go read a paperback copy of Lord of The
Rings. It's pretty lo-fi.

~~~
lotyrin
And this is why I point to Dwarf Fortress as the state of the art in games.

~~~
coldpie
I would love to play a DF with a decent UI, and preferably with simple 2D
graphics. And I'm an ASCII DCSS player.

Are there any hacks out there that make the UI less memorization-focused? I
know there are graphics hacks, but I was never able to get them to work...

~~~
lotyrin
There's tilesets, but honestly I think they detract. My extended ASCII are
like 8x12 pixels and scan really easily. Even at 32x32 I can't reliably tell
the difference between a dog a cat or a rat in most tilesets, especially on
first glance, so a lot of memorization is still required with tiles, they
allow for less information to be displayed at once, and they are more
complicated to recognize.

I don't even see the extended ASCII anymore (
[http://i.imgur.com/7ci106g.png](http://i.imgur.com/7ci106g.png))

------
mattgreenrocks
I noticed the most recent crop of engines (thinking of Frostbite 3 here
mostly) do a lot of cheating to run well. This is most apparent on the awful
draw distance of terrain decoration. Draw-in is very apparent at even the
highest graphics setting.

Worse, despite games like Battlefield having beautiful art assets, it takes
very little for the illusion of photorealism to be destroyed; it could be as
simple as getting too close to a piece of tall grass and realizing it's
actually just a sprite. Or noticing that a flag is always flapping one way,
and smoke is drifting the opposite way.

It's unfortunate, but the brain is really good at picking these
inconsistencies out. More stylized games seem to age much better; TF2 still
looks great.

------
DanielBMarkham
Takeaway: the problem isn't photorealism in games, although that's hard
enough. The problem is actually having somebody create the zillions of points
of detail required to have something to photorealistically render.

So, seems like the technical challenge is to take a small portal, say 64x64
pixels, and then create a procedurally-based photorealistic (including damage,
real-world illumination, and a damage model) engine to power it, right?

~~~
ZenoArrow
> So, seems like the technical challenge is to take a small portal, say 64x64
> pixels, and then create a procedurally-based photorealistic (including
> damage, real-world illumination, and a damage model) engine to power it,
> right?

You may be interested in .kkrieger as a proof of concept for this type of
engine. The most incredible part is the whole game is less than 100 KiB in
size, but then we're talking about coders who are used to working with in 4KiB
limits:
[http://en.wikipedia.org/wiki/.kkrieger](http://en.wikipedia.org/wiki/.kkrieger)

Talking of the demo scene, you may be interested in this 4K demo, another
great demonstration of what procedurally generated content can do:
[http://www.pouet.net/prod.php?which=52938](http://www.pouet.net/prod.php?which=52938)

------
aw3c2
Wasn't there something about people actually prefering non-photorealistic
renderings more, at least in Google Earth?

------
vacri
One of the things missing from 'realistic' games is specular reflection and
friends. Little subtleties like that help. One of my favourite moments in the
realism wars was that phase where everyone had to have 'realistic' water, and
I remember logging into Morrowind for the first time. "The water does look
nice, shimmery and shiny"... but it drew attention to the problem that nothing
else in the game shined at all, it was all dead flat.

------
drakaal
It should be "the problem OF photorealism" not "WITH". We all enjoy
photorealism, but it is hard to make. We have a problem OF 3 times 6. Not a
problem with 3 times 16.

Any how....

I worked on Flight sims in the mid 90's. We had a problem of Physics back
then. So we didn't do real physics, we precalculated the physics based on a
number of factors and loaded them in to tables, and if you were between point
A and Point B in a table we took the weighted average of the two. It worked
well enough.

Animations today do the same thing.

Anti-aliasing is now done based on point of contrast rather than the whole
image. This works well enough, you notice the jaggies on white and black
borders more than 45% grey next to 65% grey.

This doesn't strike me as a well informed post. Others have pointed out that
it is directed at those new to the space, fine, but if that's the audience
then explain "uncanny valley" in there somewhere. Where you can't be "right"
be far enough off people accept it as fake because it is. We see this all the
time in Pixar. We also see it in racing games with "arcade physics".

------
ANTSANTS
I prefer comics and cartoons to live-action movies. It's ironic, but somehow,
a primitive smile or frown on a cartoon character's face feels more genuine
and "real" to me than a professional actor pretending to be happy or sad. I
can take a drawing as the unfiltered expression of the artist, but when I
watch a live-action film and see real people carrying out the trappings of
fiction, I have a hard time taking it seriously. It feels silly, kinda like
how the so called "uncanny valley" effect makes bad CG creepy and not just
unconvincing.

I don't think that's a common opinion, given the lasting appeal of television
and Hollywood, and the lack of respect comics receive in comparison, but in my
mind, if a drawing feels more "real" than an actual human acting in front of a
camera, then photorealistic CG doesn't stand a chance.

~~~
Xixi
On the other hand it might just be a matter of getting used to it.

Japanese live-action movies are a textbook definition of overplayed, and few
years ago I couldn't stand watching them. It was making me physically
uncomfortable. Fast forward a few years and I enjoy watching them...

------
eslachance
About the last point about generating content, I think it's extremely likely
that this leads us down the path of Voxels and Procedural Generation. Both
those techniques together would help us at least get a starting point for an
environment that's generated from certain chosen parameters, and then the
"least effort" becomes modifying that environment for your game needs.

I mean, if you can have a procedurally generated jungle with a huge mountain
in the middle and a river cutting through the jungle, all you need to do
really is to cut through the mountain for your secret underground base, create
a couple of roads, camps and checkpoints and you've got a better remake of
FarCry 1 (yeah I'm extrapolating of course but you get the idea).

------
radiorental
I look at the loading scenes in GTAV and I wish the game looked like that,
rather than the 'uncanny valley' that it is.

Borderlands II is a great example of not trying to do something you cant,
instead it has great art that you can lose yourself in.

------
marcusmolchany
I've always wondered, given all the issues presented in this article and the
expertise required to solve them does $50-$60 dollars properly pay the teams
working on these games? Not even just engineers, there are artists and even
actors capturing motion of characters and authors writing up scripts it
doesn't seem like $60 would cover everyone involved in creating a video game.

~~~
SixSigma
You can buy LOTR on Blu-Ray for about $20.

------
z3phyr
Can someone list the possible technical advancements in gamedev yet to happen?

-> AI programmers are negative towards AI, saying gamers want 'fun' not Intelligence.

-> Graphics programmers say, the graphics will be photorealastic soon. What will graphics programmers do after that?

-> Gameplay, they are focusing more on creating the same style?

------
olouv
Reminded me this "What Is Real" trailer
([https://www.youtube.com/watch?v=4dnw9dWISvw](https://www.youtube.com/watch?v=4dnw9dWISvw)),
I welcome any trick that can help the player's immersion.

------
benched
I am a long-time 3D graphicist, and have written numerous software and
accelerated renderers. I am really struggling to see the point of this article
(it seems trivial and tautological to me), although I'll be glad if it
generates some discussion.

This is an area where I feel very strongly that perception is reality, or at
least so close to it as to make no difference. The game world is virtual, so
the fact that say a rock is not made of zillions of atoms computing quantum
chromodynamics doesn't come as a surprise to anybody. Well, that's a shortcut
right there. Immediately, the game world is not arbitrarily "realistic".

But I don't think that's what the word "photorealism" usually means. I know
I've never used it that way. I use it to mean "feels subjectively real". By
that definition, as the OP alludes to, today's most advanced game engines come
very close. In my opinion, they come so close, that the rest is gravy.

That's what will be coming in the next decade. More gravy, more cake, more
icing. In the next year or so, somebody (Sony? Everquest Landmark?) will start
the inevitable development toward Minecraft-style procedural/cellular automata
based worlds, combined with increasingly photorealistic rendering.

~~~
RyanZAG
_> so the fact that say a rock is not made of zillions of atoms _

But that's the problem. When the player misses the target with a rocket and
hits the rock, it gets a strange looking explosion and then the rock is either
still there, or simply 'removed'. Sense of realism? Lost.

In an open world game, the player might want to pick up the rock. If you took
a shortcut and just made the rock an object that sits on top of the ground
plane, when you pick up the rock there is normal grass underneath. If you
throw the rock at a building, it bounces off and rolls around stupidly. Sense
of realism? Lost.

 _> That's what will be coming in the next decade._

That's what the article is saying - you're deluded into thinking we're close
to solving this stuff. There is no way it's coming in the next decade, it is a
long way off with a huge number of very difficult problems to solve. This is
like the AI debate. AI is always 'just around the corner'. No it isn't, we are
nowhere close to even beginning to understand what we actually need to
accomplish once the tricks are taken away.

~~~
dragontamer
Which is why tons of people play "Minecraft", because the voxel technology and
ability to deform the environment is much more real than many other games.
There are some advancements in this field, but really... at the end of the
day... the best "deformable field" is Minecraft still. And it hardly even
tries to be realistic.

Fortunately, most gamers don't care about realism to this degree. Many FPS
gamers want a tight engine, fun mechanics. Bunny Hopping was left in the Halo
series for this reason. Ditto with super-cancels in Fighting Games (tell me if
MMA-fighters cancels their jabs into super-attacks).

Realism for the sake of realism is hopeless. Game programmers need to focus
upon entertaining the gamer. Some gamers require a degree of realism... but it
should never be the primary point of a game.

~~~
mr_luc
I was wondering when someone was going to mention Minecraft because that's a
great way of illustrating to people _something_ of just how vast the gulf to
be crossed is.

There are graphical mods for Minecraft, but none that bridge the gap from
"cube world" to "our world."

And even Minecraft has artifacts like floating islands and limitless water --
the voxels are meant to represent fairly unchanging substances like dirt and
stone, and even the trunk and leafy volume of a tree, and they support a
limited amount of state being attached to them, but they are not being
computed constantly in a way that would make it possible to implement
something like erosion or a landslide, or realistic water motion, etc. And
when you try to imagine going much beyond minecraft, you realize that, crap,
you'd have to be computing the entire (essentially unlimited) voxel world
maybe several times a second ...

I think it will be done, but there's a good case for it NEEDING to be done in
a massively multiplayer way, because the work of simulating the world becomes
too large to be done by any individual user's machine.

~~~
Crito
I think that the difficulties you would face in trying to make minecraft more
realistic are far greater than the difficulties you would face if you were
starting from scratch. Trying to make minecraft realistic is like trying to
paint over _A Sunday Afternoon on the Island of La Grande Jatte_ to make it
realistic. It just wasn't created with that sort of realism in mind, so you
are working from a disadvantaged position.

~~~
mr_luc
Yeah, I don't mean a literal mod of minecraft. I just mean that it's a great
illustration of what'd be entailed.

If you start from scratch, you still need something like voxels, ie keeping
track of many points in 3d space representing different kinds of materials,
which is what minecraft has now.

But then, graphical or behavioral perspective, Minecraft gives you an idea of
what problems you'll run up against.

Graphically ... holy cow. I don't know of anyone that's doing that, ie taking
voxel data and making it look amazing. I did see a demo that looked a bit less
blocky than minecraft, it wasn't cube-based, but rather used the 3d hexagonal
mesh I think. It looked less blocky of course, but it worked with only grass
and stone material.

Behaviorally .. also, holy cow. I'm not sure if there's anyone doing that in a
3d game engine. Ie, voxel or volume world data representing materials, in a
way similar to minecraft, but if you undercut a bank of dirt, it collapses ...
to do that, you need to be constantly simulating the entire world with CA
rules meant to emulate various physical properties of earth, water etc (not
just the currently loaded world chunks, and not just a few kinds of very
limited material types, like Minecraft's water and lava) probably at least
several ticks per second.

I suppose there might be ways to cheat or to legitimately exclude 'all-quiet'
areas from needing to be updated, in some cases -- like the dirt example, I
can see approaches that would propagate simulation outwards from an area
centered on user-generated changes at the maximum speed that the evolution
rules transmit information, and that might be able to leave large portions of
the world relatively static ... although that wouldn't work so well for things
like fluid simulation ... but the point is that, holy cow, we're a long way
off!

~~~
dragontamer
[https://www.youtube.com/watch?v=Gshc8GMTa1Y](https://www.youtube.com/watch?v=Gshc8GMTa1Y)

Voxel technology is improving, and certainly exists at realistic levels. The
question is... what game _cares_ about such details?

