
How a new emulator generates 3D scenes from 2D NES games - bpierre
http://arstechnica.com/gaming/2016/03/how-a-new-emulator-generates-3d-scenes-from-2d-nes-games/
======
sebringj
Wow, take this idea further with how google lets an AI "dream" with fish and
pigs onto normal everyday scenes, you could allow the AI to visually interpret
a game scene in real time but train it with domain-specific imagery so it's
more fantastic. In other words, you get a scarier Dracula or cooler looking
hero because its from the "imagination". We always say reading a book is more
amazing because you imagine it to be so. Why not with AI doing the imagining
for us?

~~~
sebringj
Let's go even further beyond graphics, imagine a Visual Imagination API that
could take in video as well as textual/voice input. This would enable a
storyteller in front of a huge display, intensely narrating while a living
canvas responds to the words spoken for everyone to see, now imagine this in
VR, living inside your own AI's imagination of your own narration.
Woooooooooooeeeeeeeeee -

~~~
sebringj
OK because you keep up-voting... now imagine an augmented life filter that is
an AI interpreting reality for you in a very rosy and agreeable way. It seems
everyone and everything are pretty much awesome. Your AI output filter is
particularly suave these days. After all, your personality is the best money
can buy.

------
tlrobinson
_This isn 't just a conversion of every pixel into a uniform voxel, either. In
a game like Super Mario Bros., for instance, 3DNES converts pipes to into
cylindrical 3D models_

I actually think voxels might look way cooler than attempting to generate
smooth 3D objects with the original sprites textured mapped on top. It would
retain more of the original character of the game, while adding some depth.

~~~
ZenoArrow
> "I actually think voxels might look way cooler than attempting to generate
> smooth 3D objects with the original sprites textured mapped on top. It would
> retain more of the original character of the game, while adding some depth."

Do you mean like this mock up?

[http://cdn.ubergizmo.com/photos/2008/7/3d-smb.jpg](http://cdn.ubergizmo.com/photos/2008/7/3d-smb.jpg)

~~~
deepnet
Would like to see shadows, they are a great depth cue.

------
jbergknoff
Relevant discussion on the nesdev forum:
[http://forums.nesdev.com/viewtopic.php?f=3&t=13552](http://forums.nesdev.com/viewtopic.php?f=3&t=13552)

------
codemonkeymike
Some NES games used some crafty hacks to get more sprites on screen then
allowed by the GPU and Video Ram, I am curious if they will every be able to
render that without having to make many custom rules for each game.

------
nickpsecurity
I want to see Contra in this one. Particularly the scenes that try to
approximate a FPS.

~~~
brickmort
There's a video in the article that shows Contra in 3d!

~~~
nickpsecurity
Haha didn't even see it. Thanks. Went back and rewatched someone playing it
without the code while I was at it.

[https://www.youtube.com/watch?v=wcuAYSN-
CvY](https://www.youtube.com/watch?v=wcuAYSN-CvY)

------
empressplay
_and can load arbitrary (and definitely legally obtained) ROMs from the
cloud._

I laughed.

------
daodedickinson
Ack. That doesn't look very good. And it runs unplayably slow and glitchy /
flashy.

"Your first assumption might be that geod has hard-coded specific algorithms
for each supported game, training the emulator in how to detect specific
objects and create good-looking 3D models for them."

I don't know why anyone would assume that, given how poorly it works. My
current assumption would be that game-specific algorithms would be the only
way to make this work given how ambiguous NES graphics are. I suppose a top-
tier neural network expert could make a game-agnostic algorithm... but it
would probably handle a frame per minute on a multi-thousand-dollar machine.

I wish someone would come up with better filters for emulating CRT glow so the
game would look like a series of photos of the game being played on a top-of-
the-line CRT (PVM / BVM). But that would probably be way too slow, too.
Actually I have an idea of how to do it...

If I could get an array of pixels per frame into Unity or some other game
engine and make each pixel a light... or even just make each pixel a partially
transparent shaded sphere bitmap and multiply them together where they over
lap... it would probably run too slow but I once made something like that and
looked pretty good and ran plenty fast albeit at a sub-NES resolution.

~~~
pcwalton
You're way too pessimistic about doing image recognition on NES tiles. NES
games don't update the tileset very often during gameplay. They _can 't_\--the
NES has no framebuffer and a very underpowered CPU, so the VBLANK time is a
mad dash to push all per-frame data to the GPU (PPU in NES-speak) by the time
the video signal is going to begin. Most games barely slide under this window.
Lots of games didn't even try to do dynamic tileset updates and stored tileset
data in ROM.

Because of this, you can do a lot of graphics processing up front and cache it
on a per-tile basis. The hit rates on that cache are going to be massive.

~~~
khedoros
Agreed. The hardware itself does palette and tile lookups for each pixel, then
shoves the data out the wire. Building an emulator that way is more literally
how the hardware works, but it's slow. It's easier (and usually just as
accurate) to cache the tiles for the current palette settings and blit them to
the playfield as needed. There's no reason that the same wouldn't be true for
3D representations of those same sprites.

Updating the sprites in CRAM is fairly rare (and impossible in any game that
uses CROM).

