
The Edge of Emulation - NieDzejkob
https://byuu.org/articles/edge-of-emulation
======
wgjordan
> If we latch too late, after cycle 40, Full Throttle Racing will glitch. But
> if we guess a value too soon, say on cycle 0, that might be too early, and
> it might cause issues in another game.

> We would have to retest the entire SNES library to be sure such a change did
> not cause any regressions, which just isn't practical with a library of
> 3,500+ game titles.

Running automated regression-tests against a library of known-accurate titles
actually sounds extremely practical. This sounds like standard software-
engineering practice to me.

I'm surprised that a test-harness like this doesn't already exist by now given
the relatively mature state of SNES emulation. It seems like it would be an
extremely helpful tool to benchmark accuracy of new or existing emulators,
prevent regressions from advanced optimizations, etc.

Automated testing could transform the 'blind guessing' approach on tweaking
PPU timings from a game of 'whack a mole' to a structured, iterative linear
algebra solver. Which could end up being quite a bit more feasible than
acquiring 100x die scans of the chip and tracing the logic from the hardware
directly.

~~~
kevingadd
You can't just start it up and run it for 5 seconds, though. A given game
might be a hundred hours long and only exercise certain functionality at
minutes 5, 37 and 60. It's non-trivial to automate the playback of the whole
package and then verify the whole stream against a known good. So you'd need
to identify the key points in each game that are important, and replay those
with save states and hope that the save states aren't unintentionally encoding
corrupted data that came from an existing bug in the emulator you used to run
them.

Also, none of that automation you just did will work w/the actual hardware.

~~~
theresistor
The input stream for an SNES isn't very complicated - just the state of a few
buttons at every cycle, and the transition edges are quite sparse relative to
the clock frequency. You could easily record human play sessions and replay
them.

~~~
byuu
The SNES output is non-deterministic across runs.

Not only is there unitialized RAM and I/O registers, and some analog effects,
the really big elephant in the room is that the system has _two_ oscillators.
A ~21MHz CPU/PPU crystal oscillator, and a ~24MHz SMP/DSP ceramic oscillator.

Given that not only do these exact frequencies change between systems due to
margins of error on clocks, they also change slightly as the system runs (and
gets warmer, for example.)

Every SNES game has sound routines that synchronize the CPU to the SMP.

It wouldn't be possible to make a literal 1:1 play log unless you a) ran a
custom register and memory initialization at system startup, and b) replaced
the two oscillators with a much faster single oscillator and then used a clock
dividier to drive both the CPU and APU off of it.

(You _can_ TAS certain SNES games anyway, of course. It really depends on how
the game is programmed to react when the exact CPU<>SMP communications change.
If it seeds a random number generator based around the PPU H/V counters that
are polled after a CPU<>SMP sync for example, forget about it.)

~~~
vardump
> Given that not only do these exact frequencies change between systems due to
> margins of error on clocks, they also change slightly as the system runs
> (and gets warmer, for example.)

Maybe it'd be possible to modify some boards to use a CPLD (or an FPGA) for
synthesizing those clocks from a common source. This could eliminate most
(all?) uncertainty.

~~~
byuu
It would certainly help a lot. People have talked about doing this, but I
don't think there's been enough interest to actually make it happen.

------
nitrogen
One of the later comments on the article offers custom PCB design to make a
test harness for the PPUs. It seems like a perfect partner for a manually
controllable clock for the whole SNES, e.g. as Ben Eater's 6502 demos use. If
you can freeze the clock at will, you can more easily inspect the data and
address lines. But this wouldn't work if the PPU latch timing depends on
analog effects or the PPUs have an internal clock.

Combine the test harness with a USB logic analyzer and signal generator, and
borrow some concepts from fuzzers like AFL, and it should be possible to
automatically identify critical phases of the PPU operations.

Question: do the PPUs themselves generate analog video, or is there a separate
DAC whose bus could be tapped to figure out what color the PPU is producing?

~~~
byuu
The PPUs use static logic and an external 21MHz oscillator (which also powers
the CPU), so they're perfect for single-stepping in isolation from the rest of
the SNES.

The PPUs themselves output only analog RGB, there's an analog pin for each
color channel rather than a digital pin for each color bit.

On some level, I don't doubt it's possible to build a test harness that can
automate things based on the analog RGB values with some fuzzy matching, but
if we are going that far, it just makes more sense to snoop the bus traffic
directly. That would reveal a lot more information and in digital form.

~~~
londons_explore
> it just makes more sense to snoop the bus traffic directly

It seems the bus traffic won't tell you everything. Just because the PPU reads
a specific piece of a sprite from RAM, doesn't tell you which pixels of a
scanline it'll get rendered on.

If I were approaching this problem, I would observe that the 'analog' output
of a SNES is probably a bunch of discrete values with discrete timing
intervals, and therefore can be perfectly captured with no error, especially
if you can slow the clock down to eliminate signal reflections etc.

I would then make a test harness consisting of a PPU chip, the ability to read
and write registers (either with the rest of the SNES, or some other
microcontroller, whichever is easier) and the ability to capture complete
frame outputs from the 'analog' outputs without error.

I would then do the same with an emulator, and run code on both which pokes
random registers at random (clock accurate) times, and whenever a difference
in output appears, debug it.

~~~
sitkack
I think you could use a NN to attempt to replicate the circuit under test,
perhaps generating verilog, programming a chip and then running it against a
set of autogenerated test vectors.

A generalized system for duplicating the functionality of a boolean circuit.
If can use its own axioms and generalities, once it "understands" flip-flops
and nand gates, can start to directly synthesize more complex logic.

~~~
planteen
That sounds like a great idea, but you today we can't even synthesize
arbitrary VHDL/Verilog code efficiently. The author needs an understanding of
hardware architecture to write something that synthesizes versus just a test
harness not intended to be synthesized.

------
krallja
I love that there are people whose hobby is preserving things. SNES was an
important part of so many 80s and 90s kids’ childhoods! But how many of them
have digital logic experience, especially at the reverse engineering level?
And actually want to do it outside of compensated labor?

~~~
byuu
> how many of them have digital logic experience, especially at the reverse
> engineering level?

Like programming and reverse-engineering, I taught myself the basics of
electrical engineering.

The Neo Geo Pocket's SoC is basically nothing _but_ raw logic circuits for
explanations of how things work. Imagine 200 pages of diagrams like this:
[https://i.imgur.com/2LZ2UWY.png](https://i.imgur.com/2LZ2UWY.png)

(oh and as a bonus, the diagrams often contain errors.)

If you're only looking to create a basic emulator, you can mostly get away
without knowing all of this stuff by reading tech docs and the source code of
other emulators. But if you really want to get low-level and get things clock-
cycle accurate, or want to work on a system that's not been well-emulated to
date, digital logic pretty much a hard requirement.

> And actually want to do it outside of compensated labor?

From 2004 - 2018 for the SNES, it was mostly only me. As of today, three
people.

In general only one person is really needed. I couldn't say if someone else
would have taken my place had I not been around. I suspect so, given how
popular the SNES continues to be. Whether they would have done a better job of
it than I have is a question that often keeps me up at night.

In any case, the answer is certainly "not nearly enough."

------
gregoryl
What's the barrier to getting the chip decapped and scanned at the needed
resolution?

~~~
byuu
Really, just someone with the skill and interest in doing it. Money's no issue
unless they want something crazy like $10,000+.

~~~
zamadatix
John D McMaster seems to have previously done 100x scans for some chips if I'm
reading the site right e.g. the latest has a link for 20x and 100x
[https://siliconpr0n.org/map/generalplus/gplb52a24a-049a/](https://siliconpr0n.org/map/generalplus/gplb52a24a-049a/).
Is it as simple as seeing if he'll rescan the PPUs at 100x or are these not
the quality that I was thinking?

~~~
byuu
He has a tremendous backlog of work to do at the moment.

I would prefer to not bother him unless it was a last resort, but that is
indeed an option if my breadboard idea does not work out and no others are up
to the task of decapping the PPUs.

------
peter_d_sherman
Excerpt:

"If we could recruit a talented electrical engineer, I believe that a custom
PPU breakout board could be designed that would aid us substantially in
reverse engineering..."

Perhaps Andrew "Bunnie" Huang -- would be someone to go to, for this?:

[https://en.wikipedia.org/wiki/Andrew_Huang_(hacker)](https://en.wikipedia.org/wiki/Andrew_Huang_\(hacker\))

------
jimmaswell
Why are test roms too slow if there are at most 41 cases to try? Is getting a
game on an snes cart arduous?

~~~
byuu
There are 52 registers comprising approximately 100 settings, there are
probably close to twice that number if you include the internal register
latches (some of which we know about, some of which we don't know exist yet.)
The SNES PPUs are _heavily_ based around combinatorial logic. Changing the
timing of one variable can alter the timing needed by other variables. Think
of each setting as potentially doubling the number of tests needed (at least
when it comes to a blind, brute-force approach.) There are more combinations
of settings and pixel generation patterns than atoms in the universe.

The way we've gotten as far as we have is that not every combination needs to
be tested. It is a ballpark estimate on my part, but I'd estimate us needing a
few million tests to have a high degree of confidence our emulation is
correct.

If we had to make those tests into visual patterns on a screen that had to be
checked by eye each time, it would be overwhelming even if we only needed a
few thousand tests made.

I can test code on a live SNES extremely easily with my 21fx board (
[https://github.com/defparam/21FX](https://github.com/defparam/21FX) ), but
it's still far too much to do this by hand.

~~~
auto
byuu I'm curious, is there some magical piece of documentation/schematic that
would break open an entirely untapped area in your SNES research? Or are you
at the point the hardware is almost entirely transparent, and it's just a
matter of chipping away at the edge cases, or halo projects like a replacement
clock for the dual oscillators?

It'd be interesting if some Nintendo engineer for the early 90s has a box in
his garage with documents that could change the whole course of your current
work.

