Related, I think it would be a lot of fun to interface the Pi with the original (not very good) LCD. Then you could make some kind of 'alternate history' nextgen gameboy where the processor was massively upgraded but the screen was not - enabling low res 3D, temporal dithering to expand beyond the 2bit palette, etc.
(I guess you could also simulate this on a regular computer or even the mod discussed here, but that's somehow slightly less charming)
Although myself, much like the OP, would almost certainly run RetroPi off of it.
There are a few emulation stations kicking around (you can find them on Amazon) but nothing near this compact or with this good of controls.
I've found a DS flash cart and a DSiXL to do this job well enough for what the system is capable of emulating (up to GBA, Genesis, SNES). Not sure what the state of 3DS flash carts is. For ~N64/PS1 a modded PSP is a great choice from what I've heard.
It'll do just about anything up to SNES, which includes Neo Geo, and a huge subset of the MAME library.
just checked ebay since the company that made the GP2X-Wiz is out of business, holy smokes I can't believe how much they're going for
These days, I'd probably go in for a GCW-Zero, an A380e, or if you have the money, get an NVIDIA Shield (on my list to replace my GP2X-Wiz).
All of this said, the PSP (1001?) was/is an awesome emulator... Capable of playing everything up to PSP backups, you could play all of the above, with PS1, NeoGeo/Capcom, and PSP support... Definitely a very nice gaming system with a lot of power and capability... In the end, it was the GBAs battery life, wonderfully thought out screen and compact size that always made me come back to the gba. The PSP got a lot of play, but being able to stomp turtles and collect rubies while sitting on a beach without shade just made the gba the general purpose go-to for this casual gamer (as the PSP could serve as little more than a mirror and hand warmer in this setting).
While generally frowned on by gaming manufacturers for obvious and founded piracy concerns, there are some awesome hybrid systems that can be made out of existing on-market systems. While I am sure that there are many whose ultimate goal is piracy, there are a lot of people who enjoy modded systems for their flexibility and enhanced capabilities.
But as far as my original comment, this is actually what I meant: https://news.ycombinator.com/item?id=8082102
A more intense and compact route could be to use the Compute Module, but it might actually end up a little more expensive doing it that way.