N64 emulators are all pretty bad (inaccurate, use a decent amount of resources) and upscaling is relatively expensive. At least, way too expensive for an rpi to handle.
The question is now, would it be possible for someone to make a port of Mario 64 that runs on the Pi, instead of trying to emulate it?
Usually after you get source releases to games, you get people that port them to different platforms. Like how we had Doom on iPods and Kodak digital cameras.
N64 only supported up to 240p, or in rare cases, 480i, which is basically the same thing computationally. Displaying on higher resolution just involves scaling (or up-sampling, but at that kind of resolution jump scaling is probably more appropriate).
I haven't tested N64 games on a RPi personally, but I imagine it would have no trouble with it, and there seem to be several retro-gaming projects that involve N64 games and use the Rpi.
I got a pi 3b+ as I romanticized the idea of it, but it struggles a bit with SM64. I just use OpenEmu on my higher-powered laptop and HDMI out instead.
Pi format is still fun to tinker with and I encourage you to get one if you're at all interested. The 3b+ just wasn't the right tool for the job in my case. I haven't tried the pi 4, however.
