What you can do, and many emulators do, is provide a hook to say "when the game draws texture / sprite X, instead draw higher res texture / sprite Y". Obviously, this can be rendered at a much high resolution than the original system. In the case of Gameboy there is no explicit pixel buffer anywhere, so there is no way for the game to even know the emulator has done this.
