I'd recommend a Gameboy Advance SP, because the older, non-backlit, screens were genuinely poor under some non-ideal lighting situations, and you still get support for older generation games. But a standard Gameboy Advance or even Gameboy Color is still good. I'd avoid things prior to the Color personally.
GBA SP: Play Summon Night Swordcraft Story, SRW Original Generation 1+2, Castlevania Aria of Sorrow, Yggdra Union (Warning: extremely difficult), Harvest Moon, Riviera (I played the PSP version)
NDS: Ghost Trick, The World Ends With You, Ragnarock DS, Devil Survivor, 999, Rune Factory 3, Disgaea DS