Usually an XP system improves a character's numerical stats without adding or removing specific abilities; or, it acts as a catalyst where every time you accumulate some amount of XP you get to choose a new ability from a list. You can have the hero's journey progression without using numeric XP or leveling to keep track of progress, e.g. Zelda, Metroid, Megaman.
A reason for XP is to serve as some kind of yardstick to your progress. A non-linear quest (such as in the Fallout games) uses XP to give you new abilities the more you play the game, whatever you happen to do in it. Whereas in linear stories you can simply present the hero with whatever bonus their progress deserves.
Note that you can have the character-improvement mechanic in more than RPGs; a game like Half-Life introduces weapons, equipment, and new ways to manipulate the environment as you progress.
Well, I'll admit I haven't played Metroid (shame on me, I know) but both Zelda and Megaman add complexity and power to the player throughout the game. They do this in the form of new abilities instead of levels, but the principle is the same: your character now can do more than he could do before. There's no way I'd say on either of those games the "character came through about as strong at the end as it was at the beginning". XP may not be the underlining mechanic to improvement, but character improvement is a(if not the) core mechanic of both games.
The one game that I can think of as having an excellent portrayal of the hero's journey without character improvement is Journey.
The difference is that RPG leveling systems don't care about you passing a specific challenge - they just care that you've spent a certain amount of time in an area. In Metroid and Zelda, if you know the solutions to the puzzles, or are just really good, you can blaze through them and get to the good part in a fraction of the time spent before. That's why there's an entire subculture of people who speed-run Metroid-style games, and not so much for Final Fantasy.
I think that that kind of progression is a more accurate indicator of a) whether the player is ready to handle the complexity that the new ability brings, and b) the value the player has received from the game, so far (if we're dismissing the notion that time spent is necessarily proportional to value).
