So instead of speculating on why the change happened, I've found a few bits of information around the limits that make me feel like it often simply won't be a problem.
There's some interesting discussion around the limits across various platforms here .
Whilst I'm not entirely clear on the motivation, even Lua running inside a browser under emcripten has a stack limit of over 4000, which is somewhat decent even for recursive functions.
Whereas on Linux you seem to be able to tweak a config when building and safely have it in the region of 20,000, and quite possibly more.
Even under a tight restriction, Lua seems to cope fairly well with it when recursing (the example crashed at 1758 recursions for a ulimit of 400).
Whilst Python has a terrible limit of 1000 for the default recursion limit which is somewhat comparable to this aggressive test, Python also has no tail call elimination - but Lua does.
I have a somewhat large Lua project (6000 lines of C, 10,000 lines of Lua) that is a recursive parser for a format that I passionately hate (think self-modifying LaTeX). With the default 5.4 limits it never hits a point where it crashes, though I had expected it to.
I have a few of those badly behaved recursive functions in the parser I mentioned at the end of my comment, and expected them to overflow, but they didn't hit the default limit.
Lua 5.4 adds a new function, lua_setcstacklimit, but this merely exposes what was a compile-time constant, LUAI_MAXCCALLS, in earlier versions. Lua can't avoid making some use of the C stack as it supports intermingling of lua_call's from C code with in-language VM calls, which necessarily will use some amount of C stack. Lua has lua_callk for yield/resuming across C code invocations, but not everybody makes use of this and in any event it's only for coroutine semantics, not for recursion.
The limits for recursion have undertaken a significant change. (Though it might not be that concerning, if you see my sibling answer).
> As already discussed, Lua 5.4 is using a "non stackless" implementation. That means that it uses the C stack for recursion in Lua functions. Because of that, the maximum depth of recursion is much lower in 5.4 than it was in 5.3. - Roberto Ierusalimschy 
The embedding API explicitly exposes a stack, so I assumed that it was implemented that way.