Lua 5.4 adds a new function, lua_setcstacklimit, but this merely exposes what was a compile-time constant, LUAI_MAXCCALLS, in earlier versions. Lua can't avoid making some use of the C stack as it supports intermingling of lua_call's from C code with in-language VM calls, which necessarily will use some amount of C stack. Lua has lua_callk for yield/resuming across C code invocations, but not everybody makes use of this and in any event it's only for coroutine semantics, not for recursion.
The limits for recursion have undertaken a significant change. (Though it might not be that concerning, if you see my sibling answer).
> As already discussed, Lua 5.4 is using a "non stackless" implementation. That means that it uses the C stack for recursion in Lua functions. Because of that, the maximum depth of recursion is much lower in 5.4 than it was in 5.3. - Roberto Ierusalimschy