There are some 4xx codes that should be retried. 408, 413, and 429 at the very least. There are also cases where it makes sense to retry 404, such as when polling for a resource that is being created and will soon be available.
You are right about the 429, of course, and my more "advanced" clients take it into account and only reissue requests after the requested backoff time. The smaller "stupid" clients error out, as they should, because a dumb retry would just cause "angrier" responses from the server.
I am not sure about the other two though.
413 definitely does not seem retryable, if the client continues sending too large content. Special handling would be needed to break down the data into smaller chunks and when I encounter this error I just break it down to smaller chunks beforehand so built-in handling is not needed.
Never encountered 408. It might be because I never keep connections too long...