It is extremely common -- i.e., absolutely normal -- to write code after an assertion that would be UB if the assertion were false. Any worry about eliding checks should apply even moreso to all that UB code. But people who hate optimization based on assertions have implicitly chosen to ignore all the UB, and concentrate only on the elided checks.
It's like complaining you don't have a parachute when you know the doors couldn't be opened anyway.
In a language that doesn't seem to have UB, those worries might seem less. But every substantial library makes its own versions of UB that, while they may have less drastic effects on the runtime consistency of the process, equally impact the coherent behavior of the program.