Just because potential attacks exist doesn't mean it isn't a solved problem in practice.
There are plenty of well-research PRNG algorithms out there, some of which are used in very high-profile applications like TLS and have been exposed to hostile parties for over a decade. If you are building an HSM you are almost guaranteed to already have a battle-hardened implementation of one lying around.
It really depends on what you expect as a threat. Ask a Casino owner if they fully trust an RNG system and you will get laughed at. Randomness is not encryption and can be broken with enough knowledge of the system.
https://en.wikipedia.org/wiki/Random_number_generator_attack...