I'm probably biased because I spend good part of quarter by designing reasonably secure CSPRNG for smartcard chip without hardware RNG (and ended up exploiting essentially any cross-clockdomain communication as entropy source) and thus I assume that typical smardcard vendors don't care about that (too much work) while HSM vendors simply leverage infrastructure of whatever (RT)OS they use and probably harden that somewhat.
