My greatest concern with the Ledger hardware wallet has been making sure the device hasn't been tampered with during shipment. Fortunately they provide a script to check hardware integrity, it's probably a good idea to run it before doing anything with the device.
Note that this doesn't actually do anything to attest the safety of the device, as has been pointed out in a CCC talk recently. It attempts to confirm that the code running on another processor is legitimate by asking it to read its entire flash to a "HSM" chip, which is obviously simple to deceive by reading back something that is not the processors flash. I personally think that this is deceptive and counter productive.
Remote attestation implementations via HSMs will always remain subject to a confused-deputy problem, but they're still leaps and bounds better than pure software solutions. Any threat you can describe that involves a facade hardware UI is much easier to implement in software, meaning that attackers are more likely to invest resources in software attacks (like spraying bad Electrum servers into the pool) than hardware attacks (like modifying hardware wallets and setting up a storefront on eBay).
https://support.ledger.com/hc/en-us/articles/115005321449