You're missing one key difference: Switzerland never had the overseas garrisons to start with. For the US not to, it will have to withdraw from them. That will be seen as a sign of underlying weakness. The strength of any currency is relative to the confidence that the government that operates it will continue to do what it does.
There's precedent: look at GBP as the UK dismantled its empire.
There's precedent: look at GBP as the UK dismantled its empire.