Yep, this is what I meant to allude to, and you've worded it much better than I could have.
Perhaps a nice way to say it is that the mathematical objects necessary for physics that I can think of are separable (such as the real numbers). Basically, whenever you have uncountable sets, they come along with some topological structure which must be handled continuously.
There's even an argument that separability is sufficient for most of mathematics let alone physics (http://arxiv.org/pdf/math/0509245.pdf) but you're going to have to work very hard to persuade me that if we're going to describe any sets as physical then uncountable ones are less physical than countable ones.
Perhaps a nice way to say it is that the mathematical objects necessary for physics that I can think of are separable (such as the real numbers). Basically, whenever you have uncountable sets, they come along with some topological structure which must be handled continuously.