Actually, at least on an iDevice with iOS 4, they don't have access to your phone number. Unless the user types it in (or selects themselves from the address book).
Seems like it's easier on Android, but I'm not certain, just googled a bit.
They removed it in 4. It was there before but it was the number from iTunes activation, which wasn't necessarily worth anything (or filled in). Apple wouldn't approve you if you touched it, either, and it was undocumented.
Device ID they don't seem to care about; I send it in my app's User-Agent and I've survived several approvals. That's uniquely identifiable, but at least you can't call the user. As you suggest, most apps pop up contacts and have you pick yourself, it seems.
