But you can. What the GPL says is that if you distribute/sell the object code, you must distribute the source code as well. So you have no obligation to distribute anything to anyone, but if you do distribute the code, you are obliged to distribute the source.
Quoting from Stallman's The Free Software Definition:
You should also have the freedom to make modifications and use them privately in your own work or play, without even mentioning that they exist. If you do publish your changes, you should not be required to notify anyone in particular, or in any particular way.
You can refuse to distribute GPLed software to anyone you want.
The fact that the Nexus One phone has GPLed software in it doesn't obligate Google to give me one.
The purpose of the GPL is to ensure that 1) When you do distribute the program to someone in binary format, you also make available the source and 2) You don't restrict their rights to modification and distribution with additional requirements.
What section of the GPL do these rules keep you from your obligations under?
Remember that the GPL does not require that you distribute to anyone. Instead it says that if you do distribute then you must provide source and cannot prevent the recipient's freedom to further redistribute. But regulations saying that you cannot personally distribute to person A don't prevent you from distributing to anyone else, so long as you follow the requirements of the copyright license.