Ultimately, the cops let me go, because there was nothing they could do.
In this particular instance it may have seemed this way, but if you continue, you will likely find out it's not really true. They'd just rather not go to the effort of doing what they can do. Enforcement is selective, so unless you think you can get a lot of attention in time to do you some good, it's best to pick your battles wisely.
If they're annoyed enough, they can find something you did do, and charge you for it. Depending on how visibly you live your life, this might be easy or difficult, but ultimately they can just "remember" something you said that you didn't say, and then you could go to jail. All this is chancy, since the jury, should it come to that, may be sympathetic to you, but if you're habitually confrontational, that might be less likely. Basically, the Martha Stewart conviction destroyed what remained of my faith in the "justice" system. Here was a woman who, in theory, had the deck stacked her way... money, fame, etc. What was she convicted of? Lying to a federal officer, not under oath, about not committing a crime that they couldn't convict her of in the first place. The conclusive evidence basically boiled down to one or more officers' word that she denied wrongdoing.
Let me recap: if some police are determined to put you in jail, you can go to jail for lying to an officer, when you assert that you didn't commit a crime. They do not have to prove that you did commit the crime. Just saying you didn't do it is presumptively a lie, and a federal crime. If you give them reason to be mad at you, you're gambling with your freedom, no matter what you have or haven't done.
Note that not talking to the police is not sufficient to save you from prosecution, since the vast majority could be considered guilty of misprision of felony -- http://www.law.cornell.edu/uscode/18/usc_sec_18_00000004----... -- which is itself a federal felony, so if someone tells you they saw a felony committed (possession, say) and you know they didn't report it, now you're committing a felony unless you report them.
In this particular instance it may have seemed this way, but if you continue, you will likely find out it's not really true. They'd just rather not go to the effort of doing what they can do. Enforcement is selective, so unless you think you can get a lot of attention in time to do you some good, it's best to pick your battles wisely.