I see it rather differently. Referrer is a 1990s concoction based on the idea that the Web is a collaborative effort that we all engage in. I don't personally have any problem in a Web site knowing where I arrive from in most cases. It's harmless him of information (in most cases, as I say) which makes the Web site owner's life a little better.
Same here. Referer is part of the way the Web works, and I, for one, don't give two shits if Site A knows that I arrived via a link from Site B.
That said, I can see why certain people might see it as problematic... if you're browsing http://www.anarchistsbombmakingforum.com and somebody posts a link to http://www.fbi.gov, then maybe you don't want the FBI knowing you were at anarchistsbombmakingforum.com. But, still, barring other privacy problems, the FBI don't know who you are when you visit their site, just that you came from anarchistsbombmakingforum.com.
I have a hard time getting worked out about this though... for one, if you're surfing anarchistsbombmakingforum.com, common sense would dictate that following a link to fbi.gov isn't such a good idea (and a forum that fosters discussions of anything controversial should probably munge links to go through an anonymizer anyway) AND the people for whom this really matter are the people who have a referer blocking plugin installed in their browser.
