
Stackoverflow abuses nofollow (2011) - nerfhammer
http://www.brianbondy.com/blog/id/104/stackoverflow-amongst-nofollow-web-abuse-sites
======
kmontrose
This was dealt with more than a year ago, when we did remove nofollow from
trusted posts links' [http://meta.stackoverflow.com/questions/111279/remove-
nofoll...](http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-
links-deemed-reputable).

Would have been dealt with even earlier, but our experiments with it
(<http://meta.stackoverflow.com/a/69032/130213>) lead us to believe Google was
starting to treat us as a link farm.

What we were seeing in our earlier attempt wasn't pages appearing lower in
results (which could reasonably be expected in some cases as a result of the
change; and we wouldn't care, 2nd on a search under an equally good resource
is fine by us), but pages being removed entirely; almost universally pages
that had newly un-nofollow-ed links to stores, or ad-laden sites (mostly
legitimate posts, but some spam that stuck around for a bit before the
community deleted it). So, "classified as link farm" seemed like the most
likely problem. Google naturally won't tell you why your ranking drops, so it
was (and remains) just an educated guess as to what happened.

So we stopped, putting it on the "wish we could, but reality doesn't let
us"-list until Google reached out to us (among thousands of others, I'm sure)
to change our nofollow practices. Google didn't describe any changes in their
algorithm, but it seems reasonable to me that there would have been some
tweaks around nofollow to accompany a new policy; again just an educated
guess.

Basically, this is an old post complaining about a long since addressed
concern that we had tried to address even earlier but ran into practical
problems with.

While the exact details of our algorithm are secret by necessity, I will say
that we've had to consider posts individually to prevent spammers from kiting
a single account up to post nofollow-less spam content. People still try, it's
kind of astounding how much spammers try (I suspect SEO's opaqueness cuts both
ways here), but it doesn't work (well, you can get _one_ link in your profile;
but there's less SEO juice to pass and you are hard-capped at one, no matter
how long it takes someone to delete your account).

Disclaimer: Stack Exchange employee, I was on all the relevant calls but has
been a _couple years_ so grain of salt and all that.

~~~
codinghorror
This was also the exact time of all the Google Panda tweaks and the fallout
from Google starting to penalize creative commons scrapers of Stack Overflow
content. There were a lot of balls in the air and only a few of them were
ours, so we wanted to be careful.

When Google is 90% of your traffic, you REALLY REALLY REALLY do not want to
get on their bad side, even accidentally, so I can hope you understand why we
wanted to be cautious to the extreme here. If Google decides you're not doing
things right, it is literally a business ending move.

Unless your VCs and investors are cool with you losing, y'know, _ninety
percent of your traffic_.

~~~
patio11
Somebody please remind the community about this post the next time someone
says "You only have to care about Google algorithms if you're a fly-by-night
spam site."

------
eli
I guess I'm in the minority, but I'm not really buying that this is abusive.

Yes, it would be nice if SO removed the nofollow from known-good posts (which,
indeed, it seems they are now doing), but adding nofollow is a pretty simple
and reasonable way to make the site a whole lot less attractive to some of the
most obnoxious sort of spammers. I would hope nobody is adding links to their
SO answers with the _expectation_ of an SEO benefit.

~~~
bryanlarsen
If there's a link in a stackoverflow answer of mine, it's there because
there's really good information at the end of the link. That's the sort of
information Google & Bing use to make their searches useful, it's the sort of
information I want Google et al to know.

~~~
danielweber
Google and Bing are free to decide to just disregard 'nofollow' tags on
stackoverflow.com if they want.

Remember the tag was only put in place to help search engines in the first
place. (Well, to stop people from spamming to get good search engine results.
I think you get my point.)

~~~
davorak
I remember a court case where a website's terms of use precluded search engine
bots for indexing the site. The fact that the site owner did not use the
industry standard of the nofollow tag ultimately worked against them.

So yes google can disregard nofollow on a case by case basis and face the
associated potential legal consequences. I am sure that this would not be a
problem in the case of stackoverflow, but I am not sure this is true in
general.

~~~
Firehed
I'm going to assume you actually mean robots.txt instead of nofollow, since
that's the only accepted way to automatically state "do not index this site"
(or portions of it). The big difference here is that the site in question
controls the content of robots.txt, while it's the owners of the sites with
inbound links that control the presence of rel=nofollow.

As such, I don't think nofollow could have any possible legal consequences.
It's intent is to indicate that you haven't vetted the links in question and
as such are specifying a lack of trust in their content (i.e., user-submitted
links); though like the article says, many people attempt to micromanage SEO
through them which dilutes their usefulness (I've been told we have some links
that do the same thing; no doubt someone thinking they were smarter than
Google... so now we have to track them down and undo these pointless
additions)

------
gojomo
Sites are free to overuse 'rel=nofollow', and search engines are free to
selectively ignore it (for either link-discovery or ranking purposes).

The strongest point I see made by this author is about SO's hypocrisy: SO
requires attribution to be with a link, and specify that link must not have
'rel=nofollow'. Yet their content relies heavily on references to elsewhere
which are _all_ 'rel=nofollow'ed.

A sense of fair play in attribution, and spirit of mutual assistance between
reliable authorities, would suggest allowing at least some well-vetted
outlinks to be unencumbered.

~~~
nerfhammer
Quoting Jeff Atwood:

 _If you republish this content, we require that you: [...]_

 _Hyperlink directly to the original question on the source site [...]_

 _By “directly”, I mean each hyperlink must point directly to our domain in
standard HTML visible even with JavaScript disabled, and not use a tinyurl or
any other form of obfuscation or redirection. Furthermore, the links must not
be nofollowed._

 _This is about the spirit of fair attribution. Attribution to the website,
and more importantly, to the individuals who so generously contributed their
time to create that content in the first place!_

 _Anyway, I hope that clears up any confusion — feel free to remix and reuse
to your heart’s content, as long as a good faith effort is made to attribute
the content!_

<http://blog.stackoverflow.com/2009/06/attribution-required/>

------
VMG
The author doesn't address this response by Jeff Atwood:

 _URGENT UPDATE We were seeing a significant drop in Google (organic) traffic
for Server Fault after instituting this "follow links if enough upvotes post-
edit or post-create" policy.

We traced it back to what we currently think are a string of posts on Server
Fault that got nofollow removed through "trust", but were being interpreted by
Google as link farms or spammy pages.

[....]_

<http://meta.stackoverflow.com/a/51156>

~~~
bryanlarsen
There's a follow-up to the URGENT UPDATE on SO:
[http://meta.stackoverflow.com/questions/111279/remove-
nofoll...](http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-
links-deemed-reputable)

So it appears that they do remove rel=nofollow from reputable links, although
their threshold for reputable appears to be very high.

------
sologoub
Although I get the concern here, it would seem that if you removed the
nofollows, it would open up the site to all kinds of abuse and diminish the
value of the content very quickly. It doesn't strike me as a very difficult
task to create a few thousand profiles and have them vote for each other (and
also give you a ton of votes to random people to hide better), and then use
the collected power for SEO spam.

I agree with you that there has to be a better solution, but it doesn't strike
me as a very trivial one... any thoughts on how one would approach this?

~~~
greenyoda
"it would seem that if you removed the nofollows, it would open up the site to
all kinds of abuse"

That's why the author of the article suggested that they remove "nofollow" for
users who have an established reputation, like SlashDot does. That algorithm
would seem to be very easy to implement.

~~~
manojlds
What happens when a less established user edits the post of a more established
one ( and maybe also vice versa)

~~~
bryanlarsen
You need a good amount of karma to edit someone else's posts, so that's a non-
issue. If it was an issue, the rel=nofollow should be based on the rep of the
user creating the link, whether that was the poster or the editor.

~~~
jakub_g
You need 2000 rep to do this, which is not so easy to gain, but certainly
achievable for a highly determined person. However, what @wookietrader wrote
applies -- each edit is marked with the person who has done it which solves
the problem.

------
jeremysmyth
This topic is discussed quite frequently in the meta pages on StackOverflow.
For example, [http://meta.stackoverflow.com/questions/111279/remove-
nofoll...](http://meta.stackoverflow.com/questions/111279/remove-nofollow-on-
links-deemed-reputable) covers exactly this subject.

SO gets lots of googlejuice, and because it's gameable, they appear to want to
give some of that googlejuice only to "reputable" links (which they do for
links in high-rep users' profiles). That's a really hard thing to judge, but
the fact that they're thinking about it, talking about it, and soliciting
feedback on it means that they're trying to do the right thing, even if the
article's author doesn't think their "right thing" is good enough for him.

------
HPBEggo
I can understand why they do it - diminishing the page rank of sites that give
answers to questions on Stack Overflow causes Stack Overflow to gain more
traffic - but it still seems kind of underhanded. They rely on others for the
content of the site, so they should give credit to the producers of that
content - without it, they would have no service at all.

------
netcan
This nofollow stuff has always seemed very weird to me for several reasons.

\- Search Engines are supposed to be analyzing the web, figuring out whats
important. They have the incentive to do this well. The websites they are
analyzing do not. Websites have an incentive to nofollow everything. They
might be worried about their own rankings, but why should they care about the
sites they link to. Nofollow is safe and harmless. No-nofolow should get you
in trouble. _what does a site gain from not using nofollow links on
everything?_

\- To enforce nofollow rules, Google are supposedly disciplining sites by
hurting their organic rankings. But a page with nofollow links appears exactly
the same to a user. It's just as right an answer as it was before. Is Google
lowering the quality of its results to police the web?

\- If Google are able to detect "link farms" and user generated content that
should have been nofollow-ed, why don't they just treat those links as
nofollow and ignore them. Use this detection to _analyze the web_ , not police
it.

\- Are they ignoring important information sources? A huge, information rich
portion of the web is user generated. Eg wikipedia & stackoverflow. How can
Google really be ignoring these links here as data sources? The links on a
wikipedia page for example, are very informative. If some webpage is mentioned
frequently in stackoverflow questions and answers, it's probably important and
its probably a good answer to a lot of questions people are asking google.

------
bhanks
How is this news?

1\. This post is from 2011 2\. Google ended the ability to link sculpt a long
time ago.

I would be interested in studies showing cases where link sculpting still
worked.

~~~
kamjam
Agree. The original post is actually from from Dec 2010 and an update on Dec
2011!

------
robomartin
I really don't see SO as being the problem here. I happen to think that secret
SEO/page rank algorithms are the culprit in many ways. Google is in the unique
position as to be able to write a "how to behave" guide for the internet. Play
by their rules and you are golden, start to violate them and you start to be
"punished" in a gradual (but exponentially more severe). The whole time full
feedback and data on what you are doing wrong is provided. As this approach
evolves spammers would be less and less successful and might just have to
resort to legitimate means of marketing.

~~~
JangoSteve
Furthermore, the search engines could nip this kind of abuse in the bud, by
ignoring the "nofollow" attribute of the links on a site if it determines that
site is abusing it (e.g. if all or some high percentage of the links on the
site have "nofollow").

------
SenorWilson
They're technically using it for it's purpose, or they can claim they are even
if they have a different motive. Stackoverflow allows links in comments and
posts and this _could_ be abused.

~~~
eli
Sites with high pagerank that allow user submitted links are few and far
between. If they didn't put nofollow (at least for new accounts), SO is
popular enough that it would have to deal with targeted spam attacks aimed
specifically at them. That is a sucky problem to deal with.

------
theseanstewart
I feel the exact same way about Wikipedia.

~~~
patmcguire
Especially since the revert brigade makes it very difficult to make any
legitimate changes, let alone spammy ones, on any page that would be worth it.

------
debacle
This is an interesting reality. I know that some of the SO guys frequent HN. I
wonder if we'll see a valid response.

The only possible reason I could see is spam avoidance.

------
205guy
Esprit d'escalier: "In my day (usenet), we didn't have links--and we liked
it." (old SNL joke for those who don't get it).

This conversation has 2 angles I find very interesting. The first is the whole
question of quality of the content. Google is a huge AI trying to give you the
same quality of information that you'd select if you looked at the same
(millions of) sources. At first, I liked netcan's answer[1], which says
essentially, nofollow them all or none, Google will sort out its own. But the
more I think about it, the more context that sites can provide automatically,
the better. If a site can say "this content provided by untrusted source"
(because we're a community-driven site, and we can't police everything),
that's a help to Google's Algorithm. Google is free to still follow the link
and perhaps, using it's other knowledge of the target, assign a
trustworthiness score to the nofollow quality on the original site. If a good
community is providing good links, but the website owner still marks them
nofollow, that might become a badge of honor in Google eyes (all-crawling
robot tentacles?).

As a side note, I love how everybody, from slimy SEO to internet ethicist, is
trying to guess Google's secret formula (it used to be Coca-Cola that had a
secret formula that was releveant). Google is essentially the unknowable
proto-God of this new information universe.

The other interesting topic is getting paid for content. My first reaction to
the OP was "wah, wah, OP doesn't want to pay for his little corner of the
internet." Let's face it, the internet doesn't run for free (hardware and
electricity), and the content that attracts people doesn't write itself.
StackOverflow provides a community for people to deliver their content into
and receive socaial rewards. If SO can't do that because of some quirk of SEO,
then it's likely it wouldn't be able to host the community anymore. The
internet has still not figured out a way to pay for that other than
advertizing (the forcing of unwanted content on readers). But advertizing only
works if you can keep track of views and stay on top of the SEO game. I
totally agree they're hypocrites for hoarding the follows (must follow to us
be we nofollow to everyone), but how's that different from a corporate board's
fiduciary duty to shareholders to maximize the profit of the company? I also
think cheald[2] makes a very good point about why the dynamics must be towards
nofollow. The question is then how involved or committed are you to the
community, and how do you feel rewarded for the time you contribute to it.

[1] <http://news.ycombinator.com/item?id=4777260> [2]
<http://news.ycombinator.com/item?id=4774888>

------
cstrat
Someone have a mirror of the article?

~~~
cstrat
google cache seems ok:

[http://webcache.googleusercontent.com/search?q=cache:3sKnQOn...](http://webcache.googleusercontent.com/search?q=cache:3sKnQOnbA0UJ:www.brianbondy.com/blog/id/104/stackoverflow-
amongst-nofollow-web-abuse-sites+&cd=1&hl=en&ct=clnk&gl=au)

